Search This Blog

2010-07-06

SQshelL - Building and Installing sqsh In Linux (Fedora 13) Without The Sybase CT-Library

If you're like me then you generally prefer fast, efficient command-line interfaces to expensive (CPU- and RAM-wise) and slow GUI interfaces. Recently, my Windows 7 (Seven) box got really slow, and in particular Visual Studio and SQL Server began to crawl. I couldn't really explain it because the Windows Task Manager said that the System Idle Process was using all of the CPU and the system was only using about half of my 3 GB of RAM, but nevertheless, scrolling a single line of code took seconds! Something had to be done.

I had finally had enough and insisted to j0rb that I was going to install Linux. Now, I currently still require Windows because most of our projects are .NET-based (and bleeding edge, .NET 4.0 stuff), and it seems Mono just isn't quite there yet. I was very pleased to hear that they do support many (or most?) .NET 4.0 features, but apparently not LINQ to SQL, which we make extensive use of. To solve this contradiction, I would install Windows in a virtual machine. I ended up installing Fedora 13 (x86) as the host, with Windows 7 (Seven) Ultimate in a VirtualBox guest. I was quite nervous about whether or not it would actually perform adequately (especially considering how poorly Visual Studio and SQL Server were performing on the raw metal), but so far so good.

On to the topic at hand: building and installing sqsh in Fedora 13. sqsh is basically a shell interface for Sybase ASE (I guess?) or SQL Server. It allows you to do powerful things that you expect from your system shell, like redirecting streams, and environment variables, etc.; as well as interact with a database in SQL. I'm not overly familiar yet with what it can or can't do (I just heard about it and went ahead and installed it), but my hopes are high. Hopefully it will at least reduce the need for me to wait for SQL Server to startup and close, or ugh connect.

Naturally, when I first heard of it I instinctively asked YUM for it. Sadly, I was disappointed to learn that it is not in the default repositories. "No matter! I will just install from source," I told myself.

The first step to installing from source is obviously to get the source. So head on over to SourceForge, where it's currently being hosted, and download the latest version (I'm using 2.1.7 at the time of writing).

http://sourceforge.net/projects/sqsh/

Assuming your results are similar to mine, you should now have a sqsh-<version>.tar.gz file somewhere on your file system. Firefox downloaded it straight to my Downloads folder (dl for short). I keep all source trees in ~/src for organization, so lets go ahead and extract it to there now (you can extract it where ever you like).
[bamccaig@krypton ~]$ tar -xzf dl/sqsh-2.1.7.tar.gz -C src
[bamccaig@krypton ~]$ cd src/sqsh-2.1.7
[bamccaig@krypton sqsh-2.1.7]$ 
Great. So far so good. Now lets take a look for "readme" files that we should read before proceeding (which are typically named in all uppercase):
[bamccaig@krypton sqsh-2.1.7]$ ls -1 | egrep '^[[:upper:]]+$'
AUTHORS
COPYING
INSTALL
README
[bamccaig@krypton sqsh-2.1.7]$ 
Excellent! There is a README and most importantly an INSTALL! Go ahead and read through both of those files now. I'll wait.

... ... ...

OK, now that we've read those two files, we should all be aware of the dependencies. Basically you need Sybase CT-Library OR you need FreeTDS. This threw me for a loop at first because when I reached the Sybase CT-Library dependency it was worded in such a way that made it sound absolutely required. If you go on to read the next dependency, you'll see that you can alternatively use FreeTDS. It sounds like there is actually a couple of freely available Sybase downloads for Linux (free as in beer, not as in speech), but I'm not entirely sure which, if any, contain the Sybase CT-Library and worse you're required to register with Sybase to download them (ugh). I was pleased to discover that doing so is unnecessary.

Instead, I say again, you can use FreeTDS; and so that's what we're going to do. Lets see if we can find it.
[bamccaig@krypton sqsh-2.1.7]$ yum search freetds | grep '^freetds'
freetds-devel.i686 : Header files and development libraries for freetds
freetds-doc.i686 : Development documentation for freetds
freetds.i686 : Implementation of the TDS (Tabular DataStream) protocol
[bamccaig@krypton sqsh-2.1.7]$ 
That is in the default repositories so lets go ahead and install it (including the development files; and WTH, the documentation too).
[bamccaig@krypton sqsh-2.1.7]$ su -c 'yum install freetds freetds-devel freetds-doc'
Password:
[Hopefully YUM spits out happy here]
[bamccaig@krypton sqsh-2.1.7]$ 
With that out of the way, we're almost ready to build. Taking another look at the INSTALL file, we need to specify the base SYBASE directory in the SYBASE environment variable. I know what you're thinking. I know I was. I don't have Sybase! This doesn't apply to me! Not so fast. It turns out that it does apply to you (and I).[1] We instead need to point the SYBASE environment variable at the base directory for FreeTDS.

What this means is that $SYBASE/include and $SYBASE/lib should tell our build process where to find the necessary header and library files. So where are those files installed for FreeTDS? You can make an educated guess or you can ask rpm itself. I actually didn't know about this until now. It's hopefully going to make things a lot easier in the future.
[bamccaig@krypton sqsh-2.1.7]$ rpm -ql freetds freetds-devel | \
        egrep '/(include|lib)' | sort
/usr/include/bkpublic.h
/usr/include/cspublic.h
/usr/include/cstypes.h
/usr/include/ctpublic.h
/usr/include/sqldb.h
/usr/include/sqlfront.h
/usr/include/sybdb.h
/usr/include/syberror.h
/usr/include/sybfront.h
/usr/include/tdsconvert.h
/usr/include/tds.h
/usr/include/tds_sysdep_public_32.h
/usr/include/tds_sysdep_public.h
/usr/include/tdsver.h
/usr/lib/libct.so
/usr/lib/libct.so.4
/usr/lib/libct.so.4.0.0
/usr/lib/libsybdb.so
/usr/lib/libsybdb.so.5
/usr/lib/libsybdb.so.5.0.0
/usr/lib/libtds-0.82.so
/usr/lib/libtdsodbc.so
/usr/lib/libtdsodbc.so.0
/usr/lib/libtdsodbc.so.0.0.0
/usr/lib/libtds.so
[bamccaig@krypton sqsh-2.1.7]$ 
You can check the man or info page(s) for details, but essentially what we did is query the package manager to list the files that were installed; then we filtered out the results we weren't interested in (since we're only interested in files within an include or lib directory, I greped for that within the path). For good measure, I also sorted the results.

As you can see, FreeTDS is installed in /usr. No real surprise there. OK, so we need to set SYBASE to /usr.
[bamccaig@krypton sqsh-2.1.7]$ export SYBASE=/usr
[bamccaig@krypton sqsh-2.1.7]$ echo $SYBASE
/usr
[bamccaig@krypton sqsh-2.1.7]$ 
And now we're finally ready to start the build process. sqsh uses the "GNU auto-configuration package" as so many open source projects seem to do, so it uses the rather standard ./configure, make, make install build instructions. If you want readline support (and most people probably do) then you'll probably want to add the --with-readline option while configuring (you'll obviously need to have readline installed to do this though). Review the INSTALL file for more options.
[bamccaig@krypton sqsh-2.1.7]$ ./configure --with-readline
[Lots of output... Should be happy if your environment is complete...]
[bamccaig@krypton sqsh-2.1.7]$ make
[Everything is going good, the project compiles, but then all of a sudden...the linker is angry!]
gcc    -L/usr/lib  cmd_alias.o cmd_bcp.o cmd_buf.o cmd_connect.o cmd_do.o cmd_echo.o cmd_exit.o cmd_for.o cmd_func.o cmd_go.o cmd_help.o cmd_history.o cmd_if.o cmd_input.o cmd_jobs.o cmd_kill.o cmd_lock.o cmd_loop.o cmd_misc.o cmd_read.o cmd_reconnect.o cmd_redraw.o cmd_reset.o cmd_return.o cmd_rpc.o cmd_set.o cmd_shell.o cmd_show.o cmd_sleep.o cmd_wait.o cmd_warranty.o cmd_while.o var_ctlib.o var_date.o var_debug.o var_dsp.o var_hist.o var_misc.o var_passwd.o var_readline.o var_thresh.o dsp.o dsp_bcp.o dsp_csv.o dsp_conv.o dsp_desc.o dsp_horiz.o dsp_html.o dsp_meta.o dsp_none.o dsp_out.o dsp_pretty.o dsp_vert.o dsp_x.o sqsh_alias.o sqsh_args.o sqsh_avl.o sqsh_buf.o sqsh_cmd.o sqsh_compat.o sqsh_ctx.o sqsh_debug.o sqsh_env.o sqsh_error.o sqsh_expand.o sqsh_fd.o sqsh_filter.o sqsh_fork.o sqsh_func.o sqsh_getopt.o sqsh_global.o sqsh_history.o sqsh_init.o sqsh_job.o sqsh_readline.o sqsh_sig.o sqsh_sigcld.o sqsh_stdin.o sqsh_strchr.o sqsh_tok.o sqsh_varbuf.o sqsh_main.o -ldl -lm   -lreadline -lcurses  -o sqsh
cmd_bcp.o: In function `bcp_signal':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1164: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1166: undefined reference to `ct_cancel'
cmd_bcp.o: In function `cmd_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:337: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:877: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:894: undefined reference to `blk_done'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:898: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:901: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:904: undefined reference to `ct_con_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:910: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:924: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:942: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:947: undefined reference to `blk_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:952: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:953: undefined reference to `ct_con_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:957: undefined reference to `cs_loc_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:395: undefined reference to `ct_command'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:406: undefined reference to `ct_send'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:865: undefined reference to `blk_done'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:866: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:867: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:422: undefined reference to `ct_con_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:429: undefined reference to `ct_callback'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:438: undefined reference to `ct_callback'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:447: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:460: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:475: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:488: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:521: undefined reference to `ct_con_props'
cmd_bcp.o:/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:503: more undefined references to `ct_con_props' follow
cmd_bcp.o: In function `cmd_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:558: undefined reference to `cs_loc_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:539: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:566: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:600: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:582: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:617: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:630: undefined reference to `ct_connect'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:672: undefined reference to `blk_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:703: undefined reference to `blk_init'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:685: undefined reference to `blk_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:722: undefined reference to `ct_results'
cmd_bcp.o: In function `bcp_data_bind':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:973: undefined reference to `ct_res_info'
cmd_bcp.o: In function `cmd_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:818: undefined reference to `blk_done'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:837: undefined reference to `blk_done'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:797: undefined reference to `ct_fetch'
cmd_bcp.o: In function `bcp_data_xfer':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1060: undefined reference to `ct_fetch'
cmd_bcp.o: In function `cmd_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:769: undefined reference to `blk_done'
cmd_bcp.o: In function `bcp_data_bind':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1013: undefined reference to `ct_describe'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1031: undefined reference to `ct_bind'
cmd_bcp.o: In function `bcp_data_xfer':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1096: undefined reference to `blk_bind'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_bcp.c:1112: undefined reference to `blk_rowxfer'
cmd_connect.o: In function `cmd_connect':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:719: undefined reference to `ct_con_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1172: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1187: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1196: undefined reference to `ct_con_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1202: undefined reference to `cs_loc_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1208: undefined reference to `ct_exit'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1209: undefined reference to `cs_ctx_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:769: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:814: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:865: undefined reference to `cs_loc_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:869: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:908: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:925: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:949: undefined reference to `ct_connect'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:978: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:567: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:597: undefined reference to `cs_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:725: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:573: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:579: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:585: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:591: undefined reference to `cs_ctx_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:782: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:610: undefined reference to `ct_init'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:613: undefined reference to `ct_callback'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:621: undefined reference to `ct_callback'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:635: undefined reference to `ct_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:702: undefined reference to `ct_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:651: undefined reference to `ct_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:668: undefined reference to `ct_config'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:822: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:836: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:850: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:882: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1129: undefined reference to `cs_loc_drop'
cmd_connect.o: In function `check_opt_capability':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1251: undefined reference to `ct_capability'
cmd_connect.o: In function `cmd_connect':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1137: undefined reference to `ct_options'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:896: undefined reference to `cs_locale'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1037: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1089: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1094: undefined reference to `ct_command'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1115: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1105: undefined reference to `ct_send'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1111: undefined reference to `ct_results'
cmd_connect.o: In function `syb_client_cb':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1571: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_connect.c:1580: undefined reference to `ct_cancel'
cmd_do.o: In function `cmd_do_sigint_cancel':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:677: undefined reference to `ct_cancel'
cmd_do.o: In function `cmd_do_exec':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:304: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:312: undefined reference to `ct_command'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:327: undefined reference to `ct_cmd_drop'
cmd_do.o: In function `cmd_do':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:260: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:261: undefined reference to `ct_con_drop'
cmd_do.o: In function `cmd_do_exec':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:322: undefined reference to `ct_send'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:334: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:434: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:435: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:351: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:446: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:447: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:423: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:424: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:464: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:465: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:381: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:382: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:342: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:343: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_do.c:469: undefined reference to `ct_cmd_drop'
cmd_go.o: In function `cmd_go':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_go.c:468: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_go.c:475: undefined reference to `ct_command'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_go.c:511: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_go.c:481: undefined reference to `ct_cmd_drop'
cmd_reconnect.o: In function `cmd_reconnect':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_reconnect.c:58: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_reconnect.c:59: undefined reference to `ct_con_drop'
cmd_rpc.o: In function `rpc_param':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:559: undefined reference to `cs_convert'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:573: undefined reference to `ct_param'
cmd_rpc.o: In function `cmd_rpc':
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:228: undefined reference to `ct_cmd_alloc'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:441: undefined reference to `ct_cancel'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:476: undefined reference to `ct_cmd_drop'
/home/bamccaig/src/sqsh-2.1.7/src/cmd_rpc.c:279: undefined reference to `ct_command'
var_ctlib.o: In function `var_set_interfaces':
/home/bamccaig/src/sqsh-2.1.7/src/var_ctlib.c:82: undefined reference to `ct_config'
dsp.o: In function `dsp_cmd':
/home/bamccaig/src/sqsh-2.1.7/src/dsp.c:140: undefined reference to `ct_cmd_props'
/home/bamccaig/src/sqsh-2.1.7/src/dsp.c:181: undefined reference to `ct_send'
/home/bamccaig/src/sqsh-2.1.7/src/dsp.c:248: undefined reference to `ct_cancel'
dsp.o: In function `dsp_signal':
/home/bamccaig/src/sqsh-2.1.7/src/dsp.c:770: undefined reference to `ct_cancel'
dsp_bcp.o: In function `dsp_bcp':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_bcp.c:65: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_bcp.c:94: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_bcp.c:76: undefined reference to `ct_fetch'
dsp_csv.o: In function `dsp_csv':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_csv.c:66: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_csv.c:95: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_csv.c:77: undefined reference to `ct_fetch'
dsp_conv.o: In function `dsp_type_len':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_conv.c:505: undefined reference to `cs_convert'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_conv.c:520: undefined reference to `cs_convert'
dsp_conv.o: In function `dsp_datetime_conv':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_conv.c:361: undefined reference to `cs_dt_crack'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_conv.c:341: undefined reference to `cs_convert'
dsp_desc.o: In function `dsp_desc_fetch':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:434: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:553: undefined reference to `cs_convert'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:498: undefined reference to `cs_convert'
dsp_desc.o: In function `dsp_desc_bind':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:86: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:186: undefined reference to `ct_describe'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:369: undefined reference to `ct_bind'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:296: undefined reference to `ct_bind'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:392: undefined reference to `ct_compute_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:406: undefined reference to `ct_compute_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:136: undefined reference to `ct_compute_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_desc.c:163: undefined reference to `ct_compute_info'
dsp_horiz.o: In function `dsp_horiz':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:97: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:140: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:154: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:163: undefined reference to `ct_get_data'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_horiz.c:217: undefined reference to `ct_fetch'
dsp_html.o: In function `dsp_html':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:84: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:115: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:135: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:144: undefined reference to `ct_get_data'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_html.c:187: undefined reference to `ct_fetch'
dsp_meta.o: In function `dsp_meta_int_prop':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:613: undefined reference to `ct_res_info'
dsp_meta.o: In function `dsp_meta_transtate':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:640: undefined reference to `ct_res_info'
dsp_meta.o: In function `dsp_meta_desc':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:206: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:216: undefined reference to `ct_describe'
dsp_meta.o: In function `dsp_meta_fetch':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:695: undefined reference to `ct_fetch'
dsp_meta.o: In function `dsp_meta':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:65: undefined reference to `ct_results'
dsp_meta.o: In function `dsp_meta_bool_prop':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_meta.c:584: undefined reference to `ct_res_info'
dsp_none.o: In function `dsp_none':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_none.c:54: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_none.c:65: undefined reference to `ct_fetch'
dsp_pretty.o: In function `dsp_pretty':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:87: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:127: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:141: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:154: undefined reference to `ct_get_data'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_pretty.c:208: undefined reference to `ct_fetch'
dsp_vert.o: In function `dsp_vert':
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:79: undefined reference to `ct_results'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:106: undefined reference to `ct_res_info'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:121: undefined reference to `ct_fetch'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:127: undefined reference to `ct_get_data'
/home/bamccaig/src/sqsh-2.1.7/src/dsp_vert.c:305: undefined reference to `ct_fetch'
sqsh_ctx.o: In function `sqsh_ctx_pop':
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_ctx.c:104: undefined reference to `ct_con_props'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_ctx.c:113: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_ctx.c:116: undefined reference to `ct_con_drop'
sqsh_init.o: In function `sqsh_exit':
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_init.c:315: undefined reference to `ct_close'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_init.c:316: undefined reference to `ct_con_drop'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_init.c:322: undefined reference to `ct_exit'
/home/bamccaig/src/sqsh-2.1.7/src/sqsh_init.c:337: undefined reference to `cs_ctx_drop'
collect2: ld returned 1 exit status
make[1]: *** [sqsh] Error 1
make[1]: Leaving directory `/home/bamccaig/src/sqsh-2.1.7/src'
make: *** [build-subdirs] Error 2
[bamccaig@krypton sqsh-2.1.7]$ 
VERY angry. Obviously, we're missing a library. ct_blah suggested to me that it was one or more Sybase CT-Library libraries that it couldn't find. Since we're using FreeTDS in place of it, I went back to our friend rpm and with the help of grep and sed got the linking options that I was looking for.
[bamccaig@krypton sqsh-2.1.7]$ rpm -ql freetds freetds-devel | \
        grep '^/usr/lib/' | \
        sed -re 's#/usr/lib/lib##' -e 's/([^\.-]*).*/\1/' -e 's/(.*)/-l\1/' | \
        sort | xargs
-lct -lct -lct -lsybdb -lsybdb -lsybdb -ltds -ltds -ltdsodbc -ltdsodbc -ltdsodbc
[bamccaig@krypton sqsh-2.1.7]$ 
For those of you that don't speak UNIX so well yet, that basically means:
  1. List all the files that were installed by the freetds and freetds-devel packages (it almost certainly would suffice to just check freetds-devel, but it doesn't hurt to include freetds in the search).
  2. Filter out all options that don't begin with /usr/lib (since we're looking for libraries and we know they are installed in /usr/lib).
  3. Remove the directory (/usr/lib) and prefix (lib) from each library (most shared libraries are named something like libfoo.so).
  4. Remove any trailing characters (characters that are after a dot or dash, since most shared libraries have version information appended to their name, resulting in something like libfoo.so.1.2.3, for example; those are often symlinked by friendlier names, however, and the linker will do the work of figuring out what our pretty names mean).
  5. Prefix each library with -l, since that is the option that GCC uses to specify libraries.
  6. Finally, sort them for easy reading and join them onto a single line.
Reviewing the command line in the above error message, we can see that sqsh isn't being linked with any of them! That has to be problem! The h4x solution then is to edit the Makefile in question and add those to the command.

Upon opening that file, I found a handy SYBASE_LIBS variable that seemed rather relevant. Above it is even a handy comment suggesting that users of some platforms might need to manually edit that line. Open that file in your favorite editor...
[bamccaig@krypton sqsh-2.1.7]$ vim src/Makefile
...and replace that line with the results from our earlier query:
SYBASE_LIBS = -lct -lct -lct -lsybdb -lsybdb -lsybdb -ltds -ltds -ltdsodbc -ltdsodbc -ltdsodbc
Save the file and try to build again.
[bamccaig@krypton sqsh-2.1.7]$ make
[Now it should almost certainly be happy unless the world just hates you...]
[bamccaig@krypton sqsh-2.1.7]$ su -c 'make install && make install.man'
Password:
[Installation should go happily too]
[bamccaig@krypton sqsh-2.1.7]$ 
If you can run sqsh now (for example, sqsh --help to test) then the build and installation should have been successful! Congratulations! Now you (and I) just have to learn to use it. :-X

To get you started, the following allows me to connect to the SQL Server located on our LAN, running on a Windows server (**shudder**).
[bamccaig@krypton ~]$ sqsh -S <server_ip> -U <username>
[Program information]
Password: 
[I am now inside of a sqsh shell!]
> quit
[bamccaig@krypton ~]$ 
Obviously, not the most convenient way to connect, typing the server, username, and password (!) each time. Surely there must be handy configuration options to make things smoother. For now, I have created shortcut scripts (readable/writeable/executable only by me) that do the connecting for me to save me the trouble.
References

1. Thanks to the maintainer for clearing this up via E-mail.

2010-05-31

Wasabi - Apparently ASP/VBScript Isn't Evil Enough

I've been looking for "project management" software (I know, yuck) for some time. Mainly because I want to incorporate bug/issue tracking, discussion boards, and wikis into j0rb (and in personal projects, etc.). Firstly, it's very difficult to keep track of bugs by hand and often you forget about them before you fix them. When it comes to software development there's just too many issues, whether bugs, design decisions, or potential hiccups; to keep track of. I also want some persistent medium to discuss software design with colleagues. At j0rb we currently discuss things either through instant messaging or we talk in person.

The problem with instant messaging is that there's no organization to it. If you forget what was discussed or what conclusions everyone came to then you have to search through chat logs and hope to find it. If you do find it, you have to sort through intertwined discussions to find the relevant material. It's also hard to have meaningful discussions because instant messengers are generally designed to send single sentences or short paragraphs. It's hard to communicate code ideas or complex ideas that require more space to type. The problem with talking in person is that there's no log at all. There's no searching it to refresh your memory, etc.

Open source projects have gotten by with mailing lists for a long time and it seems the most experienced still prefer them to discussion boards. I figured there must be good reason for this so I've personally taken a liking to them. However, I don't think mailing lists would work well at j0rb. It's a Windows-based shop and I generally hate Windows-based mail clients for good reason. I generally use Gmail's Web interface for personal mail, which works quite well for most E-mail needs, but I wouldn't be allowed to discuss company topics through a remotely hosted service. At least, not a free one with no guarantees about security or privacy.

I'd be happy to try mutt from Cygwin against a locally hosted server, but odds are that my colleagues would be sending HTML E-mails and wouldn't understand why that was a problem (nor why I'd choose to use a plain-text client). For these reasons, I don't think a mailing list would work particularly well at j0rb. Colleague incompatibility. :P A discussion board seems to be the next best thing (with the advantage of edits to correct mistakes, etc.).

Wikis are great for keeping track of a growing and dynamic knowledge base of information. I think they'd work well for documenting certain gotchas discovered in languages, APIs, and platforms, etc.; as well as our own software's gotchas.

Anyway, I've recently been reading Joel on Software. There are a lot of really good articles that make a lot of sense. It seems Joel Spolsky is very experienced in project management and the like. The allure of fully defined specifications and feasible schedules got me interested in taking a look at FogBugz, some project management software developed by Joel's company, Fog Creek Software. I generally avoid commercial software, since as a general rule it's garbage, but I am willing to appreciate some commercial offerings (everything from Microsoft not being among them *cough*).

Anyway, it sounded like FogBugz was a pretty complete solution and I had hoped that it was better than what we have now (which can't do any of the things I mentioned above). I suggested it as an option to management, but the price seems kind of steep. Fortunately, they offer the hosted service for free to individuals so I decided to sign up for personal projects so I could get a look at just what it offered. I was pretty disappointed. At least at first glance, the UI seems rather bloated and disorganized. I'm having a hard time trying to figure out where different types of information are organized. It seems all it really tracks are "cases", which can be bugs, issues, or what have you. Then they use "filters" to determine which cases you see, by tons of criteria. This seems to be the way to show cases for a particular project. It seems awkward that way, but maybe that's just because I'm not used to it.

I decided to ask the Interwebz what it thought of FogBugz. I started where I usually start: at Wikipedia. It was there that I discovered, much to my dismay, Thistle and Wasabi.

It seems that FogBugz was originally written in Classic ASP/VBScript. I can sympathize because our main software project is mostly written in the same. What appears like an OK language from the surface is surely not. Visual Basic is bad enough, but VBScript is like a stripped down Visual Basic with half of the features missing or mutilated. Those that have worked with VBScript for any considerable period of time know that it is the decay-er of sanity. It has numerous limitations and has a weak "standard library" and most non-trivial functionality comes from server "components" that do not appear to be native VBScript at all (I've always assumed they were DLLs written in C or C++, but I suppose it's possible that they're actually written in pure evil). VBScript can't do much without these components. You apparently need these components for things like handling file uploads or connecting to databases. Things that other languages, like PHP, Python, and Perl can do "themselves". Probably because those languages are "open".

Apparently, Fog Creek Software wanted FogBugz to run on Linux servers, but ASP/VBScript is (officially) an abomination of IIS and the Windows operating system. I think at this point most would realize their mistake (developing software with ASP in the first place) and work towards correcting it by rewriting the application in a better language (and one that was cross-platform), but it seems that Fog Creek decided instead to develop a "compiler" (converter, or what ever you want to call it) that could convert ASP into PHP so that they could run their application on Linux boxes without having to completely rewrite it.

Now I don't know how many lines of code FogBugz was at the time or how complicated this "compiler", dubbed Thistle, was to write (it's over my head, I know that); but I know that it only took a few days of maintaining an ASP/VBScript application for me to begin begging almost daily to rewrite the application in something more sane. They didn't stop there, however. They apparently realized that VBScript combined with Thistle was too limited in what it could do (light bulb, anyone?). They missed their second opportunity to change platforms and instead wrote another "compiler" that extends VBScript's functionality, adding modern features that probably should have been there in the beginning, and spits out either PHP or .NET. They call this monstrosity Wasabi. All of that trouble to extend evil that never should have existed in the first place. I can't imagine that developing Thistle and Wasabi was faster than just rewriting their application in a cross-platform language, preferably an open one that will be around for a while, and leaving it at that.

Apparently they keep both of these tools internal. So not only did they create such evils, encouraging the persistence of VBScript-ness, but they didn't even release it to the world for them to benefit (free or otherwise). WTF.

I'm somehow less enthusiastic about FogBugz now..

2010-05-18

Source Code Management -- A Minor Success Story

In my quest to become an open source [UNIX-y] programmer (or hacker :-o), I try to learn about development tools and practices that make development easier. One such category of tool is source code management (SCM) tools (also called version control software). Essentially they keep track of the history of changes that you make to your code[1], as well as who made each change, when they made the change, and even comments from the author of the change explaining the change. This greatly helps developers manage a project because it allows us to keep track of what we've done and even what we're doing right now. It allows you to undo changes that you've made easily and share changes easily between developers. There are plenty of benefits so don't consider this a complete list.

For the record I've found Git to be the best SCM. At j0rb, however, we use Subversion. I myself learned about Subversion a year or so after college and taught myself to use it. Then I introduced j0rb to it and eventually managed to get it adopted. More recently (~past six months) I started using Git after watching Google Tech Talks on YouTube of Linus Torvalds and Randal Schwartz explaining why Git is the superior SCM and why everything else sucks. I didn't want to be "stupid and ugly" so I naturally adopted Git. Now I'd like to switch j0rb over to Git, but I mostly work with Windows-y, GUI-y programmers that are afraid of something like Git. Needless to say, they are refusing to change for now. We all know what that makes them.

Anyway, I've been working on something at j0rb for the past ~week. With Git, I could easily branch and/or commit locally as I go to separate changes, but with Subversion branching and merging is expensive and painful and there's no local repository to commit to. That's because everything is centralized so committing would put my changes in the central repository that everyone uses, which would mean that the application that my colleagues and I are working on would be broken until I'm done, preventing others from doing any work of their own.

Branching (Side Tracked)

One of the nicest things about Git's branching mechanism is that I don't need to go anywhere in the file system. When I change branches in Git, Git automatically makes my working directory that I'm already in the branch that I'm switching to (checking out, technically). With Subversion, a branch is really just a copy of some tree; it's a duplicate of a subtree. In order to work on the new branch, I need to check it out somewhere else on my file system (or I could remove my working directory and overwrite it with the new branch). To demonstrate:
# With Git, it's simple. Create and checkout a new branch named 'newbranch'
# based on the master branch at the current HEAD of the branch (last commit).
bamccaig@castopulence:~/src/example$ git status
# On branch master
nothing to commit (working directory clean)
bamccaig@castopulence:~/src/example$ git checkout -b newbranch master
Switched to a new branch "newbranch"
bamccaig@castopulence:~/src/example$ git status
# On branch newbranch
nothing to commit (working directory clean)
bamccaig@castopulence:~/src/example$
As can be seen above, with one simple command Git has created a branch and I'm already in it! I didn't have to do anything else. I can just start working. And Git is fast when it comes to branching so I didn't have to wait for anything. The new branch just points to the master branch so there was no need for expensive duplication of data. Subversion tells a different story, however:
# With Subversion, it's quite painful and it's also pretty slow. Create
# and checkout a new branch named 'newbranch' based on the repository
# trunk in the HEAD revision. Note that in my experience it's best to do
# branching in Subversion server-side. At least if you ever intend to
# merge back into the original branch...
bamccaig@castopulence:~/src/example/trunk$ svn cp -m 'Example...' \
        file:///home/bamccaig/src/example.repo/trunk \
        file:///home/bamccaig/src/example.repo/branches/newbranch

Committed revision 2.
bamccaig@castopulence:~/src/example/trunk$ svn up .. && \
        cd ../branches/newbranch
A    ../branches/newbranch
A    ../branches/newbranch/foo
A    ../branches/newbranch/bar
A    ../branches/newbranch/baz
Updated to revision 2.
bamccaig@castopulence:~/src/example/branches/newbranch$ 
Notice that Subversion basically requires me to not only type out a semi-lengthy URL (twice!) and download another complete copy of the original branch (in this case branches/newbranch, which is a copy of trunk), but also requires me to move around in the file system. In this simple example, I had the entire tree in my working copy (from the root of the repository, including branches, tags, and trunk). Often though you aren't interested in all of the many branches and tags that exist so you'll only checkout the branch(es) that you're interested in. In that case, you have to type out a checkout command with what's probably a semi-lengthy URL and then type out a command to change to the new branch's working directory. In short, branching in Subversion is just no fun. And don't get me started on merging... :'(

Back To The Story

OK, so here I am with a lot of changes to my working copy (give or take, 15 added or modified files). I come back in to work on Monday after the weekend and start working on a separate, though related project. When I finally get that done in mid-afternoon I get back to my original project, rebuild it and run it (something I generally do to get an idea of what state things are in and remind myself what I was working on last; again, Subversion doesn't help much when there's 15 added or modified files). To my dismay, this ASP.NET project throws a StackOverflowException immediately upon launching in Visual Studio's development Web server, which subsequently "crashes" the server since there's really no recovering from that. Unfortunately, with a StackOverflowException, there is apparently no stack trace (something I discovered right then) because the stack[2] itself is in an unholy state. On to tracking down what was causing the problem. But how? A stack overflow usually means you're either calling too many nested functions (often a result of recursion) or you've allocated too much memory on the stack.

Here's where the SCM comes in handy (albeit, this particular SCM still comes up short). I tried looking at all of the changes I had made since my last commit to see if I could spot anything suspicious.
[bamccaig@j0rb:foo]$ svn diff | less -S
Unfortunately, nothing stood out. I had added some new LINQ to SQL entities to the project and added some code to work with them. Much of the new code was generated for me by Visual Studio. The code I had worked on didn't stand out as a culprit.

This is where having Cygwin installed, a UNIX-like environment for Windows, comes in handy[3]. I decided the most efficient way for me to find the problem was to undo the changes I had made, confirm that it worked, and then redo the changes bit by bit until I encountered the StackOverflowException. This way I would know where to look for problems: the last applied changes. UNIX and UNIX-like operating systems (and Cygwin, as mentioned above) have tools that make this easy. First, I generate patches[4] with the SCM, Subversion, and a little shell scripting.
[bamccaig@j0rb:foo]$ for f in `svn st | grep '^M' | \
        sed -r 's/M *(.*)/\1/'`;
do
    svn diff "$f" 1> "$f.patch" && svn revert "$f";
done
For every file, foo/bar, that was modified since the last commit, I get a file foo/bar.patch that stores the changes made. Then I undo those changes (svn revert). For added files, the changes are irrelevant because they're basically the entire file anyway so instead I just temporarily remove them from the Visual Studio project. I can easily get a list of which files to remove though using Subversion and the shell again.
[bamccaig@j0rb:foo]$ svn st | grep '^A'
With all the changes undone (there were no deleted files in my working copy) I was able to retest the code. Lo and behold, it runs fine now. This confirmed that it was indeed me that broke it (damn). That came as no surprise though because I had been working on it for close to a week without problems and without pulling changes from the central repository. I was the only one making changes.

Now comes the fun part. Applying each patch one at a time and testing. It might sound tedious, but imagine how much more tedious it would be without the SCM or UNIX tools. To apply the patches from before, we use (surprise) the patch program.
[bamccaig@j0rb:foo]$ patch -p0 -ui path/to/the.patch
The -p0 option is required to leave paths in the patches alone. The default behavior for patch is to strip off the directory part, leaving only the filename. That only works if the file you're patching is in the current working directory. Mine are all over the working tree. The paths just happen to be correct from where I'm working though so the 0 says to strip nothing from them. The -u option tells patch that the patch file is in unified format, which is what Subversion's diff sub-command outputs by default. patch will likely figure this out on its own, but why waste the resources? :P The -i option specifies the patch file to use (which is followed by the path to the file).

Since I'm using Cygwin, with UNIX newlines[5] on a Windows-based project, however, patch is going to somewhat mangle my source code by filling my files with the wrong newline type. It's not a problem. The code will still work, but Subversion will see every line as a change, which will make reviewing the changes later on rather difficult. To fix this, we can use the unix2dos tool to convert them back. To save myself a lot of tedious typing, I created a bash function for all of this.
p() {
    patch="$1"
    file="`dirname $patch`/`basename $patch .patch`";
    patch -p0 -ui "$patch" && unix2dos "$file" && rm -i "$patch";
}
This way instead of typing out that long patch ... unix2dos command line, I can just say `p path/to/file.patch'. After each patch is applied, I'd confirm that it applied properly and remove the patch file to mark which ones I had done. Then I'd refresh the Visual Studio solution, rebuild it, and run it. If there was no StackOverflowException then I'd move on to the next patch. Once again, the SCM and UNIX tools allow me to easily track my progress. The following function listed which patches I had yet to apply:
[bamccaig@j0rb:foo]$ c() {
    svn st | grep patch;
}
I used that list to try to apply patches in order of dependencies to avoid unrelated problems.

The Suspense Is Killing Me!!!11

So what was the bug in the code causing the StackOverflowException? I have no clue... :( After going through the above, I seemingly have applied all patches and added all new files back to the project and it works fine. The only bug I encountered in the code had to do with a data-layer interface that I recently tried (again, I now realize) to get fancy with. Essentially, LINQ to SQL is handled through a DataContext class that is generated for you by Visual Studio. When you query for a set of entities, they are linked to the DataContext that retrieves them. When you make changes to them, the DataContext knows and uses those changes to generate SQL that ultimately updates the database. However, often times the changes are coming from the client, or take place over a few layers of the application. It's hard in some of these instances to maintain the original DataContext and the entities that are attached to it. It's particularly difficult when using serialization to communicate with a user agent. Fortunately, there is an interface to attach detached entities. Unfortunately, it requires both the modified object and the original unmodified object to know what record it's dealing with. This means that something as simple as saving an entity can require an entity-specific query and it just generally results in code bloat. To get around this, I created an interface that returns a LINQ Expression that identifies the entity and implement it for each entity. This way, the framework that I've developed can automatically fetch the original object, reducing the bloat to a simple call:

1
2
3
new LinqManager().Save(entity); // INSERT or UPDATE.
new LinqManager().Delete(entity); // DELETE.
 

Anyway, getting even more lazy, I also added an interface that returns the record identifier (all tables of this database have INT record identifiers). This allowed me to generate a typed LINQ Expression using generics and that interface.

1
2
3
4
5
6
public Expression<Func<IEntity, bool>> DefaultIdPredicate(
        IEntity e)
{
    return o => o.GetRecordId() == e.GetRecordId();
}
 

It sounds good and compiles happy, but it fails hard at run-time because LINQ to SQL can't translate it into SQL. During the process of applying the above patches I eventually ran into an exception whose message spoke of this. It was then that I remembered trying it previously (which is why the above method existed already), but it failed and I reverted, leaving the method intact for a future revelation. Instead I'm stuck resorting to ugly type-casting and explicit property access, which I manually re-coded throughout the project. That is the only fix that I made to the code as I applied the changes.

I can only hope that was the problem, though I'm not sure how that could cause a StackOverflowException, and be thankful that I had an SCM, even a poor excuse for one, and a UNIX-like environment to help me through this mess..

References
1. Source code management (version control software) tools aren't limited to tracking source code. They can actually track changes to any set of files (though it may depend on the particular tool), but as a general rule they don't work as well with binary files as they do with text files.

2. If you're unfamiliar with the call stack or stack vs. heap then ask Wikipedia. I'm normally happy to explain it, but I feel exceptionally lazy right now. I'll give you some hints though.

3. Though not as handy as running a UNIX-like operating system, such as Linux, would be. Unfortunately, I'm stuck with Windows at j0rb, but I digress..

4. Patches are essentially instructions for how to change a file automatically. They show the difference between two files, which can be used to automatically modify the original and produce the new one. I feel like I'm doing a horrible job explaining this today so I'm trying to refer to material that will do a better job explaining than I can right now. See here.

5. http://en.wikipedia.org/wiki/Newline.

2010-05-05

.NET + XPath + Namespaces (Conclusion)

In my last post, I discussed how XML namespaces were interfering with XPath expressions that were being used by an application to map XML data. Thanks to kind people on #xml on irc.freenode.net I finally made sense of it.

Within an XSLT document, which you of course know is an XML document, XPath expressions can apparently use the namespaces defined in the XSLT document (I assume then that they can't use namespaces defined in the transforming XML; confirmed). I think that means that if I have the following XML document:

1
2
3
4
5
6
7
8
<?xml version="1.0" encoding="utf-8"?>
<root>
    <level1 xmlns:foo="http://www.bamccaig.com/foo">
        <level2>Text</level2>
    </level1>
</root>
 
 

Then I could use an XSLT document like this to transform it:

1
2
3
4
5
6
7
8
9
10
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
        xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
        xmlns:bar="http://www.bamccaig.com/foo">
    <xsl:template match="//bar:level2">
        <xsl:value-of select="text()" />
    </xsl:template>
</xsl:stylesheet>
 
 

Note that the XPath expression used for the <xsl:template> element's match attribute...

//bar:level2

...uses the namespace prefix defined in the XSLT document (i.e., bar), not the namespace prefix defined in the XML document (i.e., foo).

What this helped me to realize is that it doesn't matter that the XML source document that I'm querying for data with XPath has child elements with default namespaces (i.e, no prefixes) because the prefixes can come from elsewhere. It was then that the .NET API began to make sense. In a way, my .NET is taking the place of the XSLT document, and it then makes sense to define namespaces in .NET. I tried to avoid this originally because it felt like hard-coding what should be soft-coded, but now it makes sense.

In my last post, I mentioned that the XPathNavigator.Evaluate method accepted an IXmlNamespaceResolver. While the XPathNavigator is itself an IXmlNamespaceResolver, it doesn't make sense that it could resolve our namespace (since the namespace in the given example is a default namespace, with no prefix).

So just like in the XSLT, we're going to define a namespace prefix for the namespace used in the XML. We can do that with an XmlNamespaceManager (which implements IXmlNamespaceResolver). Unfortunately, and it escapes me why, we need to pass an XmlNameTable to the XmlNamespaceManager's constructor. Both XmlDocument and XPathNavigator have NameTable properties of that type, but we've already concluded that those won't help us here. Why is it required? I don't know, but it is. With that information, we can finally resolve our namespace:

1
2
3
4
5
6
7
8
9
10
11
var nav = ...; // Our XPathNavigator.
var nsMan = new XmlNamespaceManager(nav.NameTable);
 
nsMan.AddNamespace("bar", "http://www.bamccaig.com/foo");
 
...
 
var it = nav.Evaluate("/root/bar:level1/bar:level2/text()",
        nsMan) as XPathNodeIterator;
 
 

There is one other thing you might be interested in. The data contract API that we're working with (from the post, LINQ to SQL + Serialization) adds a default namespace to the serialization based on the .NET namespace of the serialized type. This happens to be of the form[1],...

http://schemas.datacontract.org/2004/07/Clr.Namespace

...where the namespace is Clr.Namespace. If you don't like that then there are various ways to control it. You can pass a named parameter into the DataContract attribute, however, since in my case the types (and attributes) are generated for me, I don't have control of what parameters are passed to the attribute. The alternative solution that I preferred (because it was easy) was to set an assembly attribute.[1]

Essentially, I opened up the Properties/AssemblyInfo.cs file that Visual Studio had generated for me automatically and added the following line(s):

1
2
3
4
5
// Setting the data contract namespace for the assembly...
// See http://msdn.microsoft.com/en-us/library/ms731045.aspx.
[assembly: ContractNamespace(MyNamespace.ContractNamespace.Uri,
        ClrNamespace = "MyNamespace")]
 

Note: You'll also need a using directive for the System.Runtime.Serialization namespace, but that should be a given. ;) The latest Visual Studio should resolve that for you if you ask it to politely. Alternatively, you could fully-qualify the attribute.

Note that instead of using a string literal, which I could have done, I elsewhere declared a constant field that could be referenced (both here, and earlier, when creating the XmlNamespaceManager), which makes it a lot easier to change the namespace URI in the future.

And that's all there is to it. Let me know if this does or doesn't work for you, or if you have any questions, concerns, or advice. :P

References

  1. http://msdn.microsoft.com/en-us/library/ms731045.aspx

2010-04-30

.NET + XPath + Namespaces

Solved in a subsequent post.

This post is somewhat of a continuation to my previous post. I managed to get LINQ to SQL entities with custom properties serialized fine, however, the output contains XML namespaces which leads me to my next brick wall.

The application that I'm working on uses XPath expressions to match data within the serialization XML. XPath is namespace aware, which means that if you have a structure like this:

1
2
3
4
5
6
7
8
<?xml version="1.0" encoding="utf-8"?>
<root>
    <level1 xmlns="http://www.bamccaig.com/foo">
        <level2>Useful data!</level2>
    </level1>
</root>
 
 

Then the following XPath expression will NOT grab level2's text node:

/root/level1/level2/text()

The reason? level1 and level2 are in a different namespace than root. I have thus far failed to figure out how to work with this in .NET. In a past experience with this very problem not long ago, I was fortunate enough to be suggested a hack from the kind people on #xml on irc.freenode.net:

/*[local-name()='root']/*[local-name()='level1']/*[local-name()='level2']/text()

The asterisk (*) says to match all nodes (irrespective of namespaces) and the local-name() function returns the tag name (i.e., "root" for <root />). This works, or at least it did in the past, however, as you can imagine it's the Wrong Way(tm) and it's not particularly pretty either. The application I'm working on has a series of XPath expressions to maintain and I don't care to let this hack infect those.

Unfortunately, my attempts to solve this problem have so far failed. The XPathNavigator.Evaluate method accepts an IXmlNamespaceResolver, presumably for this very situation, but I'm unable to coerce it into resolving the namespace. :(

The most frustrating part of this is that the XML that it's [supposed to be] "navigating" is from the serialization process documented in my previous post. The XML was loaded with XmlDocument and the XmlNavigator was created with the XmlDocument's CreateNavigator method. The XmlDocument and XPathNavigator both have a NameTable property, which presumably maps out the namespaces within (it can be used to source an XmlNamespaceManager right in the constructor!), but somehow the XPathNavigator still fails to match to XPath expressions I'm feeding it. And get this: XPathNavigator implements the IXmlNamespaceResolver interface (and no, passing the navigator to the navigator does not work, I tried)!

I'm obviously missing something, but it's going to be nightmare to sort through all of the documentation, especially when I don't even know where to start...

Why, Microsoft?! Why?

Solved in a subsequent post.

LINQ to SQL + Serialization

So here I am for my first /real/ post. I recently faced off with serializing LINQ to SQL generated types with custom properties. "I" have done this before using WebMethods by simply returning them:

1
2
3
4
5
6
7
8
9
10
11
[WebMethod]
public EntityA GetEntityA(int id)
{
    using(var db = new MyDataContext())
    {
        return db.EntityAs.SingleOrDefault(
                o => o.id == id);
    }
}
 
 
Custom properties serialize fine that way and show up on the client-side (in JavaScript). The serialization is all handled for me by ASP.NET though. The way I originally was doing serialization is like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
var sb = new StringBuilder();
var xs = new XmlSerializerFactory(
        ).CreateSerializer(
        entityToBeSerialized.GetType());
 
using(var xw = XmlWriter.Create(sb))
{
    xs.Serialize(xw, entityToBeSerialized);
}
 
var serialized = sb.ToString();
 
 

This was working sufficiently for what I was using it for because the entities had no custom properties that were required in the serialization. Now I have entities that /do/ require their custom properties to be serialized. It took a little bit of Googling to find the solution (who has time to R all TFMs?). I found a solution that works (on MSDN, no less).

http://msdn.microsoft.com/en-us/library/bb546184.aspx

In short, you enable serialization for the LINQ to SQL model by opening it (the .dbml) in Visual Studio and setting the Serialization Mode (the only options I see are None and Unidirectional; guess which one to set it to). This tells LINQ to SQL to markup the generated types with serialization attributes. Next, I needed to markup my own custom properties with these same attributes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[DataMember]
public int SomeCustomProperty
{
    get
    {
        ...
    }
 
    set
    {
        ...
    }
}
 
 

Finally, I had to change the way I was serializing (my original way isn't aware of, or at least doesn't seem to care about, the attributes we've applied).

1
2
3
4
5
6
7
8
9
10
11
12
var dcs = new DataContractSerializer(
        entityToBeSerialized.GetType());
var sb = new StringBuilder();
 
using(var xw = XmlWriter.Create(sb))
{
    dcs.WriteObject(xw, entityToBeSerialized);
}
 
var serialized = sb.ToString();
 
 

This is what is currently working for me, at least. ;) Best practices and "'Ur Doin' It Wrong, n00b!" are welcomed. :P

** EDIT **

Note that the output is slightly different from the output of the original serialization classes, so code that you've written based on the old XML structure may need to be modified. I now have to remap a set of XPath expressions... Sigh. In truth, it's resulting in slightly better code though, IMHO.

2010-04-29

Introduction

This is my first blog post evar (well, outside of stupid social networking sites)! About a week ago I was trying to solve a code related problem at j0rb and realized that I could be writing blog posts about the problems and solutions I encounter. It seems more and more I'm finding the answers that I'm looking for in blogs. Often, however, I stumble across poorly written blogs that only partially answer questions or contain errors. I generally enjoy writing and try to be meticulous so hopefully I'll be able to bring some value to the "blogosphere" (I use that term loosely).

To introduce myself, I'm a computer programmer from Ontario, Canada. I live very near to the Michigan, USA border. I predominantly work with Microsoft-based Web technologies at j0rb, but I prefer free/libre/open source and text-based user interfaces! In my spare time, I play lots of video games and experiment with various software development concepts (platforms, languages, APIs, etc.).

I have self-diagnosed obsessive-compulsive personality disorder; which is just how I like to say that I try to be a perfectionist. ;) I'm obviously not infallible so I won't claim to never make mistakes. "Grammar Nazis" are welcome here. If you spot a mistake in my spelling or grammar I would appreciate if you'd point it out (try not to go overboard though). Also, I tend to edit posts a lot on forums (to correct mistakes) so expect the same behavior here.

:wq