"-//CS//Lyrics 1.0//EN" AKA Lyrics DTD/XML/XSL/XHTML/CSS

I know I hinted towards a technical blog post last time. I expected my next post to be about Java or Perl Web programming. Well, that one is coming, I suppose. I am making ground and eventually would like to write about it. I don't think I'm quite satisfied with my progress yet though (nor am I so dissatisfied that I need plea for help yet). This post is somewhat technical, but it's also somewhat personal.

I love to sing. I sing all the time. In the shower, in my car, at the computer, etc. Everywhere, basically. I love music and singing is one of the things that I like most about it. Or perhaps, more accurately, the lyrics are. I love lyrics. For me to like a song the lyrics had better be clever. You get bonus points if they're emotional or humorous and I can relate to them. I guess then it shouldn't be surprising that I also love to rap.

Indeed, it wouldn't be an exceptional day for somebody to notice me driving by them in Sault Ste. Marie with my stereo cranked, singing or rapping along. Whenever I hear a new song that I like I'll look up the lyrics online (in my younger days I would do it from the CD cover, but a wide screen monitor is a lot bigger and easier to read from in the dark). I'll then practice reciting them while listening to the song(s) on repeat. This could mean hearing a new or leaked song on YouTube and just starting the same video over and over again (shout out to YouTube Repeat!!). More often, though, it means that I just bought a new CD from one of my many preferred artists and am listening to it on repeat.

The Amarok media player is great for this because it supports a lyrics module that allows you to view the lyrics for the current song right there in the media player automatically! It's great for people like me that enjoy analyzing and learning the lyrics of a song, or even just singing them after 6 or 8 months have gone by and they aren't fresh in your mind anymore.

Often I find that the various lyrics sites online don't have 100% correct lyrics. Some people do a really bad job of transcribing them. I don't know if it's laziness or incompetence, but I personally find it really annoying when I'm trying to sing or rap along to a new song and I'm stumbling over the wrong lyrics on a page.

It seems that most user maintained lyrics sites online make it difficult to contribute though. You generally have to register with some random site, which means a whole new set of credentials to remember, and a whole new server with your personal information just waiting to be hacked and compromised. They also don't do a very good job of describing the lyrics or presenting them to the user. There's usually a mix of slang and unslang and it's never really clear when the transcriber is sure of a particular word or just making an [un]educated guess. You may be surprised just how often a word or two is a mystery in very popular songs.

This combined with my thirst for XML knowledge lead me to develop a simple XML document type for lyrics. It's a work in progress so nothing is feature-frozen yet. Currently I have a document type definition (DTD) to validate against, an XSL stylesheet to transform the lyric data into standards compliant XHTML 1.0 Strict, and a CSS stylesheet to accompany the XHTML output to prettify the output a bit (no guarantees that the CSS is standards compliant, unfortunately and the W3C's validator seems to think it is valid CSS 2.1 also!).

Currently I only have two songs transcribed. The first is We As Americans, by Eminem, from the Encore bonus CD. That was basically my initial test drive, if you will, for the format and stylesheets.

Two evenings ago I got to thinking about one of my favorite bands, Eisley, and wondering if they had released any new albums since I last checked. It turns out that indeed they had! The Valley was released back in March. I made it a plan to stop by the music store yesterday to look for it. Now Eisley is somehow still somewhat of a below-the-radar band, which is puzzling because they're so amazingly talented and creative and beautiful. In any case, a side effect of this is that music stores often don't stock them (some of them don't even have a listing for them!). I could rant all day about music stores and the music industry and how stupid things have gotten (see my previous post about the movie industry for a hint of what to expect), but I'll try to avoid doing that now and save it for another post.

Long story short, I went out to buy one CD and came home with seven...and none of them was the one that I went out for! "But, Brandon," I hear you wondering. "What about Eisley?"

Oh, I ordered it. Don't you worry. As expected, the music store didn't have it in stock. Unfortunately, it cost me $20 to order through the store. The night before I had seen that the online retailer linked from the band's Web site was only asking $10. Without knowing the shipping fees though it's hard to say if I'm any worse off. Not that I'd mind buying an Eisley CD for $20. In fact, I think I probably paid $20 for each of Room Noises and Combinations too. What bothers me is that Eisley will only see a tiny percentage of that, if any, and the crappy music stores and record companies that I'm avoiding ranting about will take most of the profit.

Moving on, the CDs that I actually came home with were as follows in no particular order: Theory of a Deadman (Theory of a Deadman), The Truth Is... (Theory of a Deadman), Taylor Swift (Taylor Swift), Greatest Hits (Guns N' Roses), 22 More Hits (George Strait), American Saturday Night (Brad Paisley), and This Is Country Music (Brad Paisley). I was also trying to buy or order The Spirit Room (Michelle Branch), but not only did the music store (the only one left in my city to my knowledge) not have any Michelle Branch CDs in stock, but her CDs weren't even available from their distributor peoples! Ugh.

So anyway, having bought all of these CDs, I then had to rip them into my computer so I could sync them with my iPod (I could also rant about how much Apple sucks, but there's a different time and post for that too). In doing so, I found myself listening to old and new music and singing along and by the time 2 AM rolled around I wasn't really tired. In fact, I couldn't seem to stop singing or rapping. I tried going to bed, but eventually just started to rap the lyrics to one of the songs I had written back in high school, and then decided that I should transcribe it and publish the lyrics on my site. That way I'd have a digital copy (it was quite a challenge to remember the words after so many years), since I had destroyed, lost, or abandoned all others copies. I eventually gave in and decided to do this immediately while it was fresh in my mind (both to do it and most of the lyrics).

So the second song I have transcribed is actually my own! It's called Wanna Be God. I separated it into a separate file to initially keep it off the radar and also to keep it separate from "real" songs. :) There are a few other songs that I wrote back in high school. One or two of them I might feel are worth publishing online also. We'll see if I ever get around to that.

In the meantime, please check out my XML document type, and accompanying DTD, XSLT, and CSS. And, of course, check out the lyrics to both songs:

• http://lyrics.castopulence.org/
  • http://lyrics.castopulence.org/lyrics.xml (This is the core document that I'll currently put all lyrics in, though eventually I might split them each into their own file)
  • http://lyrics.castopulence.org/lyrics.xsl
  • http://lyrics.castopulence.org/lyrics.dtd (https://www.castopulence.org/2011/09/lyrics)
  • http://lyrics.castopulence.org/lyrics.xhtml
  • http://lyrics.castopulence.org/lyrics.html
  • http://lyrics.castopulence.org/lyrics.css
• http://lyrics.castopulence.org/wanna_be_god
  • http://lyrics.castopulence.org/wanna_be_god.xml
  • http://lyrics.castopulence.org/wanna_be_god.xhtml
  • http://lyrics.castopulence.org/wanna_be_god.html

Eventually I'd like to implement a server-side backend to manage them (once I get either a Java or Perl Web environment setup to my satisfaction, of course ;), at which point I could potentially open the system up to open ID users and let the world help me to fix lyrics. I'm not yet certain where I stand with regards to copyright infringement though. Technically I am violating Eminem's (or his record company's) copyright by publishing the lyrics to his song without permission. I do have a disclaimer on the page to highlight the fact that I don't own the copyright... It seems harmless to me, but nevertheless, there is always the possibility that somebody will feel they are losing money as a result and try to convince a judge to make me reimburse them... Hopefully that doesn't happen... Honestly, I can't even figure out how I would contact Eminem or Interscope to ask for permission. Their Web sites are terrible. :P

If you are or represent a copyright owner and wish for me to have a work removed from my site then please kindly address an E-mail to me (bamccaig at castopulence dot org) detailing who you are and what relationship you have to the copyrighted work, and the copyright owner's wishes, and I will promptly add any disclaimers you want or, if necessary, lock that work up behind a secured connection for personal use only.

UNglobal State Programming Environment

A year or two ago I stumbled across a very useful Google Tech Talk on YouTube about global state and Singletons and why they're bad:



It makes a lot of sense and after letting it sink in and experimenting with dependency injection myself I can say that it results in much better designs, whether or not you're writing tests (the video is Java-centric and test-centric, but neither is necessary for it to apply). Thinking about it though popular programming environments do impose a lot of global state that is in my experience regularly accessed without a second thought. When you collaborate with other people it's hard to make them understand how global state and Singletons are bad when to them it's the easy and fast way to get their job done. It's my opinion that a large part of the problem is that programmers are just used to global state from their programming environments. What we need is for languages to be updated and newer languages to be done right in the first place. No more global state in the programming environment. Pass that state into the program's entry point. For example:

// C
int main(const system_t * const system, const int argc, const char * argv[])
{
    fprintf(system->stderr, "This was a triumph.\n");

    return 0;
}

// C++
int main(const System & system, const int argc, const char * const argv[])
{
    system.cerr << "This was a triumph." << std::endl;

    return 0;
}

// C#
private static int Main(System system, string[] args)
{
    system.Error.WriteLine("This was a triumph.");

    return 0;
}

/* Java (albeit, while they're at it, main should return an int in Java too) */
public static void main(System system, String [] args)
{
    system.err.println("This was a triumph.");

    system.exit(0);
}

Etc. Wouldn't that be a wonderful platform to develop on? I certainly think so.

Kernel-space Politics

I thought of an interesting analogy the other day during an IRC conversation that I thought I'd share with you. :) It was probably almost a week ago (Blogger was down for maintenance at the time) and I forget exactly what the discussion was about, but more or less just how the members of government exploit the people for their own personal gain. In defense of privatizing as much as possible, I argued that the government is a little bit like the kernel of an operating system. If you want the system to be secure and reliable then you need to move as much as possible into userland (i.e., outside of the kernel). This way the individual processes (services, businesses, etc.) can attempt to do wrong, but ultimately the authority (kernel AKA government) has the power to stop them. On the other hand, if these processes are occurring directly within the kernel then they have unrestricted access to the entire system and can do whatever they want, even if they happen to corrupt or crash the system in doing so.

Then again, with the world's major currencies under private banking monopoly control you have to ask yourself whether the government is actually the kernel or only the system drivers. :-X The government can only hope to keep the system secure and reliable if it is actually ring 0 and free of corruption itself.

.NET P/Invoke With Managed Callbacks

Update: See append at the bottom for an update.
Update2: Open sourced the bindings. See append2 at the bottom for an update.

I've finally thought of another topic to write about. Or, rather, the topic fell on my desk. Again. It is about using managed methods as callbacks in native code. I first learned about this months (a year?) ago when I was working on a ZBar.Processor proxy class for zbar-sharp (.NET bindings for the zbar bar code scanning library). This was my first real experience using P/Invoke.

The zbar_processor is effectively a high-level object to scan frames from a Web cam for bar codes and return the encoded data back to the calling application. The data is returned via a callback function. This seemed simple enough once I figured out how P/Invoke basically is done. I just wrote up a compatible method and passed it into the zbar API. While it did seem to work, the native library would eventually throw memory access violations.

It turns out that the problem was me (SHOCK). Or rather, my code. Or rather, missing code. See in .NET the equivalent of a function pointer is basically a delegate instance. This is effectively a method object (I imagine it to be similar to a functor in C++). In any case, calling the delegate instance is equivalent to calling the method. Even the this reference is preserved.

#digress Delegates In C#

A delegate type is declared like so:

namespace Application
{
    class Example
    {
        public delegate void foo(int bar);
    }
}

What this does is basically declare a delegate type named "foo" with no return value and a single int parameter. You can then create an instance of the delegate like so:

foo f = new foo(MyMethod);

Where MyMethod is a static or non-static method that matches the signature. The instantiation can also be implicit:

foo f = MyMethod;

For a more complete example:

using System;

namespace Application
{
    class Program
    {
        static void Main()
        {
            new Program().Run();
        }

        public void Run()
        {
            // Using a method, explicit instantiation.
            Example.foo f1 = new Example.foo(this.Foo);

            // Using a method, implicit instantiation.
            Example.foo f2 = this.Foo;

            // Using a lambda, explicit instantiation.
            Example.foo f3 = new Example.foo(
                    (int bar) => Console.WriteLine(bar));

            // Using a lambda, implicit instantiation.
            Example.foo f4 = (int bar) => Console.WriteLine(bar);

            f1(1);
            f2(2);
            f3(3);
            f4(4);
        }

        void Foo(int bar)
        {
            Console.WriteLine(bar);
        }
    }

    class Example
    {
        public delegate void foo(int bar);
    }
}

#enddigress

As I was saying before I so rudely interrupted me, I was missing a tiny little detail. In order to use one of these delegates as a callback in native code you'd first have to instantiate one. OK, first we actually need a native method to call, and a callback type.

namespace Application
{
    class Example
    {
        public class NativeMethods
        {
            public delegate void zbar_image_data_handler_t(
                    IntPtr image,
                    IntPtr userdata);

            [DllImport("libzbar0")]
            public static extern IntPtr zbar_processor_set_data_handler(
                    IntPtr processor,
                    zbar_image_data_handler_t handler,
                    IntPtr userdata);
        }
    }
}

Now in order to use it we simply do as above:

using System;

namespace Application
{
    class Program
    {
        static void Main()
        {
            new Program().Run();
        }

        public void Run()
        {
            IntPtr processor = IntPtr.Zero;

            /*
             * Use your imagination here. We would have had to call native
             * methods to create and initialize a zbar_processor struct.
             */
            
            var handler =
                    new Example.NativeMethods.zbar_image_data_handler_t(
                    this.ZBarProcessorImageDataHandler);

            Example.NativeMethods.zbar_processor_set_data_handler(
                    processor,
                    handler,
                    null);
        }

        void ZBarProcessorImageDataHandler(IntPtr image, IntPtr userdata)
        {
            Console.WriteLine("Bar code successfully scanned!");
        }
    }
}

Pretty straightforward, really, but there is a problem! Can you see it?! Not likely because it isn't even in the code. Remember that .NET is managed and garbage collected and delegates are basically objects. What this means is that when the garbage collector finds a delegate that is no longer referenced by anything it will eventually garbage collect it. This was something that certainly hadn't occurred to me when I was writing this code. It wasn't until I sought help online that somebody (it was many months ago, but I'm guessing in #csharp on freenode) pointed out that the delegate object could be destroyed by the time the zbar library attempted to call it. So what is the solution? We need to keep a reference to it around so it isn't destroyed. We can do that by simply creating a field to hold it.

...
    class Program
    {
        Example.NativeMethods.zbar_image_data_handler_t
                imageDataHandler_;

        ...

        public void Run()
        {
            IntPtr processor = IntPtr.Zero;

            /*
             * Use your imagination here. We would have had to call native
             * methods to create and initialize a zbar_processor struct.
             */
            
            this.imageDataHandler_ =
                    new Example.NativeMethods.zbar_image_data_handler_t(
                    this.ZBarProcessorImageDataHandler);

            Example.NativeMethods.zbar_processor_set_data_handler(
                    processor,
                    this.imageDataHandler_,
                    null);
        }

        ...
    }
...

At this point, after fixing a few other subtle bugs, the program ran fine without access violations! Unfortunately, the zbar_processor (at least, at the time) wasn't very reliable and we ultimately ended up implementing the Web cam interface ourselves and passing the scanned frames to zbar.

In any case, I've recently been writing .NET bindings for wkhtmltox, another native library. This one is for converting HTML/CSS Web pages to PDF documents. It works really well because it's based on the WebKit framework (hence, the "wk" in the name). So I forgot all about the lifetime of the callback delegates and ran into the same problem. Today I finally remembered and looked back at the zbar bindings to see what was wrong.

I've now fixed that by adding delegate fields to the appropriate proxy class, but unfortunately I'm STILL getting access violation exceptions. It's probably going to be some other subtle bug as a result of .NET speaking to native code. I haven't figured it out yet though.

If you have any ideas about what else would cause memory access violations in P/Invoke programs or have any questions about what I've shown then please leave a comment. :)

Append:

Seems there is more to the story than I thought. See the following thread for details:

http://www.gamedev.net/topic/270670-pinvoke-callback-access-violation/

The gist of it is that there is a System.Diagnostics.InteropServices.UnmanagedFunctionPointerAttribute attribute that can be used to mark up your callback delegates with P/Invoke attributes (for example, calling convention). I'm not sure why, but specifying CallingConvention.CDecl for my callback delegates seems to postpone or avoid the access violations. I don't really understand why because the functions of the library are being called with the default calling convention, which I think is CallingConvention.StdCall. Attempting to specify CallingConvention.CDecl for the native functions immediately causes the stack to become imbalanced/corrupted (according to the .NET runtime; I think I trust it in this case ;). It doesn't really make sense to me that the callbacks would use a different calling convention so I suspect that the different calling convention for the callbacks just coincidentally changes the way the memory is accessed to avoid violations... I don't know. I'm lost. :(

Append2:

I've turned the .NET bindings for wkhtmltox (wkhtmltopdf only) into an open source class library. :) You can find the code at its GitHub repository. It doesn't work properly yet (in fact, I've given up on it for now and am working on invoking the wkhtmltopdf.exe process instead), but I see no technical reason why it can't so I (or you) just need to figure out what's currently wrong. :)

Fedora Package Contributor for Allegro

It's been a long time since I've posted anything on here and I guess it's long overdue. I've either lacked a good topic to type about or was just too drained to do so. Well no longer!

I figured I would take a moment to announce that I've become a package contributor for the Fedora project. What does that mean? It just means that I'm currently contributing to the development or maintenance of one or more of the RPM packages for the project. In particular, I'm maintaining the Allegro 5 packages. What this means is that I took the Allegro 5 source code and turned it into RPM packages. The main repository that I'm using to track Allegro 5 package development is available on GitHub: http://github.com/bamccaig/allegro5-rpm. Don't confuse me though with the actual Allegro developers who deserve the real praise for the Allegro library!

There are plans for me to help out with or take over the Allegro 4 packages as well (which I think will include games and other related projects). I've just recently gotten Allegro 5 adopted into Fedora 13 and 14 (15 AKA Alpha is pending) and 15. AFAIK, Fedora is the first distribution to get Allegro 5 packages! So yay for that! I'll now be working on updating the Allegro 4 packages (allegro) to the latest stable version, 4.4.x.y.

I guess that makes me an "official" contributor to Fedora. So that's pretty exciting. I think so, at least. What are you waiting for? Go put my packages in action! You can learn how to here if you don't already know.

Since there isn't much else to say about that, I'll take a few moments to discuss a few of the things I learned about working with RPMs. RPMs are based on a .spec file that basically just describes the package in a text file: name, description, version, release, where to get the source code, the software license, build instructions, which files to install, what their permissions should be, etc. To turn a .spec file into an SRPM[1] or RPM you use the rpmbuild tool. Once installed you can easily query the package system for information about a package using rpm or yum. For example:

$ # Lets query the basic information about the core allegro5 package.
$ yum info allegro5
*snip*
Name        : allegro5
Arch        : i686
Version     : 5.0.0
Release     : 3.fc13
Size        : 994 k
Repo        : installed
From repo   : updates
Summary     : A game programming library
URL         : http://liballeg.org/
License     : zlib
Description : Allegro is a cross-platform library intended for use in computer games
            : and other types of multimedia programming. Allegro 5 is the latest major
            : revision of the library, designed to take advantage of modern hardware
            : (e.g. hardware acceleration using 3D cards) and operating systems.
            : Although it is not backwards compatible with earlier versions, it still
            : occupies the same niche and retains a familiar style.

$ # Lets query the changelog of the core allegro5 package.
$ rpm -q --changelog allegro5
* Wed Mar 09 2011 Brandon McCaig  5.0.0-3
- Adding file permissions to subpackages.
- Moving devel files (namely .so symlinks) to devel packages.
- Added %doc section proper; readmes, changes, license, etc.
- Fixed SF.net URI.
- Modified BuildRequires.
- Added main devel dependency to subpackage devels.
- Replaced many al_*.3* manpage files with a glob.
- Replaced many header files with directory and %exclude macros.
- Added allegro5.cfg file under /etc/allegro5rc.

* Fri Mar 04 2011 Brandon McCaig  5.0.0-2
- Merged primitives addon packages into core packages.
- Merged memfile addon packages into core packages.
- Merged "main" addon packages into core packages.
- Merged font packages into core packages.
- Merged color packages into core packages.
- Merged doc package into the devel package.
- Fixed spelling mistakes.
- Removed explicit library dependencies.

* Fri Feb 25 2011 Brandon McCaig  5.0.0-1
- Initial version.

$ 

I was actually quite happy to learn just how easy it is to create RPMs that work. RPM seems to be very smart and figures out a lot of the dependencies itself. Contributing simple packages really isn't very difficult if you already know a thing or two about how to build them from source (programming experience helps too). So if you're a programmer that uses Fedora (or any other Linux distribution) and wants to help out by becoming a packager then I suggest you look into it. It's not as difficult as you might think.

Thanks to my sponsor[2] for guiding me through the process and helping me out when I got stuck.

References

[1] An SRPM is a source RPM. It basically is just an RPM that contains the .spec file, source tarball, and any patches that might be bundled. The SRPM is an intermediary stage between the source code and .spec and the fully-fledged RPM.
[2] I'll refrain from naming him without his permission to do so, just in case, but you can probably find out who it is easily enough by checking out the fedora bugzilla system if you care. :) He knows who he is. :)