Kernel-space Politics

I thought of an interesting analogy the other day during an IRC conversation that I thought I'd share with you. :) It was probably almost a week ago (Blogger was down for maintenance at the time) and I forget exactly what the discussion was about, but more or less just how the members of government exploit the people for their own personal gain. In defense of privatizing as much as possible, I argued that the government is a little bit like the kernel of an operating system. If you want the system to be secure and reliable then you need to move as much as possible into userland (i.e., outside of the kernel). This way the individual processes (services, businesses, etc.) can attempt to do wrong, but ultimately the authority (kernel AKA government) has the power to stop them. On the other hand, if these processes are occurring directly within the kernel then they have unrestricted access to the entire system and can do whatever they want, even if they happen to corrupt or crash the system in doing so.

Then again, with the world's major currencies under private banking monopoly control you have to ask yourself whether the government is actually the kernel or only the system drivers. :-X The government can only hope to keep the system secure and reliable if it is actually ring 0 and free of corruption itself.

.NET P/Invoke With Managed Callbacks

Update: See append at the bottom for an update.
Update2: Open sourced the bindings. See append2 at the bottom for an update.

I've finally thought of another topic to write about. Or, rather, the topic fell on my desk. Again. It is about using managed methods as callbacks in native code. I first learned about this months (a year?) ago when I was working on a ZBar.Processor proxy class for zbar-sharp (.NET bindings for the zbar bar code scanning library). This was my first real experience using P/Invoke.

The zbar_processor is effectively a high-level object to scan frames from a Web cam for bar codes and return the encoded data back to the calling application. The data is returned via a callback function. This seemed simple enough once I figured out how P/Invoke basically is done. I just wrote up a compatible method and passed it into the zbar API. While it did seem to work, the native library would eventually throw memory access violations.

It turns out that the problem was me (SHOCK). Or rather, my code. Or rather, missing code. See in .NET the equivalent of a function pointer is basically a delegate instance. This is effectively a method object (I imagine it to be similar to a functor in C++). In any case, calling the delegate instance is equivalent to calling the method. Even the this reference is preserved.

#digress Delegates In C#

A delegate type is declared like so:

namespace Application
{
    class Example
    {
        public delegate void foo(int bar);
    }
}

What this does is basically declare a delegate type named "foo" with no return value and a single int parameter. You can then create an instance of the delegate like so:

foo f = new foo(MyMethod);

Where MyMethod is a static or non-static method that matches the signature. The instantiation can also be implicit:

foo f = MyMethod;

For a more complete example:

using System;

namespace Application
{
    class Program
    {
        static void Main()
        {
            new Program().Run();
        }

        public void Run()
        {
            // Using a method, explicit instantiation.
            Example.foo f1 = new Example.foo(this.Foo);

            // Using a method, implicit instantiation.
            Example.foo f2 = this.Foo;

            // Using a lambda, explicit instantiation.
            Example.foo f3 = new Example.foo(
                    (int bar) => Console.WriteLine(bar));

            // Using a lambda, implicit instantiation.
            Example.foo f4 = (int bar) => Console.WriteLine(bar);

            f1(1);
            f2(2);
            f3(3);
            f4(4);
        }

        void Foo(int bar)
        {
            Console.WriteLine(bar);
        }
    }

    class Example
    {
        public delegate void foo(int bar);
    }
}

#enddigress

As I was saying before I so rudely interrupted me, I was missing a tiny little detail. In order to use one of these delegates as a callback in native code you'd first have to instantiate one. OK, first we actually need a native method to call, and a callback type.

namespace Application
{
    class Example
    {
        public class NativeMethods
        {
            public delegate void zbar_image_data_handler_t(
                    IntPtr image,
                    IntPtr userdata);

            [DllImport("libzbar0")]
            public static extern IntPtr zbar_processor_set_data_handler(
                    IntPtr processor,
                    zbar_image_data_handler_t handler,
                    IntPtr userdata);
        }
    }
}

Now in order to use it we simply do as above:

using System;

namespace Application
{
    class Program
    {
        static void Main()
        {
            new Program().Run();
        }

        public void Run()
        {
            IntPtr processor = IntPtr.Zero;

            /*
             * Use your imagination here. We would have had to call native
             * methods to create and initialize a zbar_processor struct.
             */
            
            var handler =
                    new Example.NativeMethods.zbar_image_data_handler_t(
                    this.ZBarProcessorImageDataHandler);

            Example.NativeMethods.zbar_processor_set_data_handler(
                    processor,
                    handler,
                    null);
        }

        void ZBarProcessorImageDataHandler(IntPtr image, IntPtr userdata)
        {
            Console.WriteLine("Bar code successfully scanned!");
        }
    }
}

Pretty straightforward, really, but there is a problem! Can you see it?! Not likely because it isn't even in the code. Remember that .NET is managed and garbage collected and delegates are basically objects. What this means is that when the garbage collector finds a delegate that is no longer referenced by anything it will eventually garbage collect it. This was something that certainly hadn't occurred to me when I was writing this code. It wasn't until I sought help online that somebody (it was many months ago, but I'm guessing in #csharp on freenode) pointed out that the delegate object could be destroyed by the time the zbar library attempted to call it. So what is the solution? We need to keep a reference to it around so it isn't destroyed. We can do that by simply creating a field to hold it.

...
    class Program
    {
        Example.NativeMethods.zbar_image_data_handler_t
                imageDataHandler_;

        ...

        public void Run()
        {
            IntPtr processor = IntPtr.Zero;

            /*
             * Use your imagination here. We would have had to call native
             * methods to create and initialize a zbar_processor struct.
             */
            
            this.imageDataHandler_ =
                    new Example.NativeMethods.zbar_image_data_handler_t(
                    this.ZBarProcessorImageDataHandler);

            Example.NativeMethods.zbar_processor_set_data_handler(
                    processor,
                    this.imageDataHandler_,
                    null);
        }

        ...
    }
...

At this point, after fixing a few other subtle bugs, the program ran fine without access violations! Unfortunately, the zbar_processor (at least, at the time) wasn't very reliable and we ultimately ended up implementing the Web cam interface ourselves and passing the scanned frames to zbar.

In any case, I've recently been writing .NET bindings for wkhtmltox, another native library. This one is for converting HTML/CSS Web pages to PDF documents. It works really well because it's based on the WebKit framework (hence, the "wk" in the name). So I forgot all about the lifetime of the callback delegates and ran into the same problem. Today I finally remembered and looked back at the zbar bindings to see what was wrong.

I've now fixed that by adding delegate fields to the appropriate proxy class, but unfortunately I'm STILL getting access violation exceptions. It's probably going to be some other subtle bug as a result of .NET speaking to native code. I haven't figured it out yet though.

If you have any ideas about what else would cause memory access violations in P/Invoke programs or have any questions about what I've shown then please leave a comment. :)

Append:

Seems there is more to the story than I thought. See the following thread for details:

http://www.gamedev.net/topic/270670-pinvoke-callback-access-violation/

The gist of it is that there is a System.Diagnostics.InteropServices.UnmanagedFunctionPointerAttribute attribute that can be used to mark up your callback delegates with P/Invoke attributes (for example, calling convention). I'm not sure why, but specifying CallingConvention.CDecl for my callback delegates seems to postpone or avoid the access violations. I don't really understand why because the functions of the library are being called with the default calling convention, which I think is CallingConvention.StdCall. Attempting to specify CallingConvention.CDecl for the native functions immediately causes the stack to become imbalanced/corrupted (according to the .NET runtime; I think I trust it in this case ;). It doesn't really make sense to me that the callbacks would use a different calling convention so I suspect that the different calling convention for the callbacks just coincidentally changes the way the memory is accessed to avoid violations... I don't know. I'm lost. :(

Append2:

I've turned the .NET bindings for wkhtmltox (wkhtmltopdf only) into an open source class library. :) You can find the code at its GitHub repository. It doesn't work properly yet (in fact, I've given up on it for now and am working on invoking the wkhtmltopdf.exe process instead), but I see no technical reason why it can't so I (or you) just need to figure out what's currently wrong. :)