CREATING A MAPPER The Python25Mapper has several constructors, but in normal use we just pass in the path to the stub 'python25.dll'. ------------------------------------------------------------------------------------------------ mapper = Python25Mapper("build\\python25.dll"); ------------------------------------------------------------------------------------------------ When we construct a mapper like this, the following things happen: * The mapper creates a PythonEngine and an allocator; once the engine is available, the mapper creates a new module and executes DISPATCHER_MODULE_CODE (found in 'src/Python25Mapper_snippets.cs', discussed in some details later), then extracts a reference to the Dispatcher class for later use. * Assuming the stub path was passed in, the mapper also creates a StubReference, which loads the stub 'python25.dll' (which impersonates the real 'python25.dll') and makes it ready by calling its 'init' function, passing in a pair of function pointers: an 'address_getter' and a 'data_setter'. ------------------------------------------------------------------------------------------------ void init(void*(*address_getter)(char*), void(*data_setter)(char*, void*)) ------------------------------------------------------------------------------------------------ The stub dll, which exports all the same symbols as the real python25.dll, uses these functions to initialise the 3 types of exported symbols: ** Simple pointers -- the symbol points to a pointer-sized lump of memory in the library, which itself points to some actual data somewhere else. In this case, the 'init' function just calls 'address_getter' with the symbol name, and writes the returned address into the memory referred to by the symbol. For example: ------------------------------------------------------------------------------------------------ PyExc_OverflowError = address_getter("PyExc_OverflowError"); ------------------------------------------------------------------------------------------------ ** Function pointers -- the symbol points to an arbitrarily-sized lump of memory in the library, containing actual code. Function pointers also use 'address_getter', because every function is implemented in assembler as a single JMP instruction; the target of this JMP is stored in a table whose contents are set according to the results of GetAddress calls. For example: ------------------------------------------------------------------------------------------------ jumptable[649] = address_getter("Py_InitModule4"); ------------------------------------------------------------------------------------------------ causes the following assembler code to correctly intercept a call to Py_InitModule4 and pass it up to managed code. ------------------------------------------------------------------------------------------------ _Py_InitModule4: jmp [_jumptable+2596] ------------------------------------------------------------------------------------------------ ** Static data -- the symbol points to an arbitrarily-sized lump of memory in the library, containing actual data. These symbols are initialised with 'data_setter', which fills the referenced block of memory with appropriate data: ------------------------------------------------------------------------------------------------ data_setter("PyFile_Type", &PyFile_Type); ------------------------------------------------------------------------------------------------ ** Once 'address_getter' or 'data_setter' has been called for every symbol, calls to 'python25.dll' should correctly delegate to the Python25Mapper and hence affect the state of the managed ScriptEngine. * After the StubReference has been created, the mapper also creates a PydImporter for later use. IMPORTING EXTENSION MODULES Is fairly simple: ------------------------------------------------------------------------------------------------ mapper.LoadModule("C:\\Python25\\Dlls\\bz2.pyd") ------------------------------------------------------------------------------------------------ As yet, we don't have import hooks to do this automagically, but we don't forsee any intractable problems; once we've made this call, the bz2 module should be available for IronPython code to import. Calling LoadModule causes the following sequence of events: * The .pyd is loaded into the process' address space. * The module's initialisation function is called; it makes calls into the stub python25.dll, which redirects them to the Python25Mapper. Once the initialisation function is complete, the mapper's PythonEngine's sys.modules will contain a new module with appropriate contents. For example, when bz2.pyd's 'initbz2' function is called, it (almost immediately) calls the C 'Py_InitModule4' function. Because the process space already has a library called 'python25' containing the symbol 'Py_InitModule4', the code corresponding to that symbol is executed. This code is a single jmp instruction, leading to the function pointer that was returned by the call to address_getter("Py_InitModule4") a few paragraphs ago. And... execution jumps to 'Python25Mapper.Py_InitModule4'. The parameters include the CPython module's name and docstring, and a pointer to its method-names, -docstrings, and -implementations; we use this information to generate Python code which we execute in a new ScriptScope, giving it the appropriate properties (we'll take a look at this code a little bit later). We also create a dictionary mapping function names to delegates, which are themselves created from the function pointers to the actual implementations. This 'DispatchTable' dictionary, and the Python25Mapper itself, are used to construct a new Dispatcher object which is stored in the new module. At this point, the new module exists, and it becomes possible to 'import bz2'; however, it still isn't very useful. Anyway, once we have created the new module, we pass it into the Python25Mapper's 'Store' method, which: * allocates a tiny chunk of memory (enough to hold a refcount and a type pointer); * stores the module in a dictionary, keyed on the pointer we just allocated; and * returns the pointer. 'Py_InitModule4' then returns that pointer; 'initbz2' checks that the pointer is not NULL, and then uses it as the first parameter to a series of 'PyModule_AddObject' calls, which it uses to add the BZ2File, BZ2Compressor, and BZ2Decompressor types to the module. These calls are intercepted in the same way as before, so the parameters are passed up to the Python25Mapper's 'PyModule_AddObject' method. The first parameter is always the pointer we allocated previously; the Python25Mapper's 'PyModule_AddObject' looks up this pointer in the dictionary and retrieves a reference to the real module, and then generates and executes some more code (in that module) to define the new types in such a way that IronPython can use them. Any methods defined on those types are also added to the module's DispatchTable. Finally, 'PyModule_AddObject' extracts a reference to the newly-defined IronPython type, and sets the '_typePtr' attribute to a pointer to the PyTypeObject. At this point, we can start doing interesting things with the module. USING EXTENSION MODULES It should now be possible to use an extension module from IronPython just as one would from CPython -- however, it may be interesting to know how that actually occurs. The critical component is the Dispatcher object alluded to previously; we have generated IronPython code for every function and type defined in the extension module, and this generated code uses the module's Dispatcher to do all the heavy lifting. Here are a few bits of the code generated while importing the BZ2 module: ------------------------------------------------------------------------------------------------ def decompress(*args): '''decompress(data) -> decompressed data Decompress data in one shot. If you want to decompress data sequentially, use an instance of BZ2Decompressor instead. ''' return _dispatcher.function_varargs('decompress', *args) ... class BZ2Decompressor(object): '''BZ2Decompressor() -> decompressor object Create a new decompressor object. This object may be used to decompress data sequentially. If you want to decompress data in one shot, use the decompress() function instead. ''' __module__ = 'bz2' def __new__(cls, *args, **kwargs): return _dispatcher.construct('BZ2Decompressor.tp_new', cls, *args, **kwargs) def __init__(self, *args, **kwargs): _dispatcher.init('BZ2Decompressor.tp_init', self, *args, **kwargs) def __del__(self): _dispatcher.delete('BZ2Decompressor.tp_dealloc', self) def decompress(self, *args): '''decompress(data) -> string Provide more data to the decompressor object. It will return chunks of decompressed data whenever possible. If you try to decompress data after the end of stream is found, EOFError will be raised. If any data was found after the end of stream, it'll be ignored and saved in unused_data attribute. ''' return _dispatcher.method_varargs('BZ2Decompressor.decompress', self._instancePtr, *args) ------------------------------------------------------------------------------------------------ It should be reasonably clear that the callables are composed of nothing but a docstring and a call to one of the Dispatcher's methods; furthermore, that the call is essentially a pass-through of the original arguments, with the name by which the delegate has been stored prepended. Here's part of the code for the Dispatcher class: ------------------------------------------------------------------------------------------------ class Dispatcher(object): def __init__(self, mapper, table): self.mapper = mapper self.table = table def _maybe_raise(self, resultPtr): error = self.mapper.LastException if error: self.mapper.LastException = None raise error if resultPtr == nullPtr: raise NullReferenceException('CPython callable returned null without setting an exception') def _cleanup(self, *args): self.mapper.FreeTemps() for arg in args: if arg != nullPtr: self.mapper.DecRef(arg) ... def function_varargs(self, name, *args): return self.method_varargs(name, nullPtr, *args) def method_varargs(self, name, instancePtr, *args): argsPtr = self.mapper.Store(args) resultPtr = self.table[name](instancePtr, argsPtr) try: self._maybe_raise(resultPtr) return self.mapper.Retrieve(resultPtr) finally: self._cleanup(resultPtr, argsPtr) ------------------------------------------------------------------------------------------------ The Dispatcher class also contains methods for various other call scenarios (noargs, objarg, selfarg, kwargs); they are similar to the varargs case shown above. In these cases the same procedure always applies. * Store the args (and kwargs, if appropriate), effectively translating them into a form that CPython can understand. * Call the appropriate delegate (chosen by name), passing in the appropriate arguments. * Check for any error set by the C code, and raise it if it exists; also, raise an exception if the C code failed to return a meaningful value. * Retrieve the returned pointer, effectively translating it into a managed object. * Clean up by DecReffing anything we've allocated -- and the result pointer as well, since we own an unmanaged reference to it -- and return the managed result. The foregoing elides the details of what's going on in the unmanaged code when the delegate is called; a few factors worth considering are: * The unmanaged code calls PyArg_ParseTuple, which takes (C) varargs and would be a real pain to implement in managed code. Thankfully, CPython has a perfectly good implementation of this (and related functions), all of which is implemented in terms of other parts of the CPython API, so we just borrowed that. We do the same with several other CPython API functions. * Calling Store with different types will have different effects; while most types can be represented as, er, 'mostly-opaque' pointers (which is to say, a pointer to an 8-byte struct holding a refcount and a type pointer), some types may have fields which are directly accessed by CPython API macros. So, passing strings, lists, and tuples to unmanaged code is a lot more complex than passing (say) dictionaries. * If a CPython extension creates one of these types, it won't necessarily contain correct data until the function call is complete; in these cases, we temporarily map the pointer to an UnmanagedDataMarker object, and convert the unmanaged data into a managed object on Retrieve. So, modulo various tedious details, almost all calls into unmanaged code are fundamentally similar, be the callees functions or methods. Actually creating an instance of an unmanaged type, on which methods can be called, is a little more involved. MANAGED OBJECTS WITH UNMANAGED TYPES Here's the code from the Dispatcher class that deals with object creation and deletion: ------------------------------------------------------------------------------------------------ def construct(self, name, klass, *args, **kwargs): instance = object.__new__(klass) argsPtr = self.mapper.Store(args) kwargsPtr = self.mapper.Store(kwargs) instancePtr = self.table[name](klass._typePtr, argsPtr, kwargsPtr) try: self._maybe_raise(instancePtr) finally: self._cleanup(argsPtr, kwargsPtr) self.mapper.StoreUnmanagedInstance(instancePtr, instance) instance._instancePtr = instancePtr return instance def init(self, name, instance, *args, **kwargs): argsPtr = self.mapper.Store(args) kwargsPtr = self.mapper.Store(kwargs) result = self.table[name](instance._instancePtr, argsPtr, kwargsPtr) self._cleanup(argsPtr, kwargsPtr) if result < 0: raise Exception('%s failed; object is probably not safe to use' % name) def delete(self, name, instance): self.mapper.ReapStrongRefs() if self.mapper.RefCount(instance._instancePtr) > 1: self.mapper.Strengthen(instance) GC.ReRegisterForFinalize(instance) return self.table[name](instance._instancePtr) ------------------------------------------------------------------------------------------------ The actual calls are made in much the same way as before, but there's some extra bookkeeping to be done: the most important thing to take away is how object lifetimes are managed. Once the constructor delegate has been successfully called, the managed and unmanaged instances are associated with a call to StoreUnmanagedInstance; this is critically different to all the other Store methods on the mapper, in that the mapper does not keep a strong reference to the managed instance. Because of this, the managed object will be garbage-collected once the managed code runs out of references to it. However, it is still possible for unmanaged code to hold references to the object. Therefore, the delete method checks the instance's unmanaged refcount before calling the deallocator, and resurrects the object (by calling Strengthen) if anything else owns a reference to it. Doing this, of course, makes the object effectively immortal; to avoid this situation, we always check the set of strong references in the delete method, and throw them away if they are no longer required. Once a strong reference has been reaped in this way, the next garbage collection should successfully finish it off. CLEANUP We haven't worked out how to neatly clean up a Python25Mapper at finalization time, so it is vitally important to Dispose it once we've finished with it. It doesn't make sense to have more than one Python25Mapper per process, so this shouldn't be too difficult to keep track of; be aware that failing to Dispose will probably lead to crashes on shutdown, and that allowing two Python25Mappers to coexist in the same process will inevitably lead to bizarre and ugly crashes.