Emdros

I was doing some speed tests of the BPT engine on Mac OS X Tiger, which ships with SQLite 3.1.3.  I accidentally built Emdros against the SQLite 3 that ships with Tiger, and found that GET OBJECTS HAVING MONADS IN was painfully slow. It took up to 10 minutes to run a basic query.

I found out that SQLite 3.1.3 doesn’t have some optimizations for column_name BETWEEN X AND Y which are present in later versions of SQLite 3.  GET OBJECTS HAVING MONADS IN makes heavy use of precisely this construct.

So I changed all cases of this construct to the idiom “column_name >= X AND column_name <= Y”. This has no noticeable side-effects on the speed on Linux, but does give a great increase in speed on Mac OS X.

Still, BPT beats SQLite 3 by a wide margin: 270 seconds for SQLite 3, 214 seconds for BPT, to run all test-queries in one of my test suites (124 queries against a syntactic database of 1.4 million syntactic objects). That’s a 20% speed increase on this particular combination of harware and Mac OS X 10.4. (The hardware is a 2007 Mac Mini, Intel Core Duo 1.6GHz, 1GB Ram).

Ulrik

I’ve continue to make progress on the BPT engine. It now compiles, works, and is fast on Windows as well as on Linux. The speedup is about the same on Windows as on Linux.

Along the way, I’ve fixed a bug in FastSetOfMonads::isMemberOf(), and have also reimplemented some of the functions in include/string_func.h, so that they don’t use the C standard library; functions such as hex2char, octal2char, char2hex, and char2upperhex.  This means that mqldump() will most likely be faster as well (haven’t tested this yet, though!).

Ulrik

Before (and while) writing the Bit Packed Table backend, I wrote a program in Python which could create a Bit Packed Table database, given an already-created Emdros database. The purpose of writing it in Python was so as to be able to modify it quickly, given that Python is so malleable.

Today I finished porting the Python program to C++. It is much faster (predictably), and creates exactly the same databases as the Python version, bit for bit.

Remember, the BPT engine won’t be Open Source any time soon. If you are interested in licensing it, please let me know. Existing licensees will, of course, get a free upgrade.

Ulrik