Tick Databases

Ok, so the street has largely embraced KDB as a tick database for high-frequency algo. It is by far the worst DB I have every worked with in terms of reliability and functionality. The one thing it does have going for it is that it is fast.

Other than that, if you want to deal with the agony of a database that falls over on simple operations, has a largely unintelligable language Q (a dialect of APL) that noone cares for, and has next to nothing in the way of support from the vendor, be my guest.

KDB is a very (very!) raw database, not much more than a process or two around a a bunch of binary files, one for each timeseries per day.

Stay away.

Advertisements

4 Comments

Filed under Uncategorized

4 responses to “Tick Databases

  1. Anonymous

    So what’s better?

  2. Dan

    I too would like to know: what do you use now?

    We tend to use flat files of ASCII data, gzipped, and then build index files on top, also gzipped. This is for simulation purposes. For real-time stuff, we just keep everything in in-memory data-structures in whatever language the app is built in.

    • tr8dr

      Actually we use a home-brewed implementation. Uses a binary format with fixed record size (though the record types can vary depending on what one is storing in a given timeseries). We don’t bother with indexing as use a binary search like algorithm and paged memory caching to navigate around the timeseries efficiently. For fine grained data (such as level-2), the database breaks into directories by date, for coarser grained data, is held in a single file per instrument.

      The DB allows various queries, and sampling on the data before returning as a timeseries to an application. This timeseries can be “lazy” in that it will only page in required data when needed.

      Our live trading may use this as an initialization source, however, all intra-day activity is in memory.

  3. These days we have GridGain/Apache Ignite, McObject and a few others

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s