kdb+ Data Warehouse Implementation

A global investment bank was planning an internal initiative to centralise the capture and storage of all equity market and reference data into a single warehouse. Their main business objectives were to reduce duplication, hardware and support overheads, and system complexity.

It was essential that the solution be globally accessible in a timely manner by both programmatic applications and individual groups for research and be available 24 hours a day. To accommodate future growth in data volumes and in the user community, the platform had to be scalable.

The platform was built in kdb+ with a web-based UI that users could interact with and was designed with high-availability and redundancy from inception.

Another important feature of the system was the ability to classify critical users and applications, and give priority to these groups in the case of reduced capability (due to potential hardware failure) or increased volatility and data volumes (Due to market conditions).

Hardware specification was designed to provide sufficient headroom to capture ever increasing volumes of data as well as offer scalability options for future growth.

As a result of this engagement, the following benefits were delivered:

• New data warehouse with a single point of access for all equity market and reference data
• Ability to handle any individual data-node failure, machine or site outage with no loss of data availability to the end users
• Robust and easily maintainable solution that can scale horizontally for future growth