kdb+: 10 Myths Debunked
kdb+ Myths
If you’re unfamiliar with kdb+, you may be sceptical of using it for many different reasons. Perhaps you think it’s only used in finance, its processing power is overhyped, or you just don’t like the look of it. We want to show you what KDB+ really is and have addressed some of the most common concerns in this post, with the aim to reassure anyone that’s on the fence:
Myth 1: No Support Collateral
Kx have developed a site providing all the information you need to get started with kdb+, from programming idioms to comprehensive cookbooks. Over the years, kdb+ developers have produced an abundance of technical blogs and whitepapers to assist on every possible application of the technology. There is also a wealth of support across platforms such as Stack Overflow, Quora, plus a very active google group where users are keen to share their experiences. Some highly appraised books are available on Amazon too:
- Q Tips: Fast, Scalable and Maintainable kdb+
- Machine Learning and Big Data with kdb+/q
Myth 2: No IDE
At first glance of the support content, you may think that q can only be run on the console (given that most of the examples use that), but that’s not the case. Where you execute kdb+ developer code is personal preference, and there are several choices for developers who would prefer to use an IDE. Some of the most common tools available for free are:
- qPad – http://www.qinsightpad.com/
- Kx Developer – https://code.kx.com/developer/
- JupyterQ – https://github.com/KxSystems/jupyterq/
- IntelliJ IDE – https://plugins.jetbrains.com/plugin/7925-q
Myth 3: Hard To Implement
kdb+ has been used extensively in the financial industry, tackling some of the most complex problems that big data and technology can offer. This has allowed a multitude of systems and processes to be developed that showcase the best results, and there is ample content online to guide kdb+ developers on resolving issues and discovering new ideas – it’s only a matter of finding the solution.
The most difficult part for experienced developers used to other technologies is appreciating that kdb+ has a slightly different way of approaching some things. For example, abandoning the idea of loops and using the powerful kdb+ adverbs instead, coding right-to-left, or simply the terseness of q, can all be intimidating at first. But once these concepts become innate, the power of kdb+ is unparalleled.
Myth 4: Difficult To Debug
Debugging kdb+ has never been easier since the release of v3.5, which includes visual indicators and a backtrace of the q call stack. Kx Developer also has a useful quick-debugger which has a non-interactive backtrace that loads straight to where the error occurs.
Myth 5: Difficult To Interact With Other Technologies
There are several open interfaces enabling kdb+ to communicate to and from other technologies, most of which are accompanied by thorough whitepapers showing you how to use them:
- Python (PyQ/embedPy)
- Java/JDBC
- ODBC
- R
- Kafka
- JavaScript (WebSockets)
Visit https://code.kx.com/q/interfaces/ for more information on the interfaces to and from kdb+. Users regularly develop new interfaces and upload them to the Kx Community website, so the list of available interfaces is continuously growing.
Myth 6: Cannot Use It With Other Statistical Packages
The interfaces detailed above allow kdb+ to be combined with a vast number of statistical packages. Many developers unite kdb+ with the likes of MATLAB or Python for their Machine Learning toolkits, or with R’s quantmod package to illustrate charts. You can read whitepapers detailing each interface here: https://code.kx.com/q/wp/#interfaces. Further to this, Kx have introduced the Fusion initiative, making it easier than ever to integrate with the statistical packages outlined above; more information can be found at: https://code.kx.com/q/interfaces/fusion/
Myth 7: Hardware Hungry
One of the unique appeals of kdb+ is that it can efficiently handle big data at high frequency – this can give rise to the misconception that kdb+ needs a lot of hardware to feed it, however that is entirely dependent on the amount of data being processed. kdb+ itself is very economical when it comes to memory consumption, with most customers beginning with a 4- or 8-core system and growing from there. As the historical database builds up, multi-terabyte storage may be needed, but kdb+ is flexible – using local storage, SANs, or any combination to allows users to find the balance between optimal cost and performance benefits.
Myth 8: No Multithreading
kdb+ has supported multithreading since v2.4 (2012). More recently, v4.0 (2020) added an additional level of multithreading via primitives, and it requires no code change by the user to exploit it. Multithreading thrives when used in high frequency applications, and it is an important tool for kdb+ architects in a multi-CPU processing environment. It will be a hinderance if applied to applications that are low latency as the overhead of the multi-threading process will outweigh the benefits.
Myth 9: Only For Financial Sector
Industries beyond finance are also looking for a solution to their big data needs, and the small footprint and flexibility of kdb+ provides just that. Today, kdb+ is used across a vast range of sectors from space exploration to Formula 1, pharmaceuticals to manufacturing, and telco to utilities. In 2019, Aston Martin Red Bull Racing announced their initiative to start using kdb+ for ingesting and analysing sensor data (read more at https://kx.com/news/kx-appointed-innovation-partner-to-aston-martin-red-bull-racing/), and even NASA’s Frontier Development Lab has been leveraging machine learning capabilities in kdb+ (read more at https://kx.com/news/kx-leverages-ai-to-solve-new-space-challenges/).
Myth 10: It Can’t Be That Fast
Beyond the reputation of kdb+ being extremely fast at processing big data, there have been many tests carried out to prove it. The Securities Technology Analysis Center (STAC) researches technical challenges and solutions and develops technology benchmark standards within the capital markets industry. These are used by the largest financial institutions in the world to make decisions on which technology will address their problems. kdb+ continuously sets records in real-world financial data analytics benchmarks performed by STAC, particularly showcasing its speed, details of which can be read here.
We hope that this has addressed any burning questions at the back of your mind regarding kdb+, and if there is anything else that is left unanswered please get in touch with Version 1’s team of kdb+ specialists and they will be happy to help.
