{"id":7334,"date":"2020-08-16T10:00:39","date_gmt":"2020-08-16T10:00:39","guid":{"rendered":"https:\/\/www.version1.com\/en-us\/?p=7334"},"modified":"2020-08-16T10:00:39","modified_gmt":"2020-08-16T10:00:39","slug":"kdb-the-swiss-army-knife-of-tech","status":"publish","type":"post","link":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/","title":{"rendered":"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss"},"content":{"rendered":"<p>I think of <a href=\"https:\/\/www.version1.com\/sectors\/our-financial-services-expertise\/kdb-services-and-support-teams\/\">kdb+<\/a> as the swiss army knife of technologies. Not only is it a database, but with its built-in webserver and own programming language you can do almost anything that you want with it. But with great power comes great responsibility \u2013 just because you\u00a0<strong>can<\/strong>\u00a0do something doesn\u2019t mean that you should.<\/p>\n<p>I always think back to one architectural review I was called into where the client was having reliability issues. They were using kdb+ to analyse performance stats (cpu\/mem util\/etc) across their trading infrastructure (30+ machines) and loved being able to analyse these stats during spikes and trading abnormalities, as well as allow them to profile hardware utilisation and plan for growth. The issue turned out to be a simple TCP\/IP configuration issue but because the developer had written the data collection agent in kdb+ and had moved on, the client was having trouble investing with inhouse resources. We recommended rewriting the agent in java which meant that they could also install the agent on the rest of their infrastructure (100+ machines) without having to worry about increased license costs. I love the simplicity of this review:<\/p>\n<ul>\n<li>It demonstrates how kdb+ can be utilised outside the financial data domain \u2013 the structure of the data and the type of analysis the client wanted to do was ideally suited to kdb+.<\/li>\n<li>Such a simple fix gave the client a lot more comfort supporting the application with their inhouse team. The capture, storage and analyse components within a kdb+ implementation are so well established at this stage and there is a lot of collateral around them \u2013 but there was no reason why the data collection agent should be kdb+.<\/li>\n<li>In addition to resolving the client\u2019s reliability issue, we were able to add value by extending the application to monitor the clients entire infrastructure, at no additional licensing, development or support costs.<\/li>\n<\/ul>\n<h2>Common Characteristics of kdb+ Implementation<\/h2>\n<p>So \u2013 what are the common characteristics of a kdb+ implementation then, when should you use it? Most good kdb+ implementations have a few common requirements:<\/p>\n<p><b>Fast Capture\u00a0<\/b>\u2013 The ability to reliably ingest a large number of events, typically 20K+ per sec with sustained peaks of 1M+ per sec. The events are usually small (~100-300 bytes) and have 5-30 fields but can vary by use case. Batching would be common with higher ingest rates to ensure maximum throughput and optimal data processing.<\/p>\n<p><b>Real\/Near Time Analysis \u2013<\/b>\u00a0Ability to analyse events as they happen and generate aggregations and other ESP (Event Stream Processing) functionality such as windowing, upserts, correlations and as-of joins. Typically, this functionality will be performed while the working datasets are in-memory for fast calculations and best performance.<\/p>\n<p><b>Historical Data Access \u2013\u00a0<\/b>Provide access to vast amounts of historical data for pattern and trend analysis. Data which has been archived from memory into persisted historical data should be accessible in line with the current streaming data to provide a continuous time series if necessary.<\/p>\n<h2>Do you need the Swiss Army Knife?<\/h2>\n<p>There are a number of questions that should be asked before choosing kdb+. These can be summarised by Volume, Structure, Performance and Cost.<\/p>\n<p><b>Do you have data volumes that require you to consider kdb+?<\/b><\/p>\n<p>While it is regarded as the premium time-series database of choice, especially for financial services, you still need to ensure the volume of data justifies the implementation overhead compared with a more mainstream choice.<\/p>\n<p><b>Is the data suited to be stored in a column-oriented database?<\/b><\/p>\n<p>kdb+ gained its reputation storing time-series financial data because that data is ideally suited to columnar\/vector-based storage. You should make sure that the data you intend to store in kdb+ will be utilised in a similar manner. There are many sectors outside of finance where it does make sense to store data like this, such as hardware performance metrics in the example above or sensor data in IoT applications.<\/p>\n<p><b>Do you have the performance requirements to justify the investment costs of a kdb+ implementation?<\/b><\/p>\n<p>While kdb+ will probably be the fastest solution for storing and accessing your data, especially if the volume and structure requirements are satisfied, we still need to ask if the performance requirements justify the overhead. If you need a report overnight, or access to a historical source of data for a short period of time to test a new trading strategy, then you may not need the performance of kdb+ and another solution might be more suitable.<\/p>\n<p><b>Can you justify the cost of implementation and maintenance?<\/b><\/p>\n<p>The final requirement to consider is cost, not just the financial cost of the product but implementation and maintenance costs associated. In recent years we have seen more and more specific time-series databases come to the market, many of which are open source and have reduced implementation costs and should always factor into any equation.<\/p>\n<h2>Conclusion<\/h2>\n<p>One of the most important requirements for any solutions architect is to ensure that the right technology choice is made for each component within the stack, and it is no different with\u202fkdb+.<\/p>\n<p>I\u2019m passionate about\u202fkdb+ : The Swiss Army Knife of Technology and have been lucky enough to witness first-hand the enormous benefits that it has afforded to clients, but I\u2019m even more passionate about using the correct technology for the\u202foccasion. In my last blog, I outlined some the questions\/factors that should be considered when designing a good\u202fkdb+ solution, but\u202fthe first\u202fquestion any good solutions architect should answer is \u2013 is\u202fkdb+ the right technology?<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I think of kdb+ as the swiss army knife of technologies. Not only is it a database, but with its built-in webserver and own programming language you can do almost anything that you want with it. But with great power comes great responsibility \u2013 just because you\u00a0can\u00a0do something doesn\u2019t mean that you should. I always [&hellip;]<\/p>\n","protected":false},"author":0,"featured_media":7335,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[101],"tags":[],"industry":[],"class_list":["post-7334","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"translations":[{"blog_id":1,"post_id":7334,"name":"Europe","code":"EN-GB","hreflang":"en-gb","url":"https:\/\/www.version1.com\/blog\/kdb-the-swiss-army-knife-of-tech\/","is_current":false},{"blog_id":13,"post_id":7334,"name":"Americas","code":"EN-US","hreflang":"en-us","url":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/","is_current":true}],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss | Version 1 (US)<\/title>\n<meta name=\"description\" content=\"In this latest blog post, we discuss Kdb+ The Swiss Army Knife of Technology and it&#039;s common characteristics.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss | Version 1 (US)\" \/>\n<meta property=\"og:description\" content=\"In this latest blog post, we discuss Kdb+ The Swiss Army Knife of Technology and it&#039;s common characteristics.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/\" \/>\n<meta property=\"og:site_name\" content=\"Version 1 (US)\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-16T10:00:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/08\/discuss.png.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"680\" \/>\n\t<meta property=\"og:image:height\" content=\"440\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/\"},\"author\":{\"name\":\"\",\"@id\":\"\"},\"headline\":\"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss\",\"datePublished\":\"2020-08-16T10:00:39+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/\"},\"wordCount\":926,\"publisher\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/discuss.png.webp\",\"articleSection\":[\"Blog\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/\",\"url\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/\",\"name\":\"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss | Version 1 (US)\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/discuss.png.webp\",\"datePublished\":\"2020-08-16T10:00:39+00:00\",\"description\":\"In this latest blog post, we discuss Kdb+ The Swiss Army Knife of Technology and it's common characteristics.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/discuss.png.webp\",\"contentUrl\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/discuss.png.webp\",\"width\":680,\"height\":440,\"caption\":\"people in a meeting sat at a table talking\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-the-swiss-army-knife-of-tech\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#website\",\"url\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/\",\"name\":\"Version 1\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#organization\",\"name\":\"Version 1\",\"url\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.version1.com\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/cropped-cropped-Favicon-2.png\",\"contentUrl\":\"https:\\\/\\\/www.version1.com\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/cropped-cropped-Favicon-2.png\",\"width\":512,\"height\":512,\"caption\":\"Version 1\"},\"image\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss | Version 1 (US)","description":"In this latest blog post, we discuss Kdb+ The Swiss Army Knife of Technology and it's common characteristics.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/","og_locale":"en_US","og_type":"article","og_title":"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss | Version 1 (US)","og_description":"In this latest blog post, we discuss Kdb+ The Swiss Army Knife of Technology and it's common characteristics.","og_url":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/","og_site_name":"Version 1 (US)","article_published_time":"2020-08-16T10:00:39+00:00","og_image":[{"width":680,"height":440,"url":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/08\/discuss.png.webp","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/#article","isPartOf":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/"},"author":{"name":"","@id":""},"headline":"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss","datePublished":"2020-08-16T10:00:39+00:00","mainEntityOfPage":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/"},"wordCount":926,"publisher":{"@id":"https:\/\/www.version1.com\/en-us\/#organization"},"image":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/#primaryimage"},"thumbnailUrl":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/08\/discuss.png.webp","articleSection":["Blog"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/","url":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/","name":"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss | Version 1 (US)","isPartOf":{"@id":"https:\/\/www.version1.com\/en-us\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/#primaryimage"},"image":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/#primaryimage"},"thumbnailUrl":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/08\/discuss.png.webp","datePublished":"2020-08-16T10:00:39+00:00","description":"In this latest blog post, we discuss Kdb+ The Swiss Army Knife of Technology and it's common characteristics.","breadcrumb":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/#primaryimage","url":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/08\/discuss.png.webp","contentUrl":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/08\/discuss.png.webp","width":680,"height":440,"caption":"people in a meeting sat at a table talking"},{"@type":"BreadcrumbList","@id":"https:\/\/www.version1.com\/en-us\/kdb-the-swiss-army-knife-of-tech\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.version1.com\/en-us\/"},{"@type":"ListItem","position":2,"name":"kdb+ : The Swiss Army Knife of Technology \u2013 Let\u2019s Discuss"}]},{"@type":"WebSite","@id":"https:\/\/www.version1.com\/en-us\/#website","url":"https:\/\/www.version1.com\/en-us\/","name":"Version 1","description":"","publisher":{"@id":"https:\/\/www.version1.com\/en-us\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.version1.com\/en-us\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.version1.com\/en-us\/#organization","name":"Version 1","url":"https:\/\/www.version1.com\/en-us\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.version1.com\/en-us\/#\/schema\/logo\/image\/","url":"https:\/\/www.version1.com\/wp-content\/uploads\/2026\/05\/cropped-cropped-Favicon-2.png","contentUrl":"https:\/\/www.version1.com\/wp-content\/uploads\/2026\/05\/cropped-cropped-Favicon-2.png","width":512,"height":512,"caption":"Version 1"},"image":{"@id":"https:\/\/www.version1.com\/en-us\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/posts\/7334","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/comments?post=7334"}],"version-history":[{"count":0,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/posts\/7334\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/media\/7335"}],"wp:attachment":[{"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/media?parent=7334"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/categories?post=7334"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/tags?post=7334"},{"taxonomy":"industry","embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/industry?post=7334"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}