{"id":7365,"date":"2020-03-29T15:00:47","date_gmt":"2020-03-29T15:00:47","guid":{"rendered":"https:\/\/www.version1.com\/en-us\/?p=7365"},"modified":"2020-03-29T15:00:47","modified_gmt":"2020-03-29T15:00:47","slug":"kdb-attributes-for-beginners","status":"publish","type":"post","link":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/","title":{"rendered":"kdb+: Attributes for Beginners’"},"content":{"rendered":"

kdb+ is known for its ability to store, retrieve and process large data volumes at high-speed. One of the many things that make this possible is Attributes \u2013 descriptors that can be applied to lists. These descriptors allow the q interpreter to perform optimizations \u201cunder-the-hood\u201d that can drastically increase lookup speeds when used effectively.<\/p>\n

This blog will cover the Attributes used in kdb+, what each attribute tells the q interpreter, and how a given attribute could be used to increase performance.<\/p>\n

Sorted (`s#)<\/h2>\n

When the `s# attribute is applied to a list it tells the interpreter that the elements of that list are sorted in ascending order. This allows q to perform a binary search on the list by default, drastically reducing lookup times.<\/p>\n

\/\/create sorted list of 10000000 rows
\nq)names:`Alan`John`Muhammad`Mary`Joseph`Sean`Patrick`William`Katherine
\nq)n:10000000;<\/p>\n

q)employee:([] id:.Q.dd'[n?names;til n]; salary:20000+n?50000);<\/p>\n

q)employee
\nid           salary
\n-------------------
\nWilliam.0    29809
\nMary.1       44125
\nKatherine.2  31158
\nSean.3       68322
\nKatherine.4  25272
\n..<\/p>\n

q)meta employee
\nc     | t f a
\n------| -----
\nid    | s
\nsalary| j<\/p>\n

\/\/ query top earners without `s# applied (linear search)
\nq)t select from employee where salary>65000
\n126 \/\/(t=126) query ran in 126 milliseconds<\/p>\n

\/\/ sort employees on salary
\nq)`salary xasc `employee \/\/this automatically applies `s on sort column
\n`employee<\/p>\n

\/\/ query top earners with `s# applied (binary search)
\nq)t select from employee where salary>65000
\n31 \/\/(t=31) query ran in 31 milliseconds<\/code><\/p>\n

Binary searching is an efficient search algorithm on sorted lists. It works by sampling the midpoint of a list, splitting it in two. Depending on the result it picks one of the two resulting lists to sample. This process is repeated until the list is narrowed down to an individual location.<\/p>\n

\"\"<\/p>\n

Many people have described a binary search algorithm more eloquently than I ever could so here\u2019s a good article<\/a> I found.<\/p>\n

Unique (`u#)<\/h2>\n

Applying the `u# attribute to a list indicates that all the elements in the list are unique. One of the optimizations the unique attribute allows is the interpreter can break-out from some functions early e.g. the \u2018distinct\u2019 function.<\/p>\n

The real power of `u# comes from constant time look ups on unique list elements. To achieve this q creates a hash-table in the background \u2013 which takes up some of the memory available to your Q process, this may become a considering factor in applying `u# depending on list size\/kdb+ setup.<\/p>\n

Using our previous \u2018employee\u2019 table, let\u2019s look at the performance gains when using the `u# attribute.<\/p>\n

\/\/select last employee id from employees table
\nq)t select from employee where id=`Patrick.4326549
\n28 \/\/(t=28) query ran in 28 milliseconds<\/p>\n

q)t select distinct id from employee
\n256 \/\/(t=256) query ran in 256 milliseconds<\/p>\n

\/\/apply the unique attribute to the id column
\nq)employee:update id:`u#id from employee;<\/p>\n

\/\/Same queries With `u# applied
\nq)t select from employee where id=`Patrick.4326549
\n0 \/\/(t=0) query ran in less than 0 milliseconds<\/p>\n

q)t select distinct id from employee
\n0 \/\/(t=0) query ran in less than 0 milliseconds [distinct early exit]<\/code><\/p>\n

Although hash-tables are slightly beyond the scope of this blog, I found it very interesting to learn more about how database indexing with hash-tables is working in this given scenario. Anyone interested in some further reading on this I would recommend reading here<\/a>.<\/p>\n

Parted (`p#)<\/h2>\n

Application of the `p# attribute on a list tells the interpreter that all commonly occurring values are positioned adjacent in a contiguous block I.e. all common values sit beside each other.<\/p>\n

q)contiguousList:`SNY`SNY`IBM`IBM`IBM`IBM`APPL`APPL`APPL`VOD`VOD`VOD<\/code><\/p>\n

When the `p# attribute is applied q creates a dictionary storing the position of the first occurrence of each block of values, therefore q knows the exact range of addresses to look at when retrieving this data.<\/p>\n

q)first each group contiguousList
\nSNY | 0
\nIBM | 2
\nAPPL| 6
\nVOD | 9
\n<\/code><\/p>\n

Anyone who has worked with kdb+ will be aware that a table\u2019s \u2018sym\u2019 column generally has the `p# attribute applied on disk and the `g# in memory (see `g# below)\u2013 why is this the case?<\/p>\n

It\u2019s typical that when querying a securities database most queries will filter on a particular asset first i.e., \u2018APPL\u2019 or \u2018GBPUSD\u2019. By applying the `p# attribute to the column identifying an asset (\u2018sym\u2019); assets are stored in a contiguous block on disk. Therefore, when q needs to pull this data into memory the range of indices to retrieve is known in advance because of the `p#, speeding up any on-disk lookups.<\/p>\n

q)n:100000000;
\nq)syms:`VOD`GOOG`IMB`MSFT`AMZN;
\nq)trade:([]time:n?.z.t;sym:n?syms;price:n?10000f); \/\/create a trade table<\/p>\n

q)`:2020.12.01\/trade\/ set .Q.en[`:.;trade]; \/\/save table in hdb<\/p>\n

q)l . \/\/reload hdb<\/p>\n

q)meta trade \/\/meta info shows no `p#
\nc    | t f a
\n-----| -----
\ndate | d
\ntime | t
\nsym  | s
\nprice| f<\/p>\n

q)t select from trade where date=2020.12.01,sym=`GOOG;
\n1790 \/\/(t = 1790) query took 1790 milliseconds<\/p>\n

\/\/apply `p# to on-disk table
\nq)@[`sym xasc .Q.par[`:.;2020.12.01;`trade];`sym;`p#];<\/p>\n

q)meta trade \/\/meta info shows `p# applied to sym
\nc    | t f a
\n-----| -----
\ndate | d
\ntime | t
\nsym  | s   p
\nprice| f<\/p>\n

q)t select from trade where date=2020.12.01,sym=`GOOG;
\n274 \/\/(t = 274) query took 274 milliseconds<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"

kdb+ is known for its ability to store, retrieve and process large data volumes at high-speed. One of the many things that make this possible is Attributes \u2013 descriptors that can be applied to lists. These descriptors allow the q interpreter to perform optimizations \u201cunder-the-hood\u201d that can drastically increase lookup speeds when used effectively. This […]<\/p>\n","protected":false},"author":0,"featured_media":7366,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","inline_featured_image":false,"footnotes":""},"categories":[101],"tags":[],"industry":[],"class_list":["post-7365","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"translations":[{"blog_id":1,"post_id":7365,"name":"Europe","code":"EN-GB","hreflang":"en-gb","url":"https:\/\/www.version1.com\/blog\/kdb-attributes-for-beginners\/","is_current":false},{"blog_id":13,"post_id":7365,"name":"Americas","code":"EN-US","hreflang":"en-us","url":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/","is_current":true}],"yoast_head":"\nkdb+: Attributes for Beginners' | Version 1 (US)<\/title>\n<meta name=\"description\" content=\"This blog will cover the attributes used in kdb+, what each attribute tells the q interpreter, and how it may be used to increase performance\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"kdb+: Attributes for Beginners' | Version 1 (US)\" \/>\n<meta property=\"og:description\" content=\"This blog will cover the attributes used in kdb+, what each attribute tells the q interpreter, and how it may be used to increase performance\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/\" \/>\n<meta property=\"og:site_name\" content=\"Version 1 (US)\" \/>\n<meta property=\"article:published_time\" content=\"2020-03-29T15:00:47+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/03\/writing-828911_1920.png.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"680\" \/>\n\t<meta property=\"og:image:height\" content=\"440\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/\"},\"author\":{\"name\":\"\",\"@id\":\"\"},\"headline\":\"kdb+: Attributes for Beginners’\",\"datePublished\":\"2020-03-29T15:00:47+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/\"},\"wordCount\":533,\"publisher\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/wp-content\\\/uploads\\\/2020\\\/03\\\/writing-828911_1920.png.webp\",\"articleSection\":[\"Blog\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/\",\"url\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/\",\"name\":\"kdb+: Attributes for Beginners' | Version 1 (US)\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/wp-content\\\/uploads\\\/2020\\\/03\\\/writing-828911_1920.png.webp\",\"datePublished\":\"2020-03-29T15:00:47+00:00\",\"description\":\"This blog will cover the attributes used in kdb+, what each attribute tells the q interpreter, and how it may be used to increase performance\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/wp-content\\\/uploads\\\/2020\\\/03\\\/writing-828911_1920.png.webp\",\"contentUrl\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/wp-content\\\/uploads\\\/2020\\\/03\\\/writing-828911_1920.png.webp\",\"width\":680,\"height\":440,\"caption\":\"writing in notebook\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/kdb-attributes-for-beginners\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"kdb+: Attributes for Beginners’\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#website\",\"url\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/\",\"name\":\"Version 1\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#organization\",\"name\":\"Version 1\",\"url\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.version1.com\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/cropped-cropped-Favicon-2.png\",\"contentUrl\":\"https:\\\/\\\/www.version1.com\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/cropped-cropped-Favicon-2.png\",\"width\":512,\"height\":512,\"caption\":\"Version 1\"},\"image\":{\"@id\":\"https:\\\/\\\/www.version1.com\\\/en-us\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"kdb+: Attributes for Beginners' | Version 1 (US)","description":"This blog will cover the attributes used in kdb+, what each attribute tells the q interpreter, and how it may be used to increase performance","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/","og_locale":"en_US","og_type":"article","og_title":"kdb+: Attributes for Beginners' | Version 1 (US)","og_description":"This blog will cover the attributes used in kdb+, what each attribute tells the q interpreter, and how it may be used to increase performance","og_url":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/","og_site_name":"Version 1 (US)","article_published_time":"2020-03-29T15:00:47+00:00","og_image":[{"width":680,"height":440,"url":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/03\/writing-828911_1920.png.webp","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/#article","isPartOf":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/"},"author":{"name":"","@id":""},"headline":"kdb+: Attributes for Beginners’","datePublished":"2020-03-29T15:00:47+00:00","mainEntityOfPage":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/"},"wordCount":533,"publisher":{"@id":"https:\/\/www.version1.com\/en-us\/#organization"},"image":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/#primaryimage"},"thumbnailUrl":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/03\/writing-828911_1920.png.webp","articleSection":["Blog"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/","url":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/","name":"kdb+: Attributes for Beginners' | Version 1 (US)","isPartOf":{"@id":"https:\/\/www.version1.com\/en-us\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/#primaryimage"},"image":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/#primaryimage"},"thumbnailUrl":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/03\/writing-828911_1920.png.webp","datePublished":"2020-03-29T15:00:47+00:00","description":"This blog will cover the attributes used in kdb+, what each attribute tells the q interpreter, and how it may be used to increase performance","breadcrumb":{"@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/#primaryimage","url":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/03\/writing-828911_1920.png.webp","contentUrl":"https:\/\/www.version1.com\/en-us\/wp-content\/uploads\/2020\/03\/writing-828911_1920.png.webp","width":680,"height":440,"caption":"writing in notebook"},{"@type":"BreadcrumbList","@id":"https:\/\/www.version1.com\/en-us\/kdb-attributes-for-beginners\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.version1.com\/en-us\/"},{"@type":"ListItem","position":2,"name":"kdb+: Attributes for Beginners’"}]},{"@type":"WebSite","@id":"https:\/\/www.version1.com\/en-us\/#website","url":"https:\/\/www.version1.com\/en-us\/","name":"Version 1","description":"","publisher":{"@id":"https:\/\/www.version1.com\/en-us\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.version1.com\/en-us\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.version1.com\/en-us\/#organization","name":"Version 1","url":"https:\/\/www.version1.com\/en-us\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.version1.com\/en-us\/#\/schema\/logo\/image\/","url":"https:\/\/www.version1.com\/wp-content\/uploads\/2026\/05\/cropped-cropped-Favicon-2.png","contentUrl":"https:\/\/www.version1.com\/wp-content\/uploads\/2026\/05\/cropped-cropped-Favicon-2.png","width":512,"height":512,"caption":"Version 1"},"image":{"@id":"https:\/\/www.version1.com\/en-us\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/posts\/7365","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/comments?post=7365"}],"version-history":[{"count":0,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/posts\/7365\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/media\/7366"}],"wp:attachment":[{"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/media?parent=7365"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/categories?post=7365"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/tags?post=7365"},{"taxonomy":"industry","embeddable":true,"href":"https:\/\/www.version1.com\/en-us\/wp-json\/wp\/v2\/industry?post=7365"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}