A Birdseye View of kdb+ – Monitoring in the Cloud
Commercial systems involving kdb+ software, such as those in tier one banks, tend to be large, bespoke and mission critical so it’s important to get the right monitoring solution in place when monitoring in the cloud. The solution needs to be capable of dealing with large amounts of data, be highly customisable, responsive and have metrics that are both actionable and able to signal alarm states.
It must be capable of monitoring the system holistically from the level of the servers to the level of the internal kdb+ metrics like row count.
Naturally, it should be capable of collating all this information into a visually effective dashboard.
Dashboards like below can be used to monitor implementations at the server instance level and give the user an insight into the overall health of the virtual server in the cloud:
And dashboards like this offer users the ability to monitor process level statistics:
It also must have the ability to track kdb+ logs and extract insights from them whilst filtering them for potential issues so an alert can be raised, or an action automatically performed.
All of which should be displayed on a clear dashboard allowing you to cut through the noise and see exactly what you need with highly flexible content like the following dash:
With companies moving towards either a partial or full deployment of cloud based systems, wouldn’t it be great if all this was cloud native? I know what you’re thinking and you’re right, my head IS in the clouds!
Introducing CloudWatch as a consideration for monitoring of systems involving kdb+ software. All of the above was done in AWS’ native monitoring tool from scratch and with relative ease. CloudWatch allows:
- Any system metric that can be represented as a double or file to be pushed to CloudWatch based on either the occurrence of an event or a schedule.
- Centralisation and consolidation of all monitoring for your system, not just the kdb+ software aspect.
- Storage and archiving of the kdb+ metrics and logs which persist long after the system is no more.
- Kdb+ metrics to be visualised in a wide variety of graph types in customisable widgets and displayed in dashboards.
- Alarms to be set and events to be created resulting in either automatic actions or alerts sent to your phone or email for example.
- Kdb+ log files to be queried by Logs Insights, the purpose-built query language for AWS CloudWatch Logs with data meaningfully parsed and extracted so it can be displayed or actioned upon.
- Data to be encrypted both in transit and at rest by default.
- A number of features for free and a pay as you go rate for any additional features you may wish to utilise.
Is it perfect? Of course not, but what is? The monitoring is not strictly real-time as CloudWatch buckets metrics into intervals, but there is often a small delay between a monitored event and the human reaction to it anyway. Also, some competitors claim to have inbuilt ML, although AWS has a massive array of ML tools allowing the possibility of any required ML functionality to be implemented within the same ecosystem.
Although other tools may claim to have more monitoring out of the box and we have had to implement some of the monitoring above from scratch, I would argue that CloudWatch’s customisability is one of it’s main strengths, especially given potential fees with the ‘out of the box’ alternatives. Finally, CloudWatch is a highly scalable tool perfectly able to cope with any amount of data ingress.
Here at Version 1 we are trusted to reduce complexity and deliver the right Cloud based services for your organisation. We aim to partner with you on your Cloud journey to help you to discover the business benefit of our Cloud services and why it is important to get on board with monitoring in the cloud.
About Version 1
Version 1 isthe chosen Digital Services partner of central governmental departments, local government authorities, leading energy companies and major retailers, the impact of Version 1 digital solutions is evident at scale across multiple industries. We are leading the charge in empowering our public and private sector customers to become recognised as digital leaders within their individual verticals.
News & Insights
Blog: August 16, 2020
I think of kdb+ as the swiss army knife of technologies. Not only is it a database, but with its built-in webserver and own programming language you can do almost anything that you want with it. But with...
Blog: June 1, 2020
In the latest of our ‘kdb+ guides for Beginners’ series, we’ve decided to tackle kdb+ Amend/Apply and Error Trapping. Whether we’re working with tables, lists or dictionaries, sometimes we want...