A Birdseye View of kdb+ – Monitoring in the Cloud

Commercial systems involving kdb+ software, such as those in tier one banks, tend to be large, bespoke and mission critical so it’s important to get the right monitoring solution in place when monitoring in the cloud. The solution needs to be capable of dealing with large amounts of data, be highly customisable, responsive and have metrics that are both actionable and able to signal alarm states.

It must be capable of monitoring the system holistically from the level of the servers to the level of the internal kdb+ metrics like row count.

Naturally, it should be capable of collating all this information into a visually effective dashboard.

Dashboards like below can be used to monitor implementations at the server instance level and give the user an insight into the overall health of the virtual server in the cloud:

And dashboards like this offer users the ability to monitor process level statistics:

It also must have the ability to track kdb+ logs and extract insights from them whilst filtering them for potential issues so an alert can be raised, or an action automatically performed.

All of which should be displayed on a clear dashboard allowing you to cut through the noise and see exactly what you need with highly flexible content like the following dash:

With companies moving towards either a partial or full  deployment of cloud based systems, wouldn’t it be great if all this was cloud native? I know what you’re thinking and you’re right, my head IS in the clouds!

Introducing CloudWatch as a consideration for monitoring of systems involving kdb+ software. All of the above was done in AWS’ native monitoring tool from scratch and with relative ease. CloudWatch allows:

• Any system metric that can be represented as a double or file to be pushed to CloudWatch based on either the occurrence of an event or a schedule.
• Centralisation and consolidation of all monitoring for your system, not just the kdb+ software aspect.
• Storage and archiving of the kdb+ metrics and logs which persist long after the system is no more.
• Kdb+ metrics to be visualised in a wide variety of graph types in customisable widgets and displayed in dashboards.
• Alarms to be set and events to be created resulting in either automatic actions or alerts sent to your phone or email for example.
• Kdb+ log files to be queried by Logs Insights, the purpose-built query language for AWS CloudWatch Logs with data meaningfully parsed and extracted so it can be displayed or actioned upon.
• Data to be encrypted both in transit and at rest by default.
• A number of features for free and a pay as you go rate for any additional features you may wish to utilise.

Is it perfect? Of course not, but what is? The monitoring is not strictly real-time as CloudWatch buckets metrics into intervals, but there is often a small delay between a monitored event and the human reaction to it anyway. Also, some competitors claim to have inbuilt ML, although AWS has a massive array of ML tools allowing the possibility of any required ML functionality to be implemented within the same ecosystem.

Although other tools may claim to have more monitoring out of the box and we have had to implement some of the monitoring above from scratch, I would argue that CloudWatch’s customisability is one of it’s main strengths, especially given potential fees with the ‘out of the box’ alternatives. Finally, CloudWatch is a highly scalable tool perfectly able to cope with any amount of data ingress.