prognostica Forecasting Engine - Behind the Scenes

In addition to the actual forecasting and report generation, several interesting processes and technologies are running in the background of our demonstration app. We would like to introduce them briefly.

After the browser has passed the time series data to our REST API, the actual processing takes place through several microservices. In a somewhat simplified form it looks like this:

First we describe our infrastructure and then the microservices and their communication to Kafka. This architecture is able to run on premise, private & public clouds.

Infrastructure

The stack runs in a hyperconverged infrastructure. Every component is scalable (horizontal/vertical). The foundation is a Ceph Storage Cluster consumed by many VMs (KVM) for block storage and by our microservices for s3 Object Storage.

KVM Virtual Machines (QEMU, libvirt) are running on top, which are used as Docker nodes and Kafka VMs.

The scalable, fault tolerant, and hyperconverged approach gives the stack the resilience we need to provide best service to users and reduce operational work in the background.

All technologies used are 100% Open Source and powered by a great worldwide community of contributors and users.

More than that, the deployment is fully automated with Gitlab CI, which guarantees a reliable build, test and deployment process to release early and often.

Technologies:

• Ceph
• KVM
• Apache Kafka
• Docker
• Gitlab

Microservices

By loosely coupling the individual components with the distributed streaming platform Kafka, we can choose the most suitable programming language for each microservice.

For data preparation and forecasting, we use our own powerful Java library Lighthouse, which offers advanced methods for time series processing and forecasting. In case of data preparation, this includes, for example, level shift detection and outlier correction. Various methods are available for forecasting, such as exponential smoothing. In combination with the powerful Kafka interfaces in Java, such as Consumer, Producer and Streams API, this quickly results in event-driven microservices.

The real-time creation of the report is performed using the Julia language for numerical computing. This involves fetching the data from Kafka, drawing the interactive forecast chart using the powerful plotly.js JavaScript graphing library and, writing the report in Julia Markdown via Weave.jl to html. The resulting file is stored to an AWS S3 file system while forwarding the according information via Kafka to the web frontend.

The interaction between the user frontend and the events in Kafka is handled by other Java microservices, which open a REST API to the outside world with Spring Boot. Time series can be posted here and the status and content of the report can be queried.