Implementing a cloud-native target architecture
We transitioned the infrastructure from the Digital Ocean setup with self-managed Kubernetes clusters towards managed services on Google Cloud Platform, leveraging many of GCP’s autoscaling benefits.
Event-driven processing of sensor data
We opted for Pub/Sub for reliable and real-time data queueing, transitioning from what was previously a Kafka queue which created great overhead and significant costs. The Pub/Sub que also serves back-up purposes and handles the distribution to multiple subscribers.
Leveraging cloud-native components for auto scalability
We set up multiple Cloud functions to process, format and push measurements into a managed InfluxDB for real-time analytics, which was a significant optimisation compared to the prior implementation where servers were waiting for events 24/7, triggering permanent costs even when nothing was happening.
Setting up monitoring instruments to assure quality
As we were transitioning from an old towards an entirely new architecture, we needed to assure a frictionless and error prone transitioning. We set up a structural logging system for all device types to keep track of the rate of measurements per device, potential drops and the appropriate alerts if something goes wrong.