Luigi, Airflow, Pinball, and Chronos: Comparing Workflow Management Systems

Building large scale systems that deal with a considerable amount of data often requires numerous ETL jobs and different processing mechanisms. In our case, for example, the ETL process consists of many transformations, such as normalizing, aggregating, deduplicating and enriching millions of car data records. These kinds of processes generally start by running manual flows, […]

How to Count Large Scale Geohashes

As part of my team’s effort to crunch geospatial information, we needed to find a way to accurately count the number of geohashes in any given area. Knowing how many geohashes there are in any given area lets us extract information per region and compare areas with each other, for advanced analyses. This blog post […]

Deleting Code Matters

Starting out the development of a product is very exciting. You’re able to focus your attention on all the small details and you dedicate time for designing and thinking forward. It’s your chance to do it right. You can do things just the way you want it to be done – using short and readable […]

Redis, Kafka or RabbitMQ: Which MicroServices Message Broker To Choose?

When using asynchronous communication for Microservices, it is common to use a message broker. A broker ensures communication between different microservices is reliable and stable, that the messages are managed and monitored within the system and that messages don’t get lost. There are a few message brokers you can choose from, varying in scale and […]