Batch Processing: A Tutorial on Workers, Queueing and Gelato


Batch processing is one of the earliest ways of data processing, utilized by Herman Hollerith’s Tabulating Machine in 1890. Batch processing was developed to take advantage of scarce computing resources: it avoids idling these expensive resources by queueing instructions to process data without manual user intervention, and can shift workload to times when resources are less scarce1.

Today, we can leverage modern architectural patterns like worker systems, message queues and the cloud to level-up these advantages and simplify our code. Let’s look at an example of queueing and workers using a calorie-dense metaphor: gelato.


Using our favorite local gelato shop as an example, we explore how architectural concepts like queueing and workers can affect a given task. We chose gelato because:

  • Each order takes time to set up. You must examine the menu and display, choose and order.
  • Each order takes time to process. The time spent preparing an order can vary based on the order size and complexity, just as job size can vary in a worker system.
  • Adding additional workers helps. The queue of customers will be processed faster, in the same way that adding more workers to a particular batch processing job will make the queue shrink faster.
  • Like your infrastructure, gelato must be kept cold, and is delicious when consumed2.

Check it out:


2This is not true. assumes no responsibility if you eat your infrastructure.

Best Practices and Anti-Patterns for Workers and MQs

Best Practices and anti-patterns for workers and message queues

Thanks to Ruth Hartnup for the base image! CC BY 2.0

If you’ve been programming for a while, it’s probable that someone, somewhere, has recommended the Gang of Four book.

The book dissects Object Oriented programming. It lists numerous ways of royally messing things up, but it’s claim to fame is that it also lists ways to do it right! These well tested paths to success often come with explanations for when to use them and why they’re good at avoiding common pitfalls.

These are design patterns. They’re embedded in the culture of programming, and they’re an amazing way to learn from others’ mistakes. At the outset of any project, a lot of paths are open to you. Design patterns illuminate the dark paths from the healthy, low stress approaches.

Today, we’re releasing our very own set of best practices and anti-patterns in the form of a white paper. It’s a quick read and will save you time as you tinker on your own workers and message queues. So, what do you have to lose?


Message Queues & Workers: the Heart of Modern Infrastructure

Message Queues + Workers

Thanks to Sonny Abesamis for the base image! CC BY 2.0

Increasingly, message queues and workers are intertwined with the language of modern infrastructure. You might rely on explicit solutions like IronMQ or IronWorker. You might not. Whether you do or don’t is irrelevant: MQs and workers are in everything these days.

MQs and workers are hidden heroes, quietly powering a lot of the technology that many of us rely on. They’re core components in programming languages, MVC frameworks, and even web servers.

As a result, when making infrastructure decisions a good understanding of both MQs and workers is essential. The white paper below will take you from a fuzzy understanding to a well-versed conceptualization for both MQs and workers.

Download the latest white paper from a Refresher on Message Queues & Workers.
Give it a read and let us know what you think!

Smart Endpoints. Smart Pipes. Smarter Microservices.

This is the second post in a series on microservices. Read our previous post on the service computing environment: The Ephemeral Life of Dockerized Microservices.

The use of microservices is a growing trend in application development because it provides an answer to the overhead incurred by monolithic applications. The practice of breaking apart application components into independent services introduces more moving parts to manage, however, requiring a robust communication method in order to keep services and data in sync. The idea is to decouple services without derailing your application.