Rate Limiting Your Workers to Play Nicer with Other API’s

Many third party API’s have rate limits to ensure you don’t overuse or DoS them. Twitter for instance has the following rate limits:

  • Unauthenticated calls are permitted 150 requests per hour. Unauthenticated calls are measured against the public facing IP of the server or device making the request.
  • OAuth calls are permitted 350 requests per hour and are measured against the oauth_token used in the request.

Now that’s not a lot of requests! 2.5 requests per minute if unauthenticated. So how can we slow down/throttle our massive queue of Twitter related tasks in IronWorker so we don’t hit that limit?
Using IronWorker’s ‘delay’ parameter for Rate Limiting
IronWorker supports a ‘delay’ parameter (docs) when you are queuing up a task which allows you to specify a time to wait before the task will be eligible for execution.
So let’s use Twitter for example, if we have 10,000 tasks and each task will hit twitter once, we can skew our tasks to ensure we don’t hit the rate limit. If we can do 2.5 requests per minute against the Twitter API, then let’s stick to 2 IronWorker tasks per minute (leave a little room just in case). So here’s how you could queue up those jobs:
And there you have it. A nice, simple way to throttle your workers.