Rate Limiting Your Workers to Play Nicer with Other API’s

Many third party API’s have rate limits to ensure you don’t overuse or DoS them. Twitter for instance has the following rate limits:

  • Unauthenticated calls are permitted 150 requests per hour. Unauthenticated calls are measured against the public facing IP of the server or device making the request.
  • OAuth calls are permitted 350 requests per hour and are measured against the oauth_token used in the request.

Now that’s not a lot of requests! 2.5 requests per minute if unauthenticated. So how can we slow down/throttle our massive queue of Twitter related tasks in IronWorker so we don’t hit that limit?


Using IronWorker’s ‘delay’ parameter for Rate Limiting
IronWorker supports a ‘delay’ parameter (docs) when you are queuing up a task which allows you to specify a time to wait before the task will be eligible for execution.


So let’s use Twitter for example, if we have 10,000 tasks and each task will hit twitter once, we can skew our tasks to ensure we don’t hit the rate limit. If we can do 2.5 requests per minute against the Twitter API, then let’s stick to 2 IronWorker tasks per minute (leave a little room just in case). So here’s how you could queue up those jobs:


And there you have it. A nice, simple way to throttle your workers.
  • http://www.blogger.com/profile/10020509907950963431 Luke

    This also works really well when scheduling tasks:
    start_at = Time.now
    10000.times do |i|
    worker = MyWorker.new
    worker.schedule start_at: start_at
    start_at += 30

  • http://www.blogger.com/profile/01398330633165910535 Travis Reeder

    Hi Luke,

    We recommend to use the ‘delay’ parameter on queue rather than using the scheduler to do these kinds of things because there is a limit on the number of schedules you can have and it’s a lot more efficient for us on the backend.

  • http://www.blogger.com/profile/00036754824872409374 Jared

    Question on pricing… Does the delayed time count as machine time?

  • http://www.blogger.com/profile/01398330633165910535 Travis Reeder

    Hi Jared,

    No it doesn’t count towards machine time. The delay time is free.