|Too many lemmings and you’ve got a problem.|
Being able to limit how many worker tasks can run at once has been one of the most requested features of IronWorker and it is available now! There are many reasons this feature is so important, here are a few of the more common use cases for it:
- Limit load on your database. If you know your database can handle X tasks at a time, set max_concurrency to X and then queue up as many tasks as you want.
- Limit requests on third party API’s to stay under rate limits such as the Facebook API and Twitter API. The Twitter API for instance, has a 150 requests per hour limit.
- Limit requests on your own API’s. This is essentially the same as #1.
- Limit requests on third party sites if you are web crawling or scraping so you don’t get blacklisted.
- Eliminate issues related to spiky behavior. For instance, let’s say you got TechCrunched and your system wasn’t ready for it, your app/site would go down, you’re customers would be upset and you wouldn’t be able to take orders or do whatever it is your site does. If you had offloaded your work to IronWorker to begin with and set a max_concurrency on the tasks, then when you got a spike from being TechCrunched or perhaps the Colbert bump, you would be perfectly fine. Smooth sailing.
As you can see, it opens the door to a lot of new use cases.
How to Use max_concurrency
|This worker has max_concurrency of 250, you can see that only 250 of 1000 tasks are running.|