Too Many Collections Error

If you ever get the “Too Many Collections” error trying to create a new bulk job or crawl job, either via the API or the UI in your account’s dashboard, you’ve reached the limit our system supports: 1000 jobs.

This is why there is a timeout period after which inactive crawls will be removed – to make room for newer crawls.

If you have the need for more than 1000 active crawljobs, you have two options:

  1. Upgrade to a Plus or higher plan and we’ll introduce a method to your account which circumvents this limit, as needed.
  2. Delete some of your crawljobs and try again – you will find that you’ll immediately be able to create new crawljobs again.

We advise you to keep an eye on your active crawls and delete them as they are no longer needed. If the data has become stale or you have downloaded, you are encourage to remove the crawl. Likewise, if you have already downloaded the data but would like to keep the metadata (crawl settings) around, you can download these settings by using the Crawljob API. Such a backed up JSON file is enough to restore your crawljob’s settings even after years of inactivity on your accounts.

Happy crawling!