Currently Crawlbot does not interact with sites to retrieve or pursue links that appear when a page is scrolled — so-called “infinite” or “endless” scrolling. Crawlbot will only pursue links that are available upon an initial page load.
In most cases sites will offer alternative means to find the same links:
- related links (to other posts or products) on individual post or product pages
- search filters or category links that narrow the number of results
- a sitemap file (e.g. sitemap.xml) or similar map to individual item pages
If you find a site that is unable to be crawled without page-scrolling, you may be able to improve results via the following approach:
- Store your X-Evaluate header as a custom rule against the Analyze API for the site in question.
For assistance with the above, feel free to contact us at firstname.lastname@example.org.