Concatenating multiple pages with a custom API

Diffbot’s Article API automatically concatenates multiple-page articles… but what if you want to concatenate multiple pages of data for a custom API?

You can do this by creating a custom field called “nextPage” and providing it with the link/URL to the page you want to append:

  1. Create a new custom field named nextPage.
  2. Select the element that contains the link to the next page.
  3. Add an “attribute” filter using the Filters drop-down, and in this field enter HREF to make sure the URL value is returned. (Note that in some Javascript-based pagination, this URL value may not be available or may be available in a different attribute.)

Your Custom API will subsequently attempt to follow up to 20 pages of “nextPage” links, returning all available data in a single response.