Back to All

2016-10-24

October 24th, 2016 by Jerome Choo
  • The diffbotUri value for Custom APIs is now calculated from the entirety of custom content (and can be used to detect changes between extractions).
  • Various improvements to caption detection and parsing in the Article API.
  • Crawlbot now adheres to the "Diffbot" user-agent in robots.txt directives, so that our crawling can be whitelisted when crawling partner or other sites.