Data Access & Collection · Restriction or exception
Scraping or crawling third-party sites must identify the client and obey robots.txt
These items continue the Prohibited Actions list, whose lead says customers may not use a service to, nor allow their users or any third party to use a service to, do each item. Customers may not use a service to access a third-party web property for web scraping, web crawling, web monitoring or similar activity through a web client that does not take commercially reasonable efforts to identify itself with a unique User Agent string describing the purpose of the web client, and to obey the robots exclusion standard (robots.txt), including the crawl-delay directive.
Supporting quotations
Access a third-party web property for the purposes of web scraping, web crawling, web monitoring, or other similar activity through a web client that does not take commercially reasonable efforts to identify itself via a unique User Agent string describing the purpose of the web client and obey the robots exclusion standard (also known as the robots.txt standard), including the crawl-delay directive
Qualifications & exceptions
- The standard is 'commercially reasonable efforts' to identify the client and obey robots.txt, not a flat ban on scraping.
- The item concerns third-party web properties.