The /Scale crawler

/Scale is a marketing analysis service. Companies use it to check their own website and, now and then, to look at a public page of another site in their market.

Two programs fetch pages for it:

• SlashScale-Crawler reads a company's own site, a part of it at a time until it has read every page it knows of, and then each page again about once a week. One run fetches at most 1,500 pages by default and never more than 5,000, with at most four requests at a time and a pause after each. A run starts once a day, or when someone at the company asks for one.
• SlashScale-Agent opens single public pages in a browser, when a person or an analysis asks for that page.

Both follow robots.txt, never sign in, never submit a form and never fetch private addresses. To keep them away from a site, add this to its robots.txt:

User-agent: SlashScale-Crawler
Disallow: /

User-agent: SlashScale-Agent
Disallow: /

SlashScale-WordPress reads only a WordPress site whose owner connected it in /Scale: the published posts and pages and their categories and tags, through WordPress's own REST API, about once a day. It never changes anything. It signs in only when the owner gave it an application password, and the owner ends it by disconnecting WordPress in /Scale or by revoking that password.

SlashScale-WooCommerce reads only the WooCommerce shop whose owner connected it in /Scale: counts and totals of its orders, through WooCommerce's own REST API with a read-only key the owner made, about once a day. It never changes anything, and the owner ends it by disconnecting WooCommerce in /Scale or by revoking that key.

Questions or a problem with our requests: hello@slashscale.com.