About the Page Lookout scanner

What it is

Page Lookout is a service that scans websites and reports technical findings: security headers, performance, accessibility, technical SEO, and how a page handles cookie consent.

The scanner visits a site only because a Page Lookout customer added that site to their organization. The visits happen when the customer starts a scan, or on the schedule they set. It does not roam the web: it stays within the sites customers have added.

How it identifies itself

Requests made over plain HTTP send this as their User-Agent. Browser-based checks append the same token to the User-Agent text of the Chrome browser they use, so the token always reads the same:

PageLookout/<version> (+https://pagelookout.com/bot)

Browser-based checks

Some checks, such as accessibility, cookie and consent observation and visual changes, load the page in a Chrome browser, the way a visitor would. They run the page scripts and load its resources like a normal browser, and they do not hide that they are automated: the same token is appended to the browser's own User-Agent, after the Chrome text.

The consent check visits a page more than once to see what it does before any choice and after the banner is accepted or rejected, so you may see consent choices made by the scanner in your own records.

Addresses

The scanner runs on Google Cloud in the EU (europe-west1) and does not use fixed outgoing IP addresses. We do not publish an address list and do not promise one, so recognize the scanner by its User-Agent and not by IP address. The token to match on is PageLookout.

Request rate

Every scan has a budget set by the customer: a maximum number of URLs, a crawl depth and a request rate. While the scanner discovers pages, it limits its request rate for each site. Unless the customer sets another rate, that is at most two requests a second.

robots.txt

When the scanner discovers pages from sitemaps and links, it reads your robots.txt and follows the group for PageLookout, or the group for * when there is none for it. A URL that robots.txt disallows for it is neither fetched nor offered for scanning. If robots.txt cannot be read because the server is failing, the scanner looks at the starting page the customer added and goes no further.

How to block it with robots.txt

To keep the scanner out of the pages it discovers, add this to robots.txt:

User-agent: PageLookout
Disallow: /

Or block it by User-Agent

To refuse every request, including the browser-based checks, block requests whose User-Agent contains PageLookout in your web server, CDN or firewall. The customer who added your site will then see failed scans in their results.