Our collector
If you run a website and found KelletLineCollector in your logs, this page says what it does.
What it fetches
Published public registers and datasets whose licence allows reuse: their metadata, and the file itself when it has changed. Nothing behind a login unless the publisher issued us a key.
How it behaves
- It names itself in every request, with a contact address in its User-Agent.
- It obeys robots.txt (RFC 9309), including groups that name it, and the Crawl-delay you set (up to 60 seconds).
- Where robots.txt blocks a path, it stays out, unless the publisher's own API documentation invites automated use or the site has given us written permission.
- It makes at most one request per second to a host, asks only for what changed (ETag and Last-Modified), and backs off when told to (HTTP 429 or 503).
- It never rotates addresses, disguises itself or works around a block.
Its name in robots.txt
Use the product token from the User-Agent, for example:
User-agent: KelletLineCollector
Disallow: /
It also obeys rules for its earlier name, GASUKCollector, so a change of name never gets round a block.
Contact
To ask about it, or to ask it to stop: data@kelletline.com. If you ask us to stop, we stop first.