FloppyKitMapBot

FloppyKitMapBot is the crawler behind KlickyMart, a public directory of how online stores work for software agents: whether an agent can reach the store, how checkout works, and which payment methods (including Bitcoin) the store accepts. KlickyMart is a FloppyKit project.

User agent

FloppyKitMapBot/0.2 (+https://klickymart.com/bot; bot@klickymart.com)

Robots.txt token: FloppyKitMapBot. It runs a standard headless Chromium browser without stealth plugins or fingerprint changes. It currently runs from a single cloud server.

What it does

• Reads /robots.txt before anything else and follows it. If robots.txt returns an error, 401, 403 or times out, it treats the whole site as off-limits and fetches nothing more.
• Reads the site's public terms page. If the terms prohibit automated access, it does not crawl the site and records only that fact, with a quote and link to the terms.
• Visits a small number of public pages (at most 10 per site, usually fewer), such as the home page, a product page and payment or help pages, at least 2 seconds apart, one site at a time.
• Records which payment methods the store's own pages state, with a quote and the page URL for each, and notes when a page is blocked or shows a challenge.

What it never does

• Log in, create accounts, place orders, enter payment details, or submit forms (adding an item to a cart is the furthest it goes, and only where robots.txt allows the cart).
• Solve or bypass captchas, challenges or other bot protection. A block is recorded as a finding and the crawler stops.
• Disguise itself, rotate IP addresses or use proxies.
• Read customer reviews, questions or comments as facts about the store.
• Collect personal data.

How to opt out

Add this to your robots.txt; the crawler will stop visiting on its next run:

User-agent: FloppyKitMapBot
Disallow: /

You can also disallow only certain paths. The opt-out and any blocks are noted in the site's record, so the directory shows that the site is not available to agents.

To ask for a correction, a re-check or removal, open an issue at github.com/FloppyKit/klickymart/issues. Or email bot@klickymart.com.

Source and data

Code, crawl rules and record format: github.com/FloppyKit/klickymart.