Robots Websites

Tool from ASAP Consulting s.r.o. for detailed user agent string analysis using an online form. Includes databases of browsers and robots.

Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.

Standard being developed on behalf of content publishers to communicate permissions information more extensively than is the case with robots.txt. Project documents, implementation and background information.

John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.

Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.

Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.

An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.

This large database lists user agents in categories and distinguishes between robots and browsers.

A searchable database of user-agents with information about their type, purpose and origin.

Contains a database of user-agents for crawlers, spiders, browsers; tools for user-agent lookup and tools for user-agent string search.