(Internet) A software bot that autonomously follows connected paths such as links between web pages.
Subject Item
_:vb7332145
rdf:value
Crawler-based search engines have three major elements. The first is the spider, also called the crawler, which visits a web page, reads it, and then follows links to other pages within the site.
dcterms:bibliographicCitation
2008, Alex Michael, Ben Salter, Marketing Through Search Optimization, Routledge, →ISBN, page xii:
Subject Item
_:vb7332146
rdf:value
These serve as an algorithmic way to judge the “mobileness” of a site, and similar algorithms are likely to be used by the search crawlers.
dcterms:bibliographicCitation
2011, James Pearce, Professional Mobile Web Development with WordPress, Joomla! and Drupal, page 466: