What is crawling?
Crawling is how search engines discover web pages: automated programs visit pages and follow their links to find new and updated content.
Crawling is the first step in how a search engine works. Automated programs, called crawlers, bots or spiders, visit web pages, read them and follow the links they find to reach more pages. They come back over time to look for changes. A page that crawlers cannot reach, because no links point to it or because it is blocked, may never appear in search results.
Why it matters for your business: If search engines cannot crawl your pages, those pages cannot show up in search. Clear links between your pages and a sitemap help crawlers find everything that matters.
Related articles
More in Glossary
Still need help?
Ask our team