What is web crawling and google indexing

Processing of information in Google search

Before you enter a search query, web crawlers collect information from billions of websites and sort them in the search index.

Google search basics

The crawling process begins with a list of sites from previous crawls and sitemaps that site owners have submitted. When our crawlers call up these websites, they follow the links to other pages provided there. In particular, the software checks new websites, changes to existing websites and outdated links. The crawling software also defines which websites are searched when and how often and how many individual sub-pages are accessed from each website.

Using the Search Console, website owners can specify exactly how Google should crawl their website: they can provide detailed information about the individual pages of their website, ask Google to crawl the URLs again or crawl the website using a file called "robots.txt" prevent. Google doesn't take payments to crawl websites more often. We offer the same tools to everyone so that our users get the best results.

Find information by crawling

The Internet is like a constantly growing library with billions of books, but without a central cataloging system. With the help of software, the so-called web crawler, we search for publicly available websites. The crawlers visit the websites and follow the links published on them - similar to how you do when surfing the Internet. They jump from link to link and send the information behind them to the Google servers.

Organize information through indexing

When crawlers find a web page, the page content is called up like in a browser. They analyze the most important elements - from the keywords to the topicality of the website - and note them in the search index.

The Google search index includes billions of websites and is over 100,000,000 gigabytes in size. It works like the index at the end of a book: every word that crawlers have found on an indexed website receives an entry. When we index a web page, we add it to all entries for words that appear on that page.



With the Knowledge Graph, we're continuing our endeavors not only to match keywords, but to better understand the people, places and things that are important to you. To do this, we not only index information about websites, but also other types of information. For example, Google Search can help you search through texts in millions of books from large libraries, find timetables for public transport, or call up data from freely accessible sources such as the World Bank.

This is how the search works: Overview