Crawlers (or bots) are used to gather information obtainable on the internet. By utilizing web site navigation menus, and studying inside and external links, the bots begin to know the context of a web page. Of course, the words, images, and other information on pages additionally help search engines like google and yahoo perceive what they’re ab