What is a “Spider” or “Crawler?”
Google's spider nicknamed "GoogleBot" is designed to scan through the various webpages on the Internet and compile them for later processing. The function of GoogleBot is to provide more relevant results in the SERP's (Search Engine Results Pages), validate HTML code and linking, eliminate as much spam as possible, and generally deliver quick results.
How Does GoogleBot Work?
Simply defined, the update process with GoogleBot works like this: Google starts with a giant list of websites that they already know exist (because they have indexed them before). From there, it scans these websites and identifies new links which they haven't seen before and follow those links and index those new pages as well.
And as you can imagine, it grows exponentially from there.
When the spider reaches the website, it automatically navigates through the site, searching for keywords, tagging meta tags, and navigating the inbound and outbound links as well as the various components of the site. As Googlebot visits and crawls through the website, the software is essentially forming a snapshot of the website and all its individual web pages at that particular point in time. That snapshot or “memory” of the website and its individual web pages is then cached.
The cached information is then added to Google’s memory banks, also known as the index. The index is Google’s “memory,” and when a visitor types in a search term, Google searches its memory for websites and web pages that fit the bill. At different intervals, Googlebot will go back to the websites in their index. GoogleBot will "crawl" the website again, take in the new data, and form the new snapshot of this page. This new snapshot is then added to the index in Google which then updates Google’s memory of your site.
How Does Google’s Spider Affect Websites?
The best websites on the internet are dynamic and "evergreen" (always changing, updating, etc). But of course there is a lag between the time a site is changed and the time when the GoogleBot finds it. In other words, it takes time for Google to know about when a page has changed. When conducting a search on Google, the search results reflect the information that was available only during Googlebot’s last crawl of the site. If recent changes were made to your website before Google had a chance to crawl the site, then the results from the spider will not be effective until the next crawl.
A lot of companies try to "trick" the GoogleBot in a number of ways, through doorway pages, cloaked pages, etc, but most of these tricks are short lived. We never have encouraged "spamming" techniques with Google even though in the past it was much easier to trick their algorithms than it is today....and it's only getting harder. So, the bottom line is you're much better off playing the game straight and giving Google what they want rather than employing some shortcut which will only come up hurting you in the long run.
Avoiding Spider Bites
If you’re running a website that attempts to trick the bots in order to draw traffic to advertising or some other types of promotions, you WILL get bit and penalized.
Your best bet will always be to learn what Google wants (which is UNIQUE and VALUABLE CONTENT regarding your topic) and they will reward you greatly with free traffic (and money, if you're doing it right).
Note: Wanna be a GOOGLE BOT then refer to my next post
http://meaerien.blogspot.com/2011/08/become-goole-bot-rather-than-surfer.html
0 comments:
Post a Comment