Google Caffeine: Google’s New Search Engine Index

Google has just unveiled a “secret project” of “next-generation architecture for Google’s web search“. This new architecture appears to include crawling, indexing, and ranking changes. For the first time, Google isn’t simply incorporating these changes into their existing infrastructure or replacing it. Instead, they’re providing a developer preview and are asking webmasters and power searchers to try it out and give them feedback. Unlike Google’s now-defunct SearchMash, which was intended for search experiments that wouldn’t necessarily be incorporated into Google’s main web search, the caffeine index seems to be an entirely new search infrastructure that will repace what exists now.

With Caffeine, as Google crawls the Web, it immediately indexes the updated information. Google had been crawling a fraction of the Web every night and then indexing the new information in a batch. Before that, Google was updating its index every 30 days, and initially it only did so every four months.