LookSmart Looking to Crawl Every Page on the Web Every Day
LookSmart in January acquired Grub.org, which relies on the power of individuals' personal computers to search Web pages when their computers are idle. Juiced with their computing power, Grub sends results back to LookSmart to let it know whether a page on a site has been updated, added, or been expunged. The technique, called distributed computing, to search or crawl the Web is similar to that employed in the screen saver offered by SETI.org, the Search for Extraterrestrial Intelligence. This academic endeavor relies on volunteers to sign on and agree to let their idle computers filter radio signals from outer space in the pursuit of signals from intelligent life. LookSmart explains that distributed crawling technology enables them to get the community and Internet users to pitch in and help build a huge search index. Their goal is to be able to crawl every page on the Web every day, eventually creating a central registry of all Web addresses. [Full story: Human power helps build search index - Oakland Tribune]
Related Posts: