80legs makes it even easier to crawl the Web like Google

80legs launched last fall with the goal of offering Web crawling for the masses — in other words, it offered relatively cheap technology to help companies browse and index data from the Web, as giant companies like Google and Microsoft currently do to create their search engines.
80legs’ cheap technology puts Web crawling, and the interesting data analysis it allows, within reach for small companies that couldn’t afford it otherwise. To advance that goal, 80legs later added a free version, and now it’s offering prepackaged Web crawls. That means businesses who want the data no longer need to design their own Web crawl, but can just buy one off the (metaphorical) shelf.
The Houston, Texas company will offer a number of “crawl packages” that cover things like social network profiles, shopping website product listings, blog articles and comments, semantic annotations of news or product sites, property listings, company directories, and more — all delivered in the widely-used XML format.
How would someone use the data? As an example, a shopping site could take the product listing crawl and use it as an efficient way to see what prices many of their competitors are offering. Another company could average all the prices and create a consumer price index, of sorts, for online retail. Companies could use the social networking data to do market research or identify employment trends.
Chief executive Shion Deysarkar gave me a quick demo, and it only took a few seconds to select and pay for the crawl you want. In comparison, he said it would have taken a day to set up the crawl from scratch. That also means you don’t have to worry about updating the crawl as websites come and go, because the service handles that for you. 80legs, meanwhile, benefits because it’s dealing with fewer customized requests. It can just run a set of regular web crawls and share the same data with a bunch of customers.
The company has raised $400,000 from Creeris Ventures, and it launched at the DEMO conference coproduced by VentureBeat. Deysarkar wouldn’t offer too many details about the number of customers, crawls, and so on, but he said “all the interesting metrics” are growing by between 5 and 10 percent every month.
The company says a typical crawl package will cost $350 a month and will produce between 10 and 20 million records per month.
Tags: search, Web crawling
Companies: 80Legs
People: Shion Deysarkar