WbSrch Web Search Changelog

May 30, 2019: We've made some changes that will make domain detail pages significantly faster. [URLS: 23280000] [Index: 3090000] [Blocked Domains: 950000]

May 28, 2019: It's now possible to get the top 1 million most-linked-to domains via our data shop. We plan to update this monthly and add other data offerings in the future.

May 23, 2019: Version 1.02 - We've added Aragonese, Basque, Galician, and Welsh language indexes. [URLS: 21700000] [Index: 3090000] [Blocked Domains: 670000]

May 12, 2019: Our index is now over 20 million pages again. [URLS: 20000000] [Index: 3090000] [Blocked Domains: 580000]

April 17, 2019: Our index is now over 10 million pages again. [URLS: 10090000] [Index: 3090000] [Blocked Domains: 315000]

April 3, 2019: Version 1.01 - We've improved our handling of typos and correction suggestions will be better. [URLS: 5010000] [Index: 3090000] [Blocked Domains: 145000]

April 2, 2019: Version 1.00 - We've made some minor changes and improvements to our indexing algorithms. [URLS: 4980000] [Index: 3090000] [Blocked Domains: 144000]

April 1, 2019: WbSrch is once again running on dedicated hardware. Expect faster growth in the near future. [URLS: 4890000] [Index: 3090000] [Blocked Domains: 136000]

Feb. 3, 2019: We've re-added indexes for the Catalan, Croatian, Estonian, Latvian, Lithuanian, Romanian, Slovene, and Swahili languages. [URLS: 2080000] [Index: 2760000] [Blocked Domains: 44000]

March 15, 2018: The WbSrch search engine is online again with the same code, but starting over with crawling. [URLS: 10000]

June 30, 2016: The WbSrch search engine has gone offline.

June 22, 2016: Version 0.999 - We've moved Slovak-language pages into the Czech-language index so we now have a Czech/Slovak index. We've also moved Norwegian and Danish-language pages into the Swedish-language index so we now have a Scandinavian language index. [URLS: 47400000] [Index: 3640000] [Blocked Domains: 785000]

June 15, 2016: Our German-language index has grown a lot over the past three months and is now over 10 million pages, nine times what it was before. [URLS: 47840000] [Index: 4390000] [Blocked Domains: 785000]

June 14, 2016: Version 0.998 - We've removed a few language indexes due to lack of demand. They are: Catalan, Croatian, Estonian, Hausa, Icelandic, Latvian, Lithuanian, Romanian, Rwandan, Shona, Slovene, Somali, Swahili, Wolof, and Yoruba. This will allow us to focus our efforts on improving our more popular language indexes. [URLS: 47800000] [Index: 4390000] [Blocked Domains: 785000]

June 3, 2016: We now have 45 million pages in our index. [URLS: 45000000] [Index: 4960000] [Blocked Domains: 785000]

May 20, 2016: We now have 40 million pages in our index. [URLS: 40000000] [Index: 4960000] [Blocked Domains: 785000]

April 30, 2016: Version 0.997 - We've made some changes to indexing that will cause some less desirable domain extensions to rank lower, along with many content-free and parked domains. [URLS: 32300000] [Index: 4950000] [Blocked Domains: 780000]

April 27, 2016: Version 0.996 - We've made some changes to our indexer that will improve results for searches that contain non-ASCII characters. [URLS: 31000000] [Index: 4950000] [Blocked Domains: 780000]

April 25, 2016: Version 0.995 - We've made some changes to our our web engine that will improve response times a bit thanks to Django Q and ujson. [URLS: 30130000] [Index: 4950000] [Blocked Domains: 775000]

April 24, 2016: We now have 30 million pages in our index, the same amount the Infoseek (Go) search engine had during the late 1990's. [URLS: 30000000] [Index: 4950000] [Blocked Domains: 770000]

April 20, 2016: Our Italian-language index has grown to three times its previous size and has more than 1 million pages. [URLS: 28400000] [Index: 4950000] [Blocked Domains: 760000]

April 15, 2016: Our Dutch-language index has grown to seven times what it was at the beginning of the month and has more than 2 million pages. [URLS: 26200000] [Index: 4950000] [Blocked Domains: 746000]

April 14, 2016: We now have 26 million pages in our index, the same amount the original Google had when it launched in 1998. [URLS: 26000000] [Index: 4950000] [Blocked Domains: 745000]

April 9, 2016: We've more than doubled the size of our French-language index and now have more than 1 million pages in the French language. [URLS: 24100000] [Index: 4950000] [Blocked Domains: 730000]

March 29, 2016: We've added a music search engine called MusicSrch that will help you find your favorite musicians online. [URLS: 21650000] [Index: 4940000] [Blocked Domains: 726000]

March 28, 2016: Over the past month, we've nearly tripled the size of our Portuguese-language index. [URLS: 21580000] [Index: 4940000] [Blocked Domains: 725000]

March 27, 2016: Over the past month, we've more than doubled the size of our Spanish-language index. [URLS: 21500000] [Index: 4940000] [Blocked Domains: 725000]

March 23, 2016: We've more than tripled the size of our Polish-language index and have more than 1 million pages in the Polish language. [URLS: 20400000] [Index: 4940000] [Blocked Domains: 725000]

March 22, 2016: We now have 20 million pages in our index. [URLS: 20000000] [Index: 4940000] [Blocked Domains: 725000]

March 21, 2016: We've increased the size of our Danish-language index by more than 500%. [URLS: 19940000] [Index: 4940000] [Blocked Domains: 725000]

March 20, 2016: We've increased the size of our Swedish-language index by more than 500%. [URLS: 19860000] [Index: 4940000] [Blocked Domains: 725000]

March 19, 2016: We've more than quadrupled the size of our Romanian-language index. [URLS: 19600000] [Index: 4940000] [Blocked Domains: 725000]

March 17, 2016: We've more than quadrupled the size of our Hungarian-language index. [URLS: 18540000] [Index: 4940000] [Blocked Domains: 725000]

March 16, 2016: We've increased size of our Slovak-language index by more than 60%. [URLS: 18460000] [Index: 4940000] [Blocked Domains: 725000]

March 15, 2016: We've increased size of our Turkish-language index by more than 60%. [URLS: 18420000] [Index: 4940000] [Blocked Domains: 725000]

March 14, 2016: We've more than doubled the size of our Greek-language index. [URLS: 18360000] [Index: 4940000] [Blocked Domains: 725000]

March 13, 2016: We've more than quintupled the size of our Czech-language index. [URLS: 18200000] [Index: 4940000] [Blocked Domains: 725000]

March 9, 2016: We now have eighteen million pages in our index. [URLS: 18000000] [Index: 4940000] [Blocked Domains: 725000]

March 5, 2016: We now have seventeen million pages in our index. [URLS: 17000000] [Index: 4940000] [Blocked Domains: 724000]

March 4, 2016: We now have more than 1 million pages in our Spanish-language index. [URLS: 16530000] [Index: 4940000] [Blocked Domains: 720000]

March 3, 2016: We've more than tripled the size of our Norwegian-language index. [URLS: 16070000] [Index: 4940000] [Blocked Domains: 718000]

March 2, 2016: We now have sixteen million pages in our index. [URLS: 16000000] [Index: 4940000] [Blocked Domains: 715000]

March 1, 2016: We've more than quadrupled the size of our Finnish-language index. [URLS: 15700000] [Index: 4940000] [Blocked Domains: 710000]

Feb. 29, 2016: We've more than tripled the size of our Lithuanian-language index. [URLS: 15320000] [Index: 4940000] [Blocked Domains: 701000]

Feb. 28, 2016: We've more than doubled the size of our Latvian-language index. [URLS: 15160000] [Index: 4940000] [Blocked Domains: 700000]

Feb. 27, 2016: We now have fifteen million pages in our index. [URLS: 15000000] [Index: 4940000] [Blocked Domains: 699000]

Feb. 26, 2016: We've more than doubled the size of our Estonian-language index. [URLS: 147800000] [Index: 4940000] [Blocked Domains: 695000]

Feb. 25, 2016: We've more than doubled the size of our Slovene-language index. [URLS: 14700000] [Index: 4940000] [Blocked Domains: 690000]

Feb. 24, 2016: We've increased the size of our Croatian-language index by 50%. [URLS: 14600000] [Index: 4940000] [Blocked Domains: 680000]

Feb. 22, 2016: We've more than doubled the size of our Catalan-language index. [URLS: 14430000] [Index: 4940000] [Blocked Domains: 672000]

Feb. 21, 2016: Version 0.994 - We've added autocomplete to our search box. It should work in all modern HTML5-compliant browsers. [URLS: 14400000] [Index: 4940000] [Blocked Domains: 670000]

Feb. 20, 2016: Version 0.993 - We've made some minor adjustments to how some domain extensions are treated by our ranking algorithms. [URLS: 14300000] [Index: 4940000] [Blocked Domains: 660000]

Feb. 16, 2016: Version 0.992 - When you search, the phrase you searched for will be shown in bold text in the results so you can more easily judge the relevance of each link. [URLS: 14080000] [Index: 4940000] [Blocked Domains: 652000]

Feb. 15, 2016: We now have fourteen million pages in our index. [URLS: 14000000] [Index: 4940000] [Blocked Domains: 650000]

Feb. 7, 2016: We're running a crowdfunding campaign on IndieGoGo to raise funds for hardware upggrades. Please help support the growth of WbSrch. [URLS: 13840000] [Index: 4940000] [Blocked Domains: 625000]

Jan. 31, 2016: Version 0.991 - We've made a change that improves results for never-before-seen multi-word searches, especially those for proper names. [URLS: 13500000] [Index: 4940000] [Blocked Domains: 623000]

Jan. 26, 2016: We've released our own custom web browser. It's called WbBrowse and it's available at wbbrowse.com. [URLS: 13360000] [Index: 4940000] [Blocked Domains: 622000]

Jan. 25, 2016: Version 0.99 - We've made some changes to our index structure that should improve multi-word phrase search, especially for combinations of words that are less common. [URLS: 13300000] [Index: 4940000] [Blocked Domains: 621000]

Jan. 24, 2016: Version 0.98 - We've made a change that should significantly improve search results for multi-word phrases. [URLS: 13290000] [Index: 4940000] [Blocked Domains: 620000]

Jan. 18, 2016: We now have thirteen million pages in our index. [URLS: 13000000] [Index: 4940000] [Blocked Domains: 619000]

Jan. 16, 2016: Version 0.97 - We've made a change that improves search results for multi-word phrases that have never been searched for. [URLS: 12960000] [Index: 4940000] [Blocked Domains: 615000]

Jan. 12, 2016: Version 0.96 - We've made some improvements to our indexer, including combining domains with a "www" prefix and those without into a single listing (links from www.example.com and example.com would previously show up separately in the same search result). [URLS: 12870000] [Index: 4940000] [Blocked Domains: 610000]

Dec. 29, 2015: Version 0.95 - We've made some changes to our indexer that will cause domains containing numbers to rank lower for non-numeric searches and sites with excessively long titles to rank lower. We've also made some minor improvements to the domain details page. [URLS: 12670000] [Index: 4939000] [Blocked Domains: 588000]

Dec. 18, 2015: Version 0.94 - We've made a few changes to our crawler that have improved its multilingual crawling ability. [URLS: 12570000] [Index: 4935000] [Blocked Domains: 580000]

Dec. 12, 2015: We now have more than 1 million pages in our German-language index. [URLS: 12400000] [Index: 4935000] [Blocked Domains: 577000]

Dec. 7, 2015: Version 0.93 - We've made an improvement to our crawler that will cause it to better handle redirects when re-crawling non-English-language URLs. [URLS: 12010000] [Index: 4930000] [Blocked Domains: 575000]

Dec. 6, 2015: We now have twelve million pages in our index. [URLS: 12000000] [Index: 4930000] [Blocked Domains: 575000]

Nov. 24, 2015: We've crawled our eleventh million URLs. [URLS: 11000000] [Index: 4930000] [Blocked Domains: 560000]

Nov. 14, 2015: Version 0.92 - When you search for a domain name, you'll also see a link to the domain details page if we have any info for that domain. The same goes for IP addresses. [URLS: 10070000] [Index: 4930000] [Blocked Domains: 542000]

Nov. 11, 2015: We've upgraded our infrastructure to a 3-server setup from the previous single-server configuration. [URLS: 10010000] [Index: 4900000] [Blocked Domains: 540000]

Oct. 20, 2015: Version 0.91 - We've made an improvement to the crawler so that XHTML pages look better in search results. [URLS: 9640000] [Index: 4820000] [Blocked Domains: 540000]

Oct. 18, 2015: Version 0.90 - We've changed search results to show no more than 200 results in total. The previous limit was 250 for indexed phrases, or 250 per word for unindexed phrases (a search like "large yellow socks" could return 750 results). [URLS: 9400000] [Index: 4810000] [Blocked Domains: 540000]

Oct. 5, 2015: Version 0.89 - We've made some improvements to our language categorization tools that will allow us to categorize page languages for a site on a page-by-page basis. [URLS: 8550000] [Index: 4775000] [Blocked Domains: 523000]

Oct. 3, 2015: Version 0.88 - We've made some pretty big improvements to how we analyze page text. [URLS: 8490000] [Index: 4770000] [Blocked Domains: 522000]

Oct. 1, 2015: Version 0.87 - We've made more improvements to our ranking algorithms, including improvements to how we handle page text and domain names. [URLS: 8480000] [Index: 4760000] [Blocked Domains: 520000]

Sept. 26, 2015: Version 0.86 - We've added an articles section and included a couple posts from an earlier blog. [URLS: 8260000] [Index: 4750000] [Blocked Domains: 495000]

Sept. 25, 2015: Version 0.85 - We've long had the ability to perform get the search results for a specific site by clicking on the "# of results from this domain" link in search results. Now you can search for a result from a specific domain using the "site:example.com example" syntax (it doesn't matter whether you put that at the beginning or end, so "example site:example.com" is the same). Searching just for "site:example.com" will show the domain info page for that domain. [URLS: 8140000] [Index: 4750000] [Blocked Domains: 495000]

Sept. 23, 2015: WbSrch is two years old today. [URLS: 8066000] [Index: 4750000] [Blocked Domains: 494000]

Sept. 22, 2015: Version 0.84 - We've added a basic IP information view that lets you see what domains and how many pages we've found at a particular IP address. [URLS: 8040000] [Index: 4740000] [Blocked Domains: 493000]

Sept. 21, 2015: Version 0.83 - We received our first ever DMCA takedown notice today. We've created a section of the website where we can post DMCA notices that we've received and have published it online. [URLS: 8000000] [Index: 4740000] [Blocked Domains: 493000]

Sept. 15, 2015: Version 0.82 - We've made minor changes to the indexer to tune how we treat links and subdomains. The rules are the same, but effects are now more fine-grained. [URLS: 7900000] [Index: 4720000] [Blocked Domains: 491000]

Sept. 10, 2015: Version 0.81 - We've started tracking more information about the objects on a web page - iframes, audio and video tags, JavaScript, images, and CSS files. None of this information is a ranking factor yet, but some of it probably will be in the near future. [URLS: 7890000] [Index: 4680000] [Blocked Domains: 488000]

Sept. 7, 2015: Version 0.80 - We've started using the character encoding of a page as a minor ranking factor. We now also look at the MIME type of a page and use that as a ranking factor. [URLS: 7850000] [Index: 4680000] [Blocked Domains: 488000]

Sept. 6, 2015: Version 0.79 - We've made a minor change to how our indexer scores URLs. [URLS: 7800000] [Index: 4670000] [Blocked Domains: 487000]

Sept. 5, 2015: Version 0.78 - We've made a few visual design changes, moving the language change flags to the bottom of the page. We've also changed the indexer to rank some top-level domain extensions lower based on their poor reputations. [URLS: 7750000] [Index: 4670000] [Blocked Domains: 487000]

Sept. 2, 2015: The number of terms indexed shown on our index stats page has dropped drastically over the past few weeks. We've pruned the list of words we index to remove empty searches. This won't affect your search results, but it will save us a bunch of CPU cycles. [URLS: 7700000] [Index: 4660000] [Blocked Domains: 485000]

Aug. 26, 2015: Version 0.78 - We've made a minor change to our crawler to improve its ability to spider secure (HTTPS) websites. We've always been able to crawl HTTPS sites, but now we make better decisions about what URL to crawl next. [URLS: 7670000] [Index: 4740000] [Blocked Domains: 484000]

Aug. 24, 2015: Version 0.77 - Domain detail pages will now give a general indication of how many inbound links we've found for a domain. Since we've only crawled a small fraction of the web so far, these details will necessarily be vague, like "not many" or "a lot", but it should give some indication of what data we have for that domain's links. [URLS: 7650000] [Index: 5430000] [Blocked Domains: 483800]

Aug. 23, 2015: Although our crawler now supports robots.txt, a lot of pages in our index were crawled before support was added. We removed a bunch of pages that would have been blocked by the robots file from our index. Any that appear in search results will gradually disappear over time as things are reindexed. [URLS: 7660000] [Index: 5930000] [Blocked Domains: 483800]

Aug. 22, 2015: Version 0.76 - Pages with a comma in the URL will now tend to rank lower in search results. Commas in a URL are bad form, so don't do it. [URLS: 7660000] [Index: 5930000] [Blocked Domains: 483800]

Aug. 21, 2015: Version 0.75 - We've made a minor change that will cause URLs with lots of query parameters to tend to rank slightly lower than clean URLs for the same content. [URLS: 7710000] [Index: 5930000] [Blocked Domains: 483600]

Aug. 18, 2015: Version 0.74 - We've changed the way we count page titles and descriptions in our indexing process to be a bit more nuanced. Page size will also be a factor, with pages that are very tiny or extremely huge ranking lower than those that are mid-sized. [URLS: 7700000] [Index: 6800000] [Blocked Domains: 483500]

Aug. 14, 2015: Version 0.73 - Some newer top-level domains will now tend to rank lower. Think twice before buying that ".link" domain. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 483000]

Aug. 13, 2015: Version 0.72 - Ranking info is now available on domain detail pages. It will show the top search keywords that a domain ranks for on WbSrch (up to 100). Not all ranked keywords are shown for a domain yet, but they'll fill in as our indexer cycles through the remaining keywords over the next few weeks. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 482500]

Aug. 10, 2015: Version 0.71 - We've excluded a few more file types from our crawler (media, images, source code) in order to build a cleaner index. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 482500]

Aug. 7, 2015: Version 0.70 - We've started keeping track of what web server sites use to serve content. It's not likely to be a ranking factor, but it'll allow us to track the usage share of different web servers. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 482500]

Aug. 5, 2015: Version 0.69 - Our crawler now follows robots.txt rules. The crawler agent has been updated to WbSrch/1.1 to reflect that. We're not popular enough for sites to write specific rules for us, so we'll follow the rules set for Google until we notice a statistically significant number of rules specific to the WbSrch user agent. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 482000]

Aug. 2, 2015: Version 0.68 - We've made some improvements to handling session IDs that will result in crawling fewer duplicate URLs. We've also removed a lot of duplicate URLs from our index. It's amazing the terrible things people can do to session IDs with a Tomcat server. [URLS: 7690000] [Index: 10680000] [Blocked Domains: 480000]

July 27, 2015: We lost 4 drives in a 6-drive RAID array and had to restore from backup. No code was lost, but crawl and indexer data for the last couple months is missing. We'll catch up quick. It's the code that's the hard part. We cannot recommend Mushkin SSDs. Spend the extra few dollars for Kingston or Samsung. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 436000]

July 18, 2015: Version 0.67 - We've added a page that shows the 100 most-linked-to domains based on the contents of our index.

July 17, 2015: Version 0.66 - We've started using the number of different domains linking to a domain as a minor ranking factor. We don't do any fancy calculations yet (nothing like PageRank or HITS, for example), we just score based on a raw count.

July 15, 2015: Version 0.65 - We've started tracking links on a page and will soon be using them as a ranking factor.

July 14, 2015: Version 0.64 - We've started tracking detailed information about where each domain ranks for specific keywords.

March 14, 2015: Version 0.63 - We've started reading robots.txt files. We don't take any action based on them yet, but if we're reading them, reacting to rules can't be too far behind. [URLS: 10090000] [Index: 10790000] [Blocked Domains: 474000]

March 1, 2015: We've crawled our tenth million URL and now have an index the size of the original AltaVista search index. [URLS: 10000000] [Index: 10780000] [Blocked Domains: 469000]

Feb. 24, 2015: We've crawled our ninth million URLs. [URLS: 9000000] [Index: 10780000] [Blocked Domains: 449000]

Feb. 23, 2015: Version 0.62 - The age of a domain will now be a factor in rankings. Brand new domains will tend to rank lower than domains that have been around for a while. [URLS: 8740000] [Index: 10780000] [Blocked Domains: 447000]

Feb. 10, 2015: Version 0.61 - Our language categorization ability has improved thanks to the inclusion of the langid library by Marco Lui. [URLS: 8120000] [Index: 10700000] [Blocked Domains: 440000]

Jan. 26, 2015: Version 0.60 - We've added an index for the Catalan language and now support 32 languages. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 436000]

Jan. 25, 2015: Version 0.58 - We've made some changes to the site that should make it generally faster and more responsive. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 435000]

Jan. 25, 2015: Version 0.59 - We've made some changes to our crawler that will cause it to crawl cleaner URLs, leaving off a lot of common session and referral IDs. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 436000]

Jan. 24, 2015: Version 0.57 - We've made some changes to our indexer that will cause it to look at more "H" tags when analyzing page titles. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 430000]

Jan. 23, 2015: Version 0.56 - We've made some improvements to our web crawler that will result in better-looking search results. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 430000]

Jan. 22, 2015: Version 0.55 - We've made some changes to the way search results are generated that will cause them to look better. The appearance of searches will improve gradually over time as keywords are re-indexed. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 430000]

Jan. 19, 2015: Version 0.54 - We've made some big improvements to our language categorization tools. URLs from more domains will end up in the right place. [URLS: 8080000] [Index: 10670000] [Blocked Domains: 430000]

Jan. 11, 2015: Version 0.53 - We've added an index for the Shona language and now support 31 languages. [URLS: 8000000] [Index: 10670000] [Blocked Domains: 424000]

Jan. 10, 2015: We've crawled our eighth million URLs. [URLS: 8000000] [Index: 10670000] [Blocked Domains: 420000]

Dec. 23, 2014: Version 0.52 - We've added an index for the Rwandan language and now support 30 languages. [URLS: 7720000] [Index: 10654000] [Blocked Domains: 330000]

Dec. 21, 2014: Version 0.51 - We've added an index for the Somali language and now support 29 languages. [URLS: 7700000] [Index: 10651000] [Blocked Domains: 329000]

Dec. 20, 2014: Version 0.50 - We've added an index for the Hausa language and now support 28 languages. [URLS: 7690000] [Index: 10650000] [Blocked Domains: 329000]

Dec. 18, 2014: Version 0.49 - We've added an index for the Wolof language and now support 27 languages. [URLS: 7640000] [Index: 10648000] [Blocked Domains: 328000]

Dec. 16, 2014: Version 0.48 - We've added an index for the Yorùbá language and now support 26 languages. [URLS: 7630000] [Index: 10645000] [Blocked Domains: 325000]

Nov. 23, 2014: Version 0.47 - We've made more mobile-friendly improvements to our site. [URLS: 7600000] [Index: 10645000] [Blocked Domains: 320000]

Oct. 28, 2014: We've crawled our seventh million URLs. [URLS: 7000000] [Index: 10640000] [Blocked Domains: 300000]

Sept. 23, 2014: WbSrch is now one year old. Time flies! [URLS: 6500000] [Index: 10640000] [Blocked Domains: 280000]

Sept. 5, 2014: We've crawled our sixth million URLs. [URLS: 6000000] [Index: 10640000] [Blocked Domains: 219000]

Aug. 12, 2014: We've improved our non-English-language indexes quite a bit. Searches in Spanish, Italian, and other languages will have much better results than before. [URLS: 5300000] [Index: 10640000] [Blocked Domains: 216000]

Aug. 5, 2014: We've served one million searches so far. This includes bot and crawler traffic.

June 10, 2014: Version 0.46 - We've fixed a few things that could generate server errors (500's), including quickly searching for the same thing twice in a row or being referred to the site from a gigantically long URL. [URLS: 5100000] [Index: 10640000] [Blocked Domains: 187000]

June 5, 2014: We've crawled our fifth million URLs. [URLS: 5000000] [Index: 10640000] [Blocked Domains: 175000]

June 4, 2014: Version 0.45 - The WbSrch website is now SSL-only. We're proud to support the ResetTheNet movement. [URLS: 4990000] [Index: 10640000] [Blocked Domains: 174000]

June 3, 2014: Version 0.44 - We've made speed improvements to some slower areas of the site, such as the "index stats" page and the "popular searches" pages. [URLS: 4950000] [Index: 10640000] [Blocked Domains: 170000]

May 30, 2014: Version 0.43 - We've made more progress improving our site language translations for non-English languages. [URLS: 4700000] [Index: 10640000] [Blocked Domains: 150000]

May 26, 2014: Version 0.42 - We've made some changes to our site design that will make it look better on low-resolution screens. [URLS: 4600000] [Index: 10640000] [Blocked Domains: 144000]

May 24, 2014: Version 0.41 - We've made changes to our indexer that will make searches for specific domains far better. [URLS: 4550000] [Index: 10640000] [Blocked Domains: 143000]

May 23, 2014: We've created more search plugins and now have one for each language that we support. You can get them from the Mycroft Project website. [URLS: 4500000] [Index: 10640000] [Blocked Domains: 143000]

May 22, 2014: Version 0.40 - We've made some significant changes to our indexer that will improve search relevance and have much better results for multi-word phrases. [URLS: 4500000] [Index: 10640000] [Blocked Domains: 140000]

May 18, 2014: Version 0.39 - We've made some improvements to multi-word searches. [URLS: 4400000] [Index: 10630000] [Blocked Domains: 137000]

May 10, 2014: Version 0.38 - We've added the ability to block search phrases and have blocked a few dozen terms that nobody has any business searching for, such as "child porn" and some related searches. [URLS: 4120000] [Index: 10630000] [Blocked Domains: 126000]

May 7, 2014: We've crawled our fourth million URLs. [URLS: 4000000] [Index: 10630000] [Blocked Domains: 125000]

April 30, 2014: Version 0.37 - We fixed a bug that caused searches containing capital letters to return worse results than those all in lowercase. [URLS: 3750000] [Index: 10630000] [Blocked Domains: 122000]

April 25, 2014: Version 0.36 - We've made some improvements to the visual design of the WbSrch website. [URLS: 3480000] [Index: 10630000] [Blocked Domains: 120000]

April 10, 2014: Version 0.35 - We've made some improvements to our crawler with regard to handling errors. The variety and quantity of errors you can encounter on the web is amazing. We do not yet properly handle the "I'm a Teapot" error. [URLS: 3150000] [Index: 10630000] [Blocked Domains: 117000]

April 3, 2014: Our HTML parser is much better today than it was when we first started. We've done a lot of recrawling of sites that were crawled early in our existence. Search results will look better than they have in the past. [URLS: 3090000] [Index: 10630000] [Blocked Domains: 116000]

March 16, 2014: We've crawled our third million URLs. [URLS: 3000000] [Index: 10300000] [Blocked Domains: 113000]

March 12, 2014: Version 0.34 - We've added a popular searches view that lets you see what was searched for most often in a given month. [URLS: 2970000] [Index: 10630000] [Blocked Domains: 107000]

March 3, 2014: Version 0.33 - We fixed a nasty bug that caused international search results to be far lower quality than searches in English. [URLS: 2890000] [Index: 10630000] [Blocked Domains: 105000]

March 2, 2014: Version 0.32 - We've made some improvements to our crawler, many of them focused on re-crawling pages that haven't been crawled in a while or handling some of the less common HTTP errors. [URLS: 2880000] [Index: 10630000] [Blocked Domains: 105000]

Feb. 21, 2014: We've finished indexing the 10.6 million most common words across the 25 languages that we support. [URLS: 2740000] [Index: 10630000] [Blocked Domains: 99000]

Feb. 17, 2014: Version 0.31 - We've made some minor improvements to indexing of multi-word search phrases. They still need a lot of work, but are a little better now. [URLS: 2640000] [Index: 10590000] [Blocked Domains: 99000]

Feb. 13, 2014: Version 0.30 - Now that search results are grouped by domain, we've started taking the number of results from a domain for a particular search term into account in our rankings. We've also made some improvements to our admin tools so we can manage domains and URLs more effectively. [URLS: 2590000] [Index: 10300000] [Blocked Domains: 94000]

Feb. 12, 2014: Version 0.29 - We've changed the way search results are displayed. Instead of displaying everything all at once with multiple results from the same domain strewn about haphazardly, we group them together and show the best result for that search. There's now a link you can click to show all of the results for that domain for the search. Just click on the (X results for this site) to see them. [URLS: 2580000] [Index: 10300000] [Blocked Domains: 94000]

Feb. 9, 2014: Version 0.28 - We've made some improvements to our page scoring logic that include taking the length of the URL into account. If a page has a URL that's too long for a Human to type it'll rank slightly lower than a shorter one. [URLS: 2540000] [Index: 10100000] [Blocked Domains: 93000]

Feb. 8, 2014: We've finished building our search keyword index using our new fast-retrieval method. Search results will now be faster and better organized. [URLS: 2540000] [Index: 10000000] [Blocked Domains: 93000]

Feb. 2, 2014: Version 0.27 - We've come up with a method of indexing search keywords that allows for much faster retrieval. It will take a while to build new indexes for the 10-million-plus search terms we track, but searches will be quicker for those that have the new indexes. [URLS: 2500000] [Index: 9950000] [Blocked Domains: 93000]

Jan. 30, 2014: Version 0.26 - We've added more capabilities to our language detection tools by interfacing with NLTK, the Natural Language Toolkit. [URLS: 2490000] [Index: 9740000] [Blocked Domains: 93000]

Jan. 27, 2014: Version 0.25 - We weren't originally keeping track of the IP address that serves a web page. It turns out that is useful information that can be used in classification and ranking, so we're storing that now. [URLS: 2420000] [Index: 9500000] [Blocked Domains: 93000]

Jan. 19, 2014: Version 0.24 - We've created some basic tools that let us track which sites are the sources and destinations of spam comments and links. We'll be using those to rank spam-generating sites lower in the index or exclude them entirely. [URLS: 2290000] [Index: 8300000] [Blocked Domains: 92000]

Jan. 17, 2014: We've now indexed eight million search terms in the 25 languages that we support. [URLS: 2250000] [Index: 8000000] [Blocked Domains: 91000]

Jan. 3, 2014: We've crawled our second million URLs. [URLS: 2000000] [Index: 5100000] [Blocked Domains: 89000]

Dec. 29, 2013: Version 0.23 - We've made even more improvements to our language categorization tools and now have more than 370,000 URLs categorized as non-English languages. There's still a lot of work to be done, but our German, French, Spanish, Polish, and Portuguese indexes have recently become a lot more robust. [URLS: 1820000] [Index: 4500000] [Blocked Domains: 77000]

Dec. 27, 2013: Version 0.22 - We've implemented optimizations to our indexing algorithm that make it 30% faster. [URLS: 1650000] [Index: 4100000] [Blocked Domains: 75000]

Dec. 26, 2013: We've recently been redlining our CPUs indexing search terms and now have more than 3.7 million terms indexed. [URLS: 1570000] [Index: 3700000] [Blocked Domains: 75000]

Dec. 23, 2013: Version 0.21 - We've made more improvements to our language categorization tools and now have more than 200,000 URLs categorized as non-English languages.

Dec. 20, 2013: Version 0.20 - We've made improvements to our crawler that will reduce bandwidth use and avoid crawling some non-HTML object types. [URLS: 1240000] [Index: 200000] [Blocked Domains: 68000]

Dec. 17, 2013: We have a new logo thanks to Suzanne Champion.

Dec. 16, 2013: Version 0.19 - We've built a tool that lets us quickly determine the language of a page if it matches one of a few common patterns of declaring a its language. [URLS: 1100000] [Index: 84000] [Blocked Domains: 66000]

Dec. 11, 2013: We've crawled our first million URLs. [URLS: 1000000]

Dec. 9, 2013: Version 0.18 - We've made some improvements to our indexing algorithm. [URLS: 960000] [Index: 37000] [Blocked Domains: 64000]

Dec. 7, 2013: Thanks to Hermit Dave, we've built an extensive term indexing queue for the languages we support (except Swahili, we don't have one for that yet). Expect our index to improve quite a bit over time.

Dec. 4, 2013: Version 0.17 - We've cleaned out a lot of URLs in languages that we don't index (Russian, Chinese, Arabic, Japanese, etc.) and improved our crawler. [URLS: 890000] [Index: 5700] [Blocked Domains: 60000]

Dec. 2, 2013: We've moved from shared hosting to a dedicated server. Searches will be faster and more responsive and we have a lot more processing and storage capacity.

Nov. 23, 2013: Version 0.16 - We've done a lot of page crawling and added some extra error handling to the crawler. [URLS: 720000] [Index: 4800] [Blocked Domains: 36000]

Nov. 7, 2013: Version 0.15 - Made improvements to the crawler so that we can detect redirects and also not save 404 error pages to the index. We've also indexed a lot more search terms. [URLS: 205000] [Index: 4000] [Blocked Domains: 16000]

Nov. 4, 2013: Version 0.14 - Added domain lookup to the front page so you can check our database and index for a particular domain.

Oct. 31, 2013: Version 0.13 - The Zeta Web Directory is now WbSrch. We've moved to wbsrch.com. If you go to zetawebdirectory.com, you'll be redirected to the new domain.

Oct. 28, 2013: Version 0.12 - Added the Slovene language index (25 languages total). [URLS: 200000] [Index: 1300] [Blocked Domains: 15000]

Oct. 26, 2013: Version 0.11 - Added an Estonian language index (24 languages total) and made some improvements to the crawler. [URLS: 200000] [Index: 1100] [Blocked Domains: 14500]

Oct. 22, 2013: Version 0.10 - Added a Slovakian language index (23 languages total). [URLS: 190000] [Index: 1000] [Blocked Domains: 14000]

Oct. 21, 2013: Version 0.09 - Made major changes to the search algorithm, improving multi-word search quite significantly.

Oct. 20, 2013: Version 0.08 - Added 4 more language indexes - Croatian, Turkish, Latvian, and Lithuanian, bringing the total to 22 languages.

Oct. 19, 2013: Version 0.07 - Added a page where you can see the URL and index totals for each language. [URLS: 180000] [Index: 700] [Blocked Domains: 13000]

Oct. 18, 2013: Version 0.06 - Separated non-English languages into their own engines. You can click on a flag to get search results in that language. [URLS: 170000] [Index: 600] [Blocked Domains: 12000]

Oct. 13, 2013: Version 0.05 - Improved our indexing logic. [URLS: 170000] [Index: 200] [Blocked Domains: 7000]

Oct. 10, 2013: Version 0.04 - Added indexing capabilities to the engine. [URLS: 120000] [Index: 40] [Blocked Domains: 2000]

Oct. 7, 2013: Version 0.03 - Added admin tools that let us quickly exclude sites from the database. [URLS: 70000] [Blocked Domains: 2000]

Oct. 5, 2013: Version 0.02 - Better Python-based crawler. [URLS: 20000] [Blocked Domains: 150]

Sept. 23, 2013: Version 0.01 - Brittle C#-based crawler and title-only search. [URLS: 10000]

Back to search form.