WbSrch Web Søg Changelog

20. juli 2021: Version 1.15 - We have added two new ranking lists - the Most-Crawled Domains List, a page showing the top 100 domains with the most pages crawled in our index, and the Most Domain Keywords Ranked List, a page showing the top 100 domains with the most keywords ranked in our search index. [URLS: 45300000] [Index: 12200000] [Blocked Domains: 3040000]

26. juni 2021: Version 1.14 - We have added eight new experimental language indexes - Esperanto, Faroese, Latin, Malagasy, Maltese, Occitan, Quechua, and Walloon. [URLS: 43600000] [Index: 11500000] [Blocked Domains: 3000000]

1. april 2021: Version 1.13 - We have added five new language indexes - Afrikaans, Bosnian, Irish, Luxembourgish, and Slovakian. [URLS: 33600000] [Index: 4580000] [Blocked Domains: 1800000]

21. marts 2021: Version 1.12 - The tables on the index stats and top-level domain stats pages can be sorted by clicking on the header row. [URLS: 31200000] [Index: 4550000] [Blocked Domains: 1630000]

16. marts 2021: Version 1.11 - We have made some changes to our indexing algorithms so that parked domains and placeholder "coming soon" pages will rank lower. [URLS: 29890000] [Index: 4540000] [Blocked Domains: 1610000]

6. marts 2021: Version 1.10 - We have enough data on some of the newer top-level-domains to make adjustments to how they affect page rankings. About 60 or so TLS have had their modifiers adjusted. [URLS: 27400000] [Index: 4500000] [Blocked Domains: 1540000]

19. februar 2021: Version 1.09 - Our indexer now uses a larger portion of the page when analyzing web contents and we have noticed a significant improvement in quality with this change. [URLS: 27200000] [Index: 4391000] [Blocked Domains: 1500000]

18. februar 2021: Version 1.08 - We have made the list of all of the domains that we have crawled and the list of adult domains that we have excluded available for purchase via our Data Shop. [URLS: 27200000] [Index: 4391000] [Blocked Domains: 1500000]

17. februar 2021: Version 1.07 - We have changed our crawler so that it no longer crawls IP addresses. Starting now, a page must have a domain name to be included in our index. [URLS: 27200000] [Index: 4391000] [Blocked Domains: 1500000]

13. februar 2021: Version 1.06 - We have replaced our the algorithm that handles our multi-word searches with something better [URLS: 27100000] [Index: 4383000] [Blocked Domains: 1490000]

15. januar 2021: Version 1.05 - We have added back all of the language indexes that had been pruned in the past. [URLS: 24870000] [Index: 4377000] [Blocked Domains: 1100000]

26. december 2020: Version 1.04 - After a hiatus, we are back with a different approach to managing our data. [URLS: 24920000] [Index: 3090000] [Blocked Domains: 970000]

30. maj 2019: We've made some changes that will make domain detail pages significantly faster. [URLS: 23280000] [Index: 3090000] [Blocked Domains: 950000]

28. maj 2019: It's now possible to get the top 1 million most-linked-to domains via our data shop. We plan to update this monthly and add other data offerings in the future.

23. maj 2019: Version 1.02 - We've added Aragonese, Basque, Galician, and Welsh language indexes. [URLS: 21700000] [Index: 3090000] [Blocked Domains: 670000]

12. maj 2019: Our index is now over 20 million pages again. [URLS: 20000000] [Index: 3090000] [Blocked Domains: 580000]

17. april 2019: Our index is now over 10 million pages again. [URLS: 10090000] [Index: 3090000] [Blocked Domains: 315000]

3. april 2019: Version 1.01 - We've improved our handling of typos and correction suggestions will be better. [URLS: 5010000] [Index: 3090000] [Blocked Domains: 145000]

2. april 2019: Version 1.00 - We've made some minor changes and improvements to our indexing algorithms. [URLS: 4980000] [Index: 3090000] [Blocked Domains: 144000]

1. april 2019: WbSrch is once again running on dedicated hardware. Expect faster growth in the near future. [URLS: 4890000] [Index: 3090000] [Blocked Domains: 136000]

3. februar 2019: We've re-added indexes for the Catalan, Croatian, Estonian, Latvian, Lithuanian, Romanian, Slovene, and Swahili languages. [URLS: 2080000] [Index: 2760000] [Blocked Domains: 44000]

15. marts 2018: The WbSrch search engine is online again with the same code, but starting over with crawling. [URLS: 10000]

30. juni 2016: The WbSrch search engine has gone offline.

22. juni 2016: Version 0.999 - We've moved Slovak-language pages into the Czech-language index so we now have a Czech/Slovak index. We've also moved Norwegian and Danish-language pages into the Swedish-language index so we now have a Scandinavian language index. [URLS: 47400000] [Index: 3640000] [Blocked Domains: 785000]

15. juni 2016: Our German-language index has grown a lot over the past three months and is now over 10 million pages, nine times what it was before. [URLS: 47840000] [Index: 4390000] [Blocked Domains: 785000]

14. juni 2016: Version 0.998 - We've removed a few language indexes due to lack of demand. They are: Catalan, Croatian, Estonian, Hausa, Icelandic, Latvian, Lithuanian, Romanian, Rwandan, Shona, Slovene, Somali, Swahili, Wolof, and Yoruba. This will allow us to focus our efforts on improving our more popular language indexes. [URLS: 47800000] [Index: 4390000] [Blocked Domains: 785000]

3. juni 2016: We now have 45 million pages in our index. [URLS: 45000000] [Index: 4960000] [Blocked Domains: 785000]

20. maj 2016: We now have 40 million pages in our index. [URLS: 40000000] [Index: 4960000] [Blocked Domains: 785000]

30. april 2016: Version 0.997 - We've made some changes to indexing that will cause some less desirable domain extensions to rank lower, along with many content-free and parked domains. [URLS: 32300000] [Index: 4950000] [Blocked Domains: 780000]

27. april 2016: Version 0.996 - We've made some changes to our indexer that will improve results for searches that contain non-ASCII characters. [URLS: 31000000] [Index: 4950000] [Blocked Domains: 780000]

25. april 2016: Version 0.995 - We've made some changes to our our web engine that will improve response times a bit thanks to Django Q and ujson. [URLS: 30130000] [Index: 4950000] [Blocked Domains: 775000]

24. april 2016: We now have 30 million pages in our index, the same amount the Infoseek (Go) search engine had during the late 1990's. [URLS: 30000000] [Index: 4950000] [Blocked Domains: 770000]

20. april 2016: Our Italian-language index has grown to three times its previous size and has more than 1 million pages. [URLS: 28400000] [Index: 4950000] [Blocked Domains: 760000]

15. april 2016: Our Dutch-language index has grown to seven times what it was at the beginning of the month and has more than 2 million pages. [URLS: 26200000] [Index: 4950000] [Blocked Domains: 746000]

14. april 2016: We now have 26 million pages in our index, the same amount the original Google had when it launched in 1998. [URLS: 26000000] [Index: 4950000] [Blocked Domains: 745000]

9. april 2016: We've more than doubled the size of our French-language index and now have more than 1 million pages in the French language. [URLS: 24100000] [Index: 4950000] [Blocked Domains: 730000]

29. marts 2016: We've added a music search engine called MusicSrch that will help you find your favorite musicians online. [URLS: 21650000] [Index: 4940000] [Blocked Domains: 726000]

28. marts 2016: Over the past month, we've nearly tripled the size of our Portuguese-language index. [URLS: 21580000] [Index: 4940000] [Blocked Domains: 725000]

27. marts 2016: Over the past month, we've more than doubled the size of our Spanish-language index. [URLS: 21500000] [Index: 4940000] [Blocked Domains: 725000]

23. marts 2016: We've more than tripled the size of our Polish-language index and have more than 1 million pages in the Polish language. [URLS: 20400000] [Index: 4940000] [Blocked Domains: 725000]

22. marts 2016: We now have 20 million pages in our index. [URLS: 20000000] [Index: 4940000] [Blocked Domains: 725000]

21. marts 2016: We've increased the size of our Danish-language index by more than 500%. [URLS: 19940000] [Index: 4940000] [Blocked Domains: 725000]

20. marts 2016: We've increased the size of our Swedish-language index by more than 500%. [URLS: 19860000] [Index: 4940000] [Blocked Domains: 725000]

19. marts 2016: We've more than quadrupled the size of our Romanian-language index. [URLS: 19600000] [Index: 4940000] [Blocked Domains: 725000]

17. marts 2016: We've more than quadrupled the size of our Hungarian-language index. [URLS: 18540000] [Index: 4940000] [Blocked Domains: 725000]

16. marts 2016: We've increased size of our Slovak-language index by more than 60%. [URLS: 18460000] [Index: 4940000] [Blocked Domains: 725000]

15. marts 2016: We've increased size of our Turkish-language index by more than 60%. [URLS: 18420000] [Index: 4940000] [Blocked Domains: 725000]

14. marts 2016: We've more than doubled the size of our Greek-language index. [URLS: 18360000] [Index: 4940000] [Blocked Domains: 725000]

13. marts 2016: We've more than quintupled the size of our Czech-language index. [URLS: 18200000] [Index: 4940000] [Blocked Domains: 725000]

9. marts 2016: We now have eighteen million pages in our index. [URLS: 18000000] [Index: 4940000] [Blocked Domains: 725000]

5. marts 2016: We now have seventeen million pages in our index. [URLS: 17000000] [Index: 4940000] [Blocked Domains: 724000]

4. marts 2016: We now have more than 1 million pages in our Spanish-language index. [URLS: 16530000] [Index: 4940000] [Blocked Domains: 720000]

3. marts 2016: We've more than tripled the size of our Norwegian-language index. [URLS: 16070000] [Index: 4940000] [Blocked Domains: 718000]

2. marts 2016: We now have sixteen million pages in our index. [URLS: 16000000] [Index: 4940000] [Blocked Domains: 715000]

1. marts 2016: We've more than quadrupled the size of our Finnish-language index. [URLS: 15700000] [Index: 4940000] [Blocked Domains: 710000]

29. februar 2016: We've more than tripled the size of our Lithuanian-language index. [URLS: 15320000] [Index: 4940000] [Blocked Domains: 701000]

28. februar 2016: We've more than doubled the size of our Latvian-language index. [URLS: 15160000] [Index: 4940000] [Blocked Domains: 700000]

27. februar 2016: We now have fifteen million pages in our index. [URLS: 15000000] [Index: 4940000] [Blocked Domains: 699000]

26. februar 2016: We've more than doubled the size of our Estonian-language index. [URLS: 147800000] [Index: 4940000] [Blocked Domains: 695000]

25. februar 2016: We've more than doubled the size of our Slovene-language index. [URLS: 14700000] [Index: 4940000] [Blocked Domains: 690000]

24. februar 2016: We've increased the size of our Croatian-language index by 50%. [URLS: 14600000] [Index: 4940000] [Blocked Domains: 680000]

22. februar 2016: We've more than doubled the size of our Catalan-language index. [URLS: 14430000] [Index: 4940000] [Blocked Domains: 672000]

21. februar 2016: Version 0.994 - We've added autocomplete to our search box. It should work in all modern HTML5-compliant browsers. [URLS: 14400000] [Index: 4940000] [Blocked Domains: 670000]

20. februar 2016: Version 0.993 - We've made some minor adjustments to how some domain extensions are treated by our ranking algorithms. [URLS: 14300000] [Index: 4940000] [Blocked Domains: 660000]

16. februar 2016: Version 0.992 - When you search, the phrase you searched for will be shown in bold text in the results so you can more easily judge the relevance of each link. [URLS: 14080000] [Index: 4940000] [Blocked Domains: 652000]

15. februar 2016: We now have fourteen million pages in our index. [URLS: 14000000] [Index: 4940000] [Blocked Domains: 650000]

7. februar 2016: We're running a crowdfunding campaign on IndieGoGo to raise funds for hardware upggrades. Please help support the growth of WbSrch. [URLS: 13840000] [Index: 4940000] [Blocked Domains: 625000]

31. januar 2016: Version 0.991 - We've made a change that improves results for never-before-seen multi-word searches, especially those for proper names. [URLS: 13500000] [Index: 4940000] [Blocked Domains: 623000]

26. januar 2016: We've released our own custom web browser. It's called WbBrowse and it's available at wbbrowse.com. [URLS: 13360000] [Index: 4940000] [Blocked Domains: 622000]

25. januar 2016: Version 0.99 - We've made some changes to our index structure that should improve multi-word phrase search, especially for combinations of words that are less common. [URLS: 13300000] [Index: 4940000] [Blocked Domains: 621000]

24. januar 2016: Version 0.98 - We've made a change that should significantly improve search results for multi-word phrases. [URLS: 13290000] [Index: 4940000] [Blocked Domains: 620000]

18. januar 2016: We now have thirteen million pages in our index. [URLS: 13000000] [Index: 4940000] [Blocked Domains: 619000]

16. januar 2016: Version 0.97 - We've made a change that improves search results for multi-word phrases that have never been searched for. [URLS: 12960000] [Index: 4940000] [Blocked Domains: 615000]

12. januar 2016: Version 0.96 - We've made some improvements to our indexer, including combining domains with a "www" prefix and those without into a single listing (links from www.example.com and example.com would previously show up separately in the same search result). [URLS: 12870000] [Index: 4940000] [Blocked Domains: 610000]

29. december 2015: Version 0.95 - We've made some changes to our indexer that will cause domains containing numbers to rank lower for non-numeric searches and sites with excessively long titles to rank lower. We've also made some minor improvements to the domain details page. [URLS: 12670000] [Index: 4939000] [Blocked Domains: 588000]

18. december 2015: Version 0.94 - We've made a few changes to our crawler that have improved its multilingual crawling ability. [URLS: 12570000] [Index: 4935000] [Blocked Domains: 580000]

12. december 2015: We now have more than 1 million pages in our German-language index. [URLS: 12400000] [Index: 4935000] [Blocked Domains: 577000]

7. december 2015: Version 0.93 - We've made an improvement to our crawler that will cause it to better handle redirects when re-crawling non-English-language URLs. [URLS: 12010000] [Index: 4930000] [Blocked Domains: 575000]

6. december 2015: We now have twelve million pages in our index. [URLS: 12000000] [Index: 4930000] [Blocked Domains: 575000]

24. november 2015: We've crawled our eleventh million URLs. [URLS: 11000000] [Index: 4930000] [Blocked Domains: 560000]

14. november 2015: Version 0.92 - When you search for a domain name, you'll also see a link to the domain details page if we have any info for that domain. The same goes for IP addresses. [URLS: 10070000] [Index: 4930000] [Blocked Domains: 542000]

11. november 2015: We've upgraded our infrastructure to a 3-server setup from the previous single-server configuration. [URLS: 10010000] [Index: 4900000] [Blocked Domains: 540000]

20. oktober 2015: Version 0.91 - We've made an improvement to the crawler so that XHTML pages look better in search results. [URLS: 9640000] [Index: 4820000] [Blocked Domains: 540000]

18. oktober 2015: Version 0.90 - We've changed search results to show no more than 200 results in total. The previous limit was 250 for indexed phrases, or 250 per word for unindexed phrases (a search like "large yellow socks" could return 750 results). [URLS: 9400000] [Index: 4810000] [Blocked Domains: 540000]

5. oktober 2015: Version 0.89 - We've made some improvements to our language categorization tools that will allow us to categorize page languages for a site on a page-by-page basis. [URLS: 8550000] [Index: 4775000] [Blocked Domains: 523000]

3. oktober 2015: Version 0.88 - We've made some pretty big improvements to how we analyze page text. [URLS: 8490000] [Index: 4770000] [Blocked Domains: 522000]

1. oktober 2015: Version 0.87 - We've made more improvements to our ranking algorithms, including improvements to how we handle page text and domain names. [URLS: 8480000] [Index: 4760000] [Blocked Domains: 520000]

26. september 2015: Version 0.86 - We've added an articles section and included a couple posts from an earlier blog. [URLS: 8260000] [Index: 4750000] [Blocked Domains: 495000]

25. september 2015: Version 0.85 - We've long had the ability to perform get the search results for a specific site by clicking on the "# of results from this domain" link in search results. Now you can search for a result from a specific domain using the "site:example.com example" syntax (it doesn't matter whether you put that at the beginning or end, so "example site:example.com" is the same). Searching just for "site:example.com" will show the domain info page for that domain. [URLS: 8140000] [Index: 4750000] [Blocked Domains: 495000]

23. september 2015: WbSrch is two years old today. [URLS: 8066000] [Index: 4750000] [Blocked Domains: 494000]

22. september 2015: Version 0.84 - We've added a basic IP information view that lets you see what domains and how many pages we've found at a particular IP address. [URLS: 8040000] [Index: 4740000] [Blocked Domains: 493000]

21. september 2015: Version 0.83 - We received our first ever DMCA takedown notice today. We've created a section of the website where we can post DMCA notices that we've received and have published it online. [URLS: 8000000] [Index: 4740000] [Blocked Domains: 493000]

15. september 2015: Version 0.82 - We've made minor changes to the indexer to tune how we treat links and subdomains. The rules are the same, but effects are now more fine-grained. [URLS: 7900000] [Index: 4720000] [Blocked Domains: 491000]

10. september 2015: Version 0.81 - We've started tracking more information about the objects on a web page - iframes, audio and video tags, JavaScript, images, and CSS files. None of this information is a ranking factor yet, but some of it probably will be in the near future. [URLS: 7890000] [Index: 4680000] [Blocked Domains: 488000]

7. september 2015: Version 0.80 - We've started using the character encoding of a page as a minor ranking factor. We now also look at the MIME type of a page and use that as a ranking factor. [URLS: 7850000] [Index: 4680000] [Blocked Domains: 488000]

6. september 2015: Version 0.79 - We've made a minor change to how our indexer scores URLs. [URLS: 7800000] [Index: 4670000] [Blocked Domains: 487000]

5. september 2015: Version 0.78 - We've made a few visual design changes, moving the language change flags to the bottom of the page. We've also changed the indexer to rank some top-level domain extensions lower based on their poor reputations. [URLS: 7750000] [Index: 4670000] [Blocked Domains: 487000]

2. september 2015: The number of terms indexed shown on our index stats page has dropped drastically over the past few weeks. We've pruned the list of words we index to remove empty searches. This won't affect your search results, but it will save us a bunch of CPU cycles. [URLS: 7700000] [Index: 4660000] [Blocked Domains: 485000]

26. august 2015: Version 0.78 - We've made a minor change to our crawler to improve its ability to spider secure (HTTPS) websites. We've always been able to crawl HTTPS sites, but now we make better decisions about what URL to crawl next. [URLS: 7670000] [Index: 4740000] [Blocked Domains: 484000]

24. august 2015: Version 0.77 - Domain detail pages will now give a general indication of how many inbound links we've found for a domain. Since we've only crawled a small fraction of the web so far, these details will necessarily be vague, like "not many" or "a lot", but it should give some indication of what data we have for that domain's links. [URLS: 7650000] [Index: 5430000] [Blocked Domains: 483800]

23. august 2015: Although our crawler now supports robots.txt, a lot of pages in our index were crawled before support was added. We removed a bunch of pages that would have been blocked by the robots file from our index. Any that appear in search results will gradually disappear over time as things are reindexed. [URLS: 7660000] [Index: 5930000] [Blocked Domains: 483800]

22. august 2015: Version 0.76 - Pages with a comma in the URL will now tend to rank lower in search results. Commas in a URL are bad form, so don't do it. [URLS: 7660000] [Index: 5930000] [Blocked Domains: 483800]

21. august 2015: Version 0.75 - We've made a minor change that will cause URLs with lots of query parameters to tend to rank slightly lower than clean URLs for the same content. [URLS: 7710000] [Index: 5930000] [Blocked Domains: 483600]

18. august 2015: Version 0.74 - We've changed the way we count page titles and descriptions in our indexing process to be a bit more nuanced. Page size will also be a factor, with pages that are very tiny or extremely huge ranking lower than those that are mid-sized. [URLS: 7700000] [Index: 6800000] [Blocked Domains: 483500]

14. august 2015: Version 0.73 - Some newer top-level domains will now tend to rank lower. Think twice before buying that ".link" domain. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 483000]

13. august 2015: Version 0.72 - Ranking info is now available on domain detail pages. It will show the top search keywords that a domain ranks for on WbSrch (up to 100). Not all ranked keywords are shown for a domain yet, but they'll fill in as our indexer cycles through the remaining keywords over the next few weeks. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 482500]

10. august 2015: Version 0.71 - We've excluded a few more file types from our crawler (media, images, source code) in order to build a cleaner index. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 482500]

7. august 2015: Version 0.70 - We've started keeping track of what web server sites use to serve content. It's not likely to be a ranking factor, but it'll allow us to track the usage share of different web servers. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 482500]

5. august 2015: Version 0.69 - Our crawler now follows robots.txt rules. The crawler agent has been updated to WbSrch/1.1 to reflect that. We're not popular enough for sites to write specific rules for us, so we'll follow the rules set for Google until we notice a statistically significant number of rules specific to the WbSrch user agent. [URLS: 7690000] [Index: 10700000] [Blocked Domains: 482000]

2. august 2015: Version 0.68 - We've made some improvements to handling session IDs that will result in crawling fewer duplicate URLs. We've also removed a lot of duplicate URLs from our index. It's amazing the terrible things people can do to session IDs with a Tomcat server. [URLS: 7690000] [Index: 10680000] [Blocked Domains: 480000]

27. juli 2015: We lost 4 drives in a 6-drive RAID array and had to restore from backup. No code was lost, but crawl and indexer data for the last couple months is missing. We'll catch up quick. It's the code that's the hard part. We cannot recommend Mushkin SSDs. Spend the extra few dollars for Kingston or Samsung. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 436000]

18. juli 2015: Version 0.67 - We've added a page that shows the 100 most-linked-to domains based on the contents of our index.

17. juli 2015: Version 0.66 - We've started using the number of different domains linking to a domain as a minor ranking factor. We don't do any fancy calculations yet (nothing like PageRank or HITS, for example), we just score based on a raw count.

15. juli 2015: Version 0.65 - We've started tracking links on a page and will soon be using them as a ranking factor.

14. juli 2015: Version 0.64 - We've started tracking detailed information about where each domain ranks for specific keywords.

14. marts 2015: Version 0.63 - We've started reading robots.txt files. We don't take any action based on them yet, but if we're reading them, reacting to rules can't be too far behind. [URLS: 10090000] [Index: 10790000] [Blocked Domains: 474000]

1. marts 2015: We've crawled our tenth million URL and now have an index the size of the original AltaVista search index. [URLS: 10000000] [Index: 10780000] [Blocked Domains: 469000]

24. februar 2015: We've crawled our ninth million URLs. [URLS: 9000000] [Index: 10780000] [Blocked Domains: 449000]

23. februar 2015: Version 0.62 - The age of a domain will now be a factor in rankings. Brand new domains will tend to rank lower than domains that have been around for a while. [URLS: 8740000] [Index: 10780000] [Blocked Domains: 447000]

10. februar 2015: Version 0.61 - Our language categorization ability has improved thanks to the inclusion of the langid library by Marco Lui. [URLS: 8120000] [Index: 10700000] [Blocked Domains: 440000]

26. januar 2015: Version 0.60 - We've added an index for the Catalan language and now support 32 languages. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 436000]

25. januar 2015: Version 0.58 - We've made some changes to the site that should make it generally faster and more responsive. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 435000]

25. januar 2015: Version 0.59 - We've made some changes to our crawler that will cause it to crawl cleaner URLs, leaving off a lot of common session and referral IDs. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 436000]

24. januar 2015: Version 0.57 - We've made some changes to our indexer that will cause it to look at more "H" tags when analyzing page titles. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 430000]

23. januar 2015: Version 0.56 - We've made some improvements to our web crawler that will result in better-looking search results. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 430000]

22. januar 2015: Version 0.55 - We've made some changes to the way search results are generated that will cause them to look better. The appearance of searches will improve gradually over time as keywords are re-indexed. [URLS: 8100000] [Index: 10670000] [Blocked Domains: 430000]

19. januar 2015: Version 0.54 - We've made some big improvements to our language categorization tools. URLs from more domains will end up in the right place. [URLS: 8080000] [Index: 10670000] [Blocked Domains: 430000]

11. januar 2015: Version 0.53 - We've added an index for the Shona language and now support 31 languages. [URLS: 8000000] [Index: 10670000] [Blocked Domains: 424000]

10. januar 2015: We've crawled our eighth million URLs. [URLS: 8000000] [Index: 10670000] [Blocked Domains: 420000]

23. december 2014: Version 0.52 - We've added an index for the Rwandan language and now support 30 languages. [URLS: 7720000] [Index: 10654000] [Blocked Domains: 330000]

21. december 2014: Version 0.51 - We've added an index for the Somali language and now support 29 languages. [URLS: 7700000] [Index: 10651000] [Blocked Domains: 329000]

20. december 2014: Version 0.50 - We've added an index for the Hausa language and now support 28 languages. [URLS: 7690000] [Index: 10650000] [Blocked Domains: 329000]

18. december 2014: Version 0.49 - We've added an index for the Wolof language and now support 27 languages. [URLS: 7640000] [Index: 10648000] [Blocked Domains: 328000]

16. december 2014: Version 0.48 - We've added an index for the Yorùbá language and now support 26 languages. [URLS: 7630000] [Index: 10645000] [Blocked Domains: 325000]

23. november 2014: Version 0.47 - We've made more mobile-friendly improvements to our site. [URLS: 7600000] [Index: 10645000] [Blocked Domains: 320000]

28. oktober 2014: We've crawled our seventh million URLs. [URLS: 7000000] [Index: 10640000] [Blocked Domains: 300000]

23. september 2014: WbSrch is now one year old. Time flies! [URLS: 6500000] [Index: 10640000] [Blocked Domains: 280000]

5. september 2014: We've crawled our sixth million URLs. [URLS: 6000000] [Index: 10640000] [Blocked Domains: 219000]

12. august 2014: We've improved our non-English-language indexes quite a bit. Searches in Spanish, Italian, and other languages will have much better results than before. [URLS: 5300000] [Index: 10640000] [Blocked Domains: 216000]

5. august 2014: We've served one million searches so far. This includes bot and crawler traffic.

10. juni 2014: Version 0.46 - We've fixed a few things that could generate server errors (500's), including quickly searching for the same thing twice in a row or being referred to the site from a gigantically long URL. [URLS: 5100000] [Index: 10640000] [Blocked Domains: 187000]

5. juni 2014: We've crawled our fifth million URLs. [URLS: 5000000] [Index: 10640000] [Blocked Domains: 175000]

4. juni 2014: Version 0.45 - The WbSrch website is now SSL-only. We're proud to support the ResetTheNet movement. [URLS: 4990000] [Index: 10640000] [Blocked Domains: 174000]

3. juni 2014: Version 0.44 - We've made speed improvements to some slower areas of the site, such as the "index stats" page and the "popular searches" pages. [URLS: 4950000] [Index: 10640000] [Blocked Domains: 170000]

30. maj 2014: Version 0.43 - We've made more progress improving our site language translations for non-English languages. [URLS: 4700000] [Index: 10640000] [Blocked Domains: 150000]

26. maj 2014: Version 0.42 - We've made some changes to our site design that will make it look better on low-resolution screens. [URLS: 4600000] [Index: 10640000] [Blocked Domains: 144000]

24. maj 2014: Version 0.41 - We've made changes to our indexer that will make searches for specific domains far better. [URLS: 4550000] [Index: 10640000] [Blocked Domains: 143000]

23. maj 2014: We've created more search plugins and now have one for each language that we support. You can get them from the Mycroft Project website. [URLS: 4500000] [Index: 10640000] [Blocked Domains: 143000]

22. maj 2014: Version 0.40 - We've made some significant changes to our indexer that will improve search relevance and have much better results for multi-word phrases. [URLS: 4500000] [Index: 10640000] [Blocked Domains: 140000]

18. maj 2014: Version 0.39 - We've made some improvements to multi-word searches. [URLS: 4400000] [Index: 10630000] [Blocked Domains: 137000]

10. maj 2014: Version 0.38 - We've added the ability to block search phrases and have blocked a few dozen terms that nobody has any business searching for, such as "child porn" and some related searches. [URLS: 4120000] [Index: 10630000] [Blocked Domains: 126000]

7. maj 2014: We've crawled our fourth million URLs. [URLS: 4000000] [Index: 10630000] [Blocked Domains: 125000]

30. april 2014: Version 0.37 - We fixed a bug that caused searches containing capital letters to return worse results than those all in lowercase. [URLS: 3750000] [Index: 10630000] [Blocked Domains: 122000]

25. april 2014: Version 0.36 - We've made some improvements to the visual design of the WbSrch website. [URLS: 3480000] [Index: 10630000] [Blocked Domains: 120000]

10. april 2014: Version 0.35 - We've made some improvements to our crawler with regard to handling errors. The variety and quantity of errors you can encounter on the web is amazing. We do not yet properly handle the "I'm a Teapot" error. [URLS: 3150000] [Index: 10630000] [Blocked Domains: 117000]

3. april 2014: Our HTML parser is much better today than it was when we first started. We've done a lot of recrawling of sites that were crawled early in our existence. Search results will look better than they have in the past. [URLS: 3090000] [Index: 10630000] [Blocked Domains: 116000]

16. marts 2014: We've crawled our third million URLs. [URLS: 3000000] [Index: 10300000] [Blocked Domains: 113000]

12. marts 2014: Version 0.34 - We've added a popular searches view that lets you see what was searched for most often in a given month. [URLS: 2970000] [Index: 10630000] [Blocked Domains: 107000]

3. marts 2014: Version 0.33 - We fixed a nasty bug that caused international search results to be far lower quality than searches in English. [URLS: 2890000] [Index: 10630000] [Blocked Domains: 105000]

2. marts 2014: Version 0.32 - We've made some improvements to our crawler, many of them focused on re-crawling pages that haven't been crawled in a while or handling some of the less common HTTP errors. [URLS: 2880000] [Index: 10630000] [Blocked Domains: 105000]

21. februar 2014: We've finished indexing the 10.6 million most common words across the 25 languages that we support. [URLS: 2740000] [Index: 10630000] [Blocked Domains: 99000]

17. februar 2014: Version 0.31 - We've made some minor improvements to indexing of multi-word search phrases. They still need a lot of work, but are a little better now. [URLS: 2640000] [Index: 10590000] [Blocked Domains: 99000]

13. februar 2014: Version 0.30 - Now that search results are grouped by domain, we've started taking the number of results from a domain for a particular search term into account in our rankings. We've also made some improvements to our admin tools so we can manage domains and URLs more effectively. [URLS: 2590000] [Index: 10300000] [Blocked Domains: 94000]

12. februar 2014: Version 0.29 - We've changed the way search results are displayed. Instead of displaying everything all at once with multiple results from the same domain strewn about haphazardly, we group them together and show the best result for that search. There's now a link you can click to show all of the results for that domain for the search. Just click on the (X results for this site) to see them. [URLS: 2580000] [Index: 10300000] [Blocked Domains: 94000]

9. februar 2014: Version 0.28 - We've made some improvements to our page scoring logic that include taking the length of the URL into account. If a page has a URL that's too long for a Human to type it'll rank slightly lower than a shorter one. [URLS: 2540000] [Index: 10100000] [Blocked Domains: 93000]

8. februar 2014: We've finished building our search keyword index using our new fast-retrieval method. Search results will now be faster and better organized. [URLS: 2540000] [Index: 10000000] [Blocked Domains: 93000]

2. februar 2014: Version 0.27 - We've come up with a method of indexing search keywords that allows for much faster retrieval. It will take a while to build new indexes for the 10-million-plus search terms we track, but searches will be quicker for those that have the new indexes. [URLS: 2500000] [Index: 9950000] [Blocked Domains: 93000]

30. januar 2014: Version 0.26 - We've added more capabilities to our language detection tools by interfacing with NLTK, the Natural Language Toolkit. [URLS: 2490000] [Index: 9740000] [Blocked Domains: 93000]

27. januar 2014: Version 0.25 - We weren't originally keeping track of the IP address that serves a web page. It turns out that is useful information that can be used in classification and ranking, so we're storing that now. [URLS: 2420000] [Index: 9500000] [Blocked Domains: 93000]

19. januar 2014: Version 0.24 - We've created some basic tools that let us track which sites are the sources and destinations of spam comments and links. We'll be using those to rank spam-generating sites lower in the index or exclude them entirely. [URLS: 2290000] [Index: 8300000] [Blocked Domains: 92000]

17. januar 2014: We've now indexed eight million search terms in the 25 languages that we support. [URLS: 2250000] [Index: 8000000] [Blocked Domains: 91000]

3. januar 2014: We've crawled our second million URLs. [URLS: 2000000] [Index: 5100000] [Blocked Domains: 89000]

29. december 2013: Version 0.23 - We've made even more improvements to our language categorization tools and now have more than 370,000 URLs categorized as non-English languages. There's still a lot of work to be done, but our German, French, Spanish, Polish, and Portuguese indexes have recently become a lot more robust. [URLS: 1820000] [Index: 4500000] [Blocked Domains: 77000]

27. december 2013: Version 0.22 - We've implemented optimizations to our indexing algorithm that make it 30% faster. [URLS: 1650000] [Index: 4100000] [Blocked Domains: 75000]

26. december 2013: We've recently been redlining our CPUs indexing search terms and now have more than 3.7 million terms indexed. [URLS: 1570000] [Index: 3700000] [Blocked Domains: 75000]

23. december 2013: Version 0.21 - We've made more improvements to our language categorization tools and now have more than 200,000 URLs categorized as non-English languages.

20. december 2013: Version 0.20 - We've made improvements to our crawler that will reduce bandwidth use and avoid crawling some non-HTML object types. [URLS: 1240000] [Index: 200000] [Blocked Domains: 68000]

17. december 2013: We have a new logo thanks to Suzanne Champion.

16. december 2013: Version 0.19 - We've built a tool that lets us quickly determine the language of a page if it matches one of a few common patterns of declaring a its language. [URLS: 1100000] [Index: 84000] [Blocked Domains: 66000]

11. december 2013: We've crawled our first million URLs. [URLS: 1000000]

9. december 2013: Version 0.18 - We've made some improvements to our indexing algorithm. [URLS: 960000] [Index: 37000] [Blocked Domains: 64000]

7. december 2013: Thanks to Hermit Dave, we've built an extensive term indexing queue for the languages we support (except Swahili, we don't have one for that yet). Expect our index to improve quite a bit over time.

4. december 2013: Version 0.17 - We've cleaned out a lot of URLs in languages that we don't index (Russian, Chinese, Arabic, Japanese, etc.) and improved our crawler. [URLS: 890000] [Index: 5700] [Blocked Domains: 60000]

2. december 2013: We've moved from shared hosting to a dedicated server. Searches will be faster and more responsive and we have a lot more processing and storage capacity.

23. november 2013: Version 0.16 - We've done a lot of page crawling and added some extra error handling to the crawler. [URLS: 720000] [Index: 4800] [Blocked Domains: 36000]

7. november 2013: Version 0.15 - Made improvements to the crawler so that we can detect redirects and also not save 404 error pages to the index. We've also indexed a lot more search terms. [URLS: 205000] [Index: 4000] [Blocked Domains: 16000]

4. november 2013: Version 0.14 - Added domain lookup to the front page so you can check our database and index for a particular domain.

31. oktober 2013: Version 0.13 - The Zeta Web Directory is now WbSrch. We've moved to wbsrch.com. If you go to zetawebdirectory.com, you'll be redirected to the new domain.

28. oktober 2013: Version 0.12 - Added the Slovene language index (25 languages total). [URLS: 200000] [Index: 1300] [Blocked Domains: 15000]

26. oktober 2013: Version 0.11 - Added an Estonian language index (24 languages total) and made some improvements to the crawler. [URLS: 200000] [Index: 1100] [Blocked Domains: 14500]

22. oktober 2013: Version 0.10 - Added a Slovakian language index (23 languages total). [URLS: 190000] [Index: 1000] [Blocked Domains: 14000]

21. oktober 2013: Version 0.09 - Made major changes to the search algorithm, improving multi-word search quite significantly.

20. oktober 2013: Version 0.08 - Added 4 more language indexes - Croatian, Turkish, Latvian, and Lithuanian, bringing the total to 22 languages.

19. oktober 2013: Version 0.07 - Added a page where you can see the URL and index totals for each language. [URLS: 180000] [Index: 700] [Blocked Domains: 13000]

18. oktober 2013: Version 0.06 - Separated non-English languages into their own engines. You can click on a flag to get search results in that language. [URLS: 170000] [Index: 600] [Blocked Domains: 12000]

13. oktober 2013: Version 0.05 - Improved our indexing logic. [URLS: 170000] [Index: 200] [Blocked Domains: 7000]

10. oktober 2013: Version 0.04 - Added indexing capabilities to the engine. [URLS: 120000] [Index: 40] [Blocked Domains: 2000]

7. oktober 2013: Version 0.03 - Added admin tools that let us quickly exclude sites from the database. [URLS: 70000] [Blocked Domains: 2000]

5. oktober 2013: Version 0.02 - Better Python-based crawler. [URLS: 20000] [Blocked Domains: 150]

23. september 2013: Version 0.01 - Brittle C#-based crawler and title-only search. [URLS: 10000]

Back to search form.