Gold Notable Member
- Jul 3, 2021
Hidden Treasures of TLDs: How I Scraped Hackernews for Domain Names – Klaus Breyer > CTO writing about Code, Business, Product & Engineering Orgs.
Background Story ¶As somebody who often starts new projects, I often need to think about project names and domain names.
Derek Sivers once posted about how to find a good (and free) .com Domain , and I found it very inspiring. However, in some cases, there is the project name defined first - or you want to do a good play of words with the name and domain of the project. In those cases, you need to have a suitable TLD.
But finding such domain names is a tricky thing. If you go through Wikipedia, you end up with more than 1.2k TLDs . (Trust me, I did it).
So I needed to narrow it down. And I did so by running it through the filter of a bubble that a) seems relevant to me and b) was large enough: People who read and post on Hackernews . So I had my Raspberry Pi scraping the Hackernews API for about 3 Weeks (because of rate limits), and the results you find up there.
I had a database full of HN Stories since the very beginning, which accumulated to ~1GB.