Jump to content

  • Always show respect and upvote crackers for working accounts! otherwise you'll be considered as a leecher.

Sign in to follow this  

How we collected the database of all domains in the world

Recommended Posts

Once, in our company, we decided to conduct a market research CMS (Content Management System) - this is a management system, engine, platform or constructor that allows you to manage the content of the site. I was just wondering what share we have in the local market and how much on the overall scale. The question arose where to get the base of domains for researching various CMS (Magento, Joomla, Drupal). To my surprise, such a base did not exist and still does not exist. And that's what happened next ... For some separate domain zones, the base could be taken quite simply, but, globally, this was not enough for a full-fledged study. The list of zones can be viewed on the IANA. A search for options was started ... We are guys from the IT industry and we understood that the situation with domains is changing every day and it is not enough just to download the database once and calm down - we needed constant access and full automation of obtaining information about domains. For example, in the .com zone, 100-140 thousand domains are not renewed every day. Therefore, in a week, our database will contain millions of irrelevant domains. I must say that checking for employment only seems simple. Each whois service has its own rules and response form. Moreover, they often change and you have to catch such situations. Sometimes, some whois suddenly gave an answer like "query limit" and stopped giving information. I had to create a whole VPS farm to collect information from different IPs and make special delays. In general, the whole setup took a couple of months of work. And another six months to bring the process to mind and 95% automation. It will not be possible to fully automate - something breaks somewhere, changes and requires attention. Who needed it? As a result, we got our own service https://www.mailbase.shop/?cat_id=43 to support the current database of domain names sorted by CMS. It turned out that it was not in vain that we were tormented - such databases were in demand by various companies for their own research. If someone has ideas how and where to get up-to-date information on registered domain names, I will be glad to any comment, as well as ideas on the commercial (and not only) use of the domain base. The bases can be purchased with the current payment method Bitcoin!

Share this post

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

Board Life Status

Board startup date: October 30, 2017 06:45:19
  • Create New...