Hey guys,
I added Japan to our site that lists the top Drupal websites per country.
http://top-websites.burtronix.co.za/drupal/japan/2013-08-05
No idea why the scripts detect so few Japanese Drupal websites. Maybe it is due to a lot of front-end changes needed to display a website in Japanese that may interfere with detecting it as a Drupal website?
For the technically inclined:
I'm now running the statistics gathering scripts as PHP CLI scripts that are much faster that the previous Drupal-based solution. I'm also storing data YAML format which is the new configuration file storage format for Drupal 8. The previous statistics gathering scripts ran for several days per chart, the new scripts finished the job for all the countries and CMSes in just a couple of hours.
Essentially, I obtain the list of top websites per country from Amazon through their web services, I then strip out all domains not in the ccTLD of the country and proceed with analysis. For analysis I grab the logic from the Wappalizer project and programmatically turn that into PHP logic; I then grab each top domain's http://, http://www, https:// and https://www versions and analyse the first that presents a 200 HTTP code. Finally, I run all of the found websites that were detected to run one of the CMSes past PhantomJS to get a screenshot for each.
The scripts aren't perfect, but since this was a weekend project, they can be made better in time.
Finally, the YAML files those scripts spit out are imported in one Drupal text field with a custom formatter to turn them into top charts. In time we'll gittify the source scripts, use Composer to get supporting libraries, base a lot more on Symfony (which supports newer YAML), write the scripts with more robust handling of sites (it simply skips a site if it can't connect to it the first try right now) and much more automation (still have a workflow around the generation of new charts right now).
The Amazon top list is based on usage of the Alexa toolbar, so if your site is not in the top list and you think it should be, simply install the Alexa toolbar for your browser.
Suggestions are very welcome!
Kind regards,
Riaan Burger
Comments
No idea why the scripts
The answer is probably more mundane - Drupal just isn't as "big in Japan" as other frameworks. And you're also limiting things a bit by assuming that all Japanese sites will have a .jp TLD - Japanese companies can and do use other TLDs too.
But at any rate, what algorithm are you using to detect a site as a Drupal-powered site?
The Boise Drupal Guy!
Wappalizer
Hey Garrett,
The script grabs the json raw from the Wappalizer project's github repo, but I use it in PHP, so don't have access to the JS environmental state detection (I'm guessing that's mainly used for the JS frameworks' detection though).
Will reply more on dndood's post below...
Okay, for those who like me
Okay, for those who like me have no idea what Wappalyzer is, this. Basically a system to guess what back- and front-end systems a site is using. This file seems to show some regexes run against the HTML, response headers and such to guess what's being used. Search it for "Drupal." The logic seems mostly solid, assuming it's checking for one of those things and not all of them.
The Boise Drupal Guy!
Domain stats overview
Hello,
I think Garrett made a good point. It looks like you are checking .jp domains and assuming these are the domains used in Japan. That is just not so. There are about 3.5 million gTLD (com/net/org/biz/info) domains in use in Japan and about 1.2 million .jp names in use.
Better to say that you script "checks the number of Drupal sites using .jp domains which is only about a 25% slice of the websites in use in Japan".
You could go to domaintools.com and download the gTLD zone for Japan then run your script against those. This will give everyone a much better picture of how many Drupal sites are used in Japan.
Yes, limited or maybe even not useful
In South Africa, at first, I just wanted to know what the landscape of Drupal looked like. So I first got the list from Amazon to see what the top sites are and saw that we mostly access sites outside of South Africa, so the top list would have included a lot of websites not really related to our developer Drupal groups. I decided to limit them by .za for the analysis and that helped a bit, but as you say, it isn't perfect.
I didn't think of using the zone files. That's a better solution for sure! Thanks! Still not sure about it though. Are those zone files free or accessed through the domaintools' API? If free I can implement much easier, if paid for, I still have to buy so many other countries' regular old top websites data, and then still have to update them once in a while, so it may be economically nonviable.
zone files
The zone file download from domaintools.com is not free unfortunately. You could do some massive scraping from webhosting.info for the Japan domains. This will give you about 80% of the zone but require a lot of scraping of domains off their site. You could also get fancy and contact ICANN, sign a big contract about how you won't use domains for bad purposes and get access to their zone file project. All the solutions seem like a lot of work though....
Yup, a bi tmuch ;-)
It does seem a bit much, but if traffic to the topsites go up, that would mean it becomes a valuable resource for the Drupal community and would need some attention to make it more, erm, right. Thanks for the idea, I wrote it into the planning.txt file for the project so if there's ever an opportunity, I'll include that.
With so many new top level domains coming though, I'm thinking a future replacement site will simply list top domains per TLD. Hmm, but then, that would be less useful to the local Drupal groups. Maybe then just revert back to the top list as supplied by Amazon (top by access statistics from a geographic area).
TopDrops.org
TopDrops.org http://topdrops.org is a new site that is now without branding and more focussed in support of the Drupal community. It parses the top 1 million websites (if all goes well, weekly) to find the top Drupal websites, ranks them and notes the people who built them (as well as their Drupal Association membership).
Retweet TopDrops.org:
https://twitter.com/intent/retweet?tweet_id=386829076364677121
Share or +1 TopDrops.org on Google:
https://plus.google.com/105124241461962087866/posts/WL3Ux4FaFpD