Drupal search engine optimization is a full time job

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
jt6919's picture

Well, I just found about this group (and groups.drupal.org) today, and I've been using Drupal for over a year! I see this group for SEO has 59 subscribers, and yet I hadn't seen any posts from any of you on seo in the main drupal forums. Why is that? If I had seen mention of this before, I would have joined as soon as I found out - and there are questions that people are asking all the time about SEO in the main forums that deserve answers. I've been trying to answer all the ones that I can find.

I posted twice in the last month in the main forums to solicit input regarding an article I wrote on my web site: How to Optimize a Drupal web site for Google and other Search Engine Crawlers. I was surprised to find I didn't get a shred of feedback. The post has been viewed almost 1,200 times on my site and I've not gotten any feedback there either.

I really want feedback, because I know that others have information I could be adding to that article - making it even more useful. I know what I've posted works, because it's consistently been increasing the traffic on my web site. Most of it would work on any web site, but I've got Drupal specific items in there as well - including the modules that work best for me seo.

Please let me know what you think...

Comments

robots

greggles's picture

You mention

"The only file the search engine crawlers really need to get to is /index.php. For more information on creating a robots.txt file"

I don't think that's correct. If you are using clean URLs (which you should, especially if you are worried about SEO) then the robots.txt also needs allow access to node/* taxonomy/* and similar other "paths" that may not be folders, but which are URLs. If you have pathauto enabled then you also need to make sure that robots allows access to any URLs that you create via pathauto (e.g. lots of people use category/[cat] as a filter for pathauto - in that case you need access to category/). I think you're better off using disallow for things you want to hide and then allow/ for everything else.

Also, while you recommend spelling/grammar/punctuation improvement I noticed quite a few ommissions on your site as well.

Your recommendation about clean URLs is incomplete - people need Path module to make the URLs look like you describe - the Clean URLs option just gets rid of the ?q= part of the URL. They are separate options that do separate things, but they're often confused and mixed together.

In that same area, I think folks are settled on dash instead of underscore for splitting words.

Also, that page is not the top result for me for the terms that you claim it is. That could be a matter of personalized search results for you. I tried it from a machine where I don't login to google and the resuts were the same (i.e. your site wasn't #1).

Thanks for the resource - I look forward to seeing updated versions.

Greg

--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO

thanks for the feedback!

jt6919's picture

thanks for the feedback, this will help me to refine some of the information a great deal. My idea on that only page the crawler really needs to get to was index.php was based on the notion that the home page is already setup to allow the crawler to get to all other necessary places in the site. I'll try and clarify that in the article. You could surely add node/* taxonomy/* to the robots.txt, but if your home page has a sitemenu or links to categories it will find your content better.

Also, while you recommend spelling/grammar/punctuation improvement I noticed quite a few ommissions on your site as well.

can you elaborate on this?

I've read plenty of posts about dash vs. underscore....and most do say that dash is better for google (only). But, most of the research done in this area is 2-4 years old - and I think the tide has changed and become a bit more level.

Take this search for example:
http://www.google.com/search?q=download+barbie+girl+mp3&start=0&ie=utf-8...

While popularity counts as well, you can see that the URL's with %20's rank first, then underscores, and then dashes (on the second results page). I'm sure that you could come up with plenty of examples on either site. It garners putting more info into the article about the debate for sure.

On the results topic - I come up as #2 on that page now, but it was #1 on the day I wrote it. If you get anything other than #2 please let me know, because I see it that way on 8 different machines with no personalized results.

thanks again for the comments!

I use Drupal everyday in about a dozen of my web sites - check out http://www.smorgasbord.net and http://celebritynewslive.com.

...

greggles's picture

Robots: the robots.txt doesn't tell them the starting point - it tells allow/disallow. The easiest way to set it up is just Disallow on stuff you want to hide and Allow: / for everything else.

Can I elaborate on spelling mistakes? No - any good word processor will do that for you.

For dash vs. underscore, it's been a little over a year since this post: http://www.mattcutts.com/blog/dashes-vs-underscores/ but why bother risking that search engines have gotten smart enough to do the right thing? Just use the dash and you don't have to think about it.

I got you as number 2 or 3 for the different phrases - different datacenters are...different, I guess.

Greg

--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO

the main reason....

jt6919's picture

The main reason to not use dashes for me is pathauto itself. I have some 12,600 nodes published - and they all have underscores. I could probably change the configuration to use the dash, but my pathauto module is set to create a new version on update. I do this because I use aggregation, and if the title of the article in RSS changes - so does the path(auto)....but instead of losing the original URL- it creates a new one. Which would mean converting to use a dash would double my number of aliases up to 25,200 in order to keep the original dash, and the initial conversion when I changed it in admin -> settings would probably timeout and not complete since there are so many.

There are so many 'unknowns' as to what would happen to my site if I converted from underscore to dash, I haven't found it worth the risk just yet. I mean, if I knew I wouldn't drop traffic - and could possibly increase pagerank from 4 to 5 or something, I would be all for it.

I use Drupal everyday in about a dozen of my web sites - check out http://www.smorgasbord.net and http://celebritynewslive.com.

That's fine for your current

greggles's picture

That's fine for your current site - but as general advice to people you should recommend the best or safest option. Which, if you're not invested in a path already, would be dashes.

--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO

robots.txt

oadaeh's picture

I once read somewhere (I don't remember exactly where, and didn't save the link) that using a default Allow * with various Disallow this/ and Disallow that/ gives bad bots ammunition and just points them in the directions you don't want them going. I was going to try going at it from the other side on a site and using a default Disallow * with various Allow this/ and Allow that/ to see if anything changed for that site. Has anyone tried that, and if so, what were your results for SEO and/or anti-mal-stuff?

what to disallow

jt6919's picture

I've always just told it what not to index by using disallow only....this is how my robots.txt is setup:

User-Agent: *
Disallow: /.htaccess
Disallow: /ImageMagick/
Disallow: /acidfree_tmp/
Disallow: /cgi-bin/
Disallow: /files/
Disallow: /gallery_old/
Disallow: /images/
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /themes/
Disallow: /update.php
Disallow: /styles-site.css
Disallow: /user

Disallow: /admin*

I use Drupal everyday in about a dozen of my web sites - check out http://www.smorgasbord.net and http://celebritynewslive.com.

Feedback

laura s's picture

Getting community feedback can be a tricky thing. You're asking for a conversation. Maybe your forum post dropped down and the people who might have commented did not see it. (I never did.) I will add, with all due respect, that demanding feedback will not yield much.

Okay, on your site:

Metatags are less important than they once were, but they still count. You might add nodewords to your module list.

But fundamentally the most important aspect of SEO, I believe, is clarity. You need to have content that attracts people, not just search bots. That means your posts must be informative, useful, easy to read and on a site that is easy to use.

Your post has a lot of information, but it could use some restructuring. I'd recommend making use of the header tag hierarchy (and possibly some CSS fixes) to make the structure of the page content clearer. Having a summary with target tag links to the relevant content down the page would also help.

I also feel that having a big huge adsense ad at the top makes the page not all that friendly. It is a disinvitation to actually read your site.

I'm not speaking to your site, but I've found it ironic that SEO websites in general area pretty messy, cluttered and hard to navigate. They're seemingly built for search bots, but are designed to do nothing but attract traffic to drive up CPMs. Your site at least doesn't have all that flashy crap or demands for private information.

But a little work at making it more user friendly could go along way, especially if you're trying to create conversations. (The highest-ranking sites on the net all enable conversations, something the SEO industry doesn't seem to do.)


Laura
pingVision, LLC

Laura Scott
PINGV | Strategy • Design • Drupal Development

Node (key)words

oadaeh's picture

I just thought I'd give a direct link to the Node (key)words module so that others could go straight to it:
http://drupal.org/project/nodewords

first question...

jt6919's picture

thanks for the feedback. I have many questions regarding your comments - but the first is, do you have web sites that you make adsense and affiliate revenue from? I'm trying to categorize your adsense comments, because (in my experience) I've found that most people adverse to the adsense block before the content are site owners that have never made money from adsense. I used to have a very small block of text links there, but since moving to this format I have changed my month adsense revenue from under a few dollars per month to hundreds+. In addition - my traffic has gone up, and time spent per page (and number of pages per visit) has increased.

I use nodewords everyday - but only for description ( I should add that ). Also (with due respect), I can point to dozens and dozens of places all over the web from respected authorities ( and search engines ) that metatags are dead. The most reliable of which is probably Danny Sullivan @ Search Engine Watch:

http://searchenginewatch.com/showPage.html?page=2167931
The Meta Keywords Tag

The meta keywords tag allows you to provide additional text for crawler-based search engines to index along with your body copy. How does this help you? Well, for most major crawlers, it doesn't. That's because most crawlers now ignore the tag. The few supporting it can be found on the Search Engine Features page).

also: Death of a metatag

you said:

But fundamentally the most important aspect of SEO, I believe, is clarity. You need to have content that attracts people, not just search bots. That means your posts must be informative, useful, easy to read and on a site that is easy to use.

Your post has a lot of information, but it could use some restructuring. I'd recommend making use of the header tag hierarchy (and possibly some CSS fixes) to make the structure of the page content clearer. Having a summary with target tag links to the relevant content down the page would also help.

While I appreciate the feedback, I would point out that these comments are about usability and not SEO - unless you believe that a cleaner and more usable site is better indexed and found in search. I have a very similar page on another topic with very nice header tag hierarchy - and I've seen no evidence that it ranks any better in search and traffic that this one.

SEO to me is getting your site ranked as many times as high as you can in all of the search engines for more traffic. Even though they sometimes overlap - the user experience and converting users to revenue are separate topics for me.

I use Drupal everyday in about a dozen of my web sites - check out http://www.smorgasbord.net and http://celebritynewslive.com.

it's a big circle, though...

greggles's picture

I guess you are referring to onsite optimization then, becuase when you say:

While I appreciate the feedback, I would point out that these comments are about usability and not SEO - unless you believe that a cleaner and more usable site is better indexed and found in search. I have a very similar page on another topic with very nice header tag hierarchy - and I've seen no evidence that it ranks any better in search and traffic that this one.

Higher usability means that you are more likely to get links from people. Organic links in content is almost certainly the most important thing to have for SEO for the next 5 years.

I'm almost never going to link to a site that is painfully ugly or has articles that are hard to force myself to read - not that your site is that way, but usability and offsite-optimization are definitely related.

Greg

--
Growing Venture Solutions
Drupal Implementation and Support in Denver, CO

ok......I see....

jt6919's picture

ok - I see what you mean:

Higher usability means that you are more likely to get links from people. Organic links in content is almost certainly the most important thing to have for SEO for the next 5 years.

This is though a kind of blanket statement, and refers to what you would do as a user. There are on the web may types of users, and many types of sites. I find most aspects of Yahoo! completely annoying and refuse to use most all of their services and sites for any reason. However - they are among the top 3 used sites in the world.

My point being - take for example sites with free song lyric listings or myspace layouts and html codes. These are not corporate sites, business sites, or even what you or I would consider useful informational sites. But both examples are among the top 100 searched keywords in the last 90 days out of 348 million searches in the top SE's. As such - they have some of the most annoying ads and they are sometimes painfully ugly and hard to read, and yet they have no problem at all getting links from people because they have what they wanted. Also - these types of sites (when created properly) can generate tens of thousands of dollars per month.

Those kind of sites aren't exactly 'black-hat' SEO, but they do ride the very blurry line in between.

I use Drupal everyday in about a dozen of my web sites - check out http://www.smorgasbord.net and http://celebritynewslive.com.

Search Engine Optimization Analysis

davidoc's picture

Although I have never worked on Drupal, but your article was really an interesting one.As far as Drupal optimization is concerned, you have sucessfully covered most of the related SEO topics.

Actual Search Engine Rankings depends on various factors. 1) On-Page Optimization, 2) Off-Page Optimization. Most of the topics were pertaining to On-page optimization, and were truly interesting. Although Search Engine Rankings and Search Engine Optimization are interrelated, but both are different.

These days search engines are following a complete different route, and webmasters do not understand why their website is not ranking on search engines even after following every SEO (Search Engine Optimization) guidelines/rule.

Google is following a monopoly to maximize their revenues through Google Adwords. This is something which is making Google lag behind other search engines like Yahoo, and MSN. Google has introduced several perplexing theories like "sandbox", "content sandbox/penaly" etc etc which is making things difficult for genuine websites to generate traffic out of Google particularly.

Google became popular because of its latest/accurate results. These days Google is surely aiming to deliver accurate results , but their ranking filters and algorithm can eventually hit back Google like a Boomerang.

They are trying to maximize their Adwords/Adsense revenue, and to enable this they are deliberately holding back ranking of several useful websites. This is baffling webmasters which are aiming to score top rankings on Google.

Whenever I have to search for some latest news, free software, free script etc etc, I have to rush immediately to MSN since it delivers latest results. Moreover MSN has emerged as a NO-ENTRY zone for Adult related websites. This will be a major boon for MSN. Their Image search is far much advance than Google :)

These days even Yahoo is gaining pace over Google by delivering latest results in their search engine. Surely Google has to wake up before it is too late. Their useless filters are making life miserable for genuine websites/forums to score top rankings on Google. Moreover it is making life miserable for search users by restricting their search results.

Nowdays, those websites which do not optimize at all are scoring top rankings on search engines :). This clearly shows that Google, Yahoo, and other search engines are giving less attention to on-page optimization.

David, E-Marketing Consultant
Search Engine Optimization India : SEO Hawk

David, E-Marketing Consultant
Search Engine Optimization - SEO Hawk, India