High performance

Trellon recently released the Xapian search module for Drupal. This replaces Drupal's standard search feature seamlessly (except for a core patch) with an interface to the Xapian search engine. There's a post about it over on Trellon.com.

The Observer Media Group, the company that runs Observer.com and Politicker.com, is hiring again!

I have a quick question for Drupal Performance Experts.

We wish to show stories from a particular section, Say Health, on all our pages.

The plan is to show the latest stories from that section as a Block on every page of the site.

What will be better way is terms of performance ?

Using RSS feed of that section in News aggregator ?
Or making a view to fetch Stories from that section and to display them on the pages ? (i guess this will use more resources)

Please help.

Thanks
Dhaliwal

http://events.gigaom.com/structure/08/schedule/

Structure 08 is a conference developed by award winning technology writer Om Malik. Structure 08 is created to inform businesses so they can plan their future computing infrastructure needs. With a uniquely privileged vantage brought by our industry-leading speakers and their deep insight, Structure 08 will allow our audience to filter the new technology explosion and determine the right choices. We will highlight the best implementations, case studies, ideas and startups.

Cloud Computing: Infrastructure for Entrepreneurs

Hi (This is a WIKI -- please edit as you see fit)

I thought I'd create a Wiki listing of all modules that can help beginners and intermediary drupal users (like myself) to optimise their websites' pageloads. Please enter the modules that you are familiar with, and few words of description/review/rating would also be helpful.

Join Amazon (Kieran Lal) and other Drupal community members and leaders at the Drupal Birds of a Feather session at the MySQL Conference & Expo on Wednesday, April 16th at 7:30pm. The BoF takes place in Ballroom “E” at the Santa Clara Convention Center. Afterwards, Sun will be giving away a Playstation 3 and Sun Fire X2100 M2 Server ($2,495) at their party next door. Wear your Drupal shirt!

More info about the conference:
http://en.oreilly.com/mysql2008

More info about the Sun party:

We are looking for a couple of part or full time php programmers to work on some exciting web projects.

You must have the following skillset:
Strong PHP coding skills (3+ years)
Ability to write SQL Queries on the fly
AJAX Experience
Experience with Drupal theme and module development

Our post if for part or full time as we know many of you already have full time jobs and are looking to pick up some extra side projects. That is acceptable for us as long as we know the situation up front.

Hours are completely flexible, we are just looking for some great developers.

So, some people have been asking about memcache for Drupal 6 and my personal modules have been neglected, but meanwhile I decided to come up with something new.

Let me introduce you to Cache Router. http://drupal.org/project/cacherouter

From the project page:

"CacheRouter is a caching system for Drupal allowing you to assign individual cache tables to specific cache technology. CacheRouter also utilizes the page_fast_cache part of Drupal in order to reduce the amount of resources needed for serving pages to anonymous users."

Moved to official ideas list

We have a website which is mainly based on the Search done by users.
Out of our total traffic is around 20k uniques per day, 40% is only searching on the site for information.

Can someone suggest the best module / system which will help us to increase the website speed.

Due to too many search queries, our server is running under heavy load.

Thanks
Dhaliwal

As information over the Internet becomes more and more revolved around communities and as the need for fast, dynamic content increases, simple caching can't do it anymore. In order to really stand out, a new application logic layer, which would interface with the database abstraction layer and provide tools for easy database sharding would be great. The idea is to be able to split data across several tables, but at the same time keep the caching features and the code intact(meaning that in order to use sharding features, one would only have to plot some rules for the data logic and that's it, no need to redo parts of the code or anything). The idea is inspired by the hibernate sharding feature, and by the ever increasing need for scalability.
Later edit: The nicest thing about sharding is that you can have parts of the data in different databases, spread across multiple servers.

hi,

i just submitted a simple patch to try to get the ball rolling on memcache support for drupal 6.

has anyone else started on this? is anyone interested on collaborating on updating the memcache module for drupal 6?

cheers
justin

This is the result of a chat in IRC one day. It got really positive feedback, but the tone was suggested it would never be done. I don't know why it couldn't actually happen.

Hello. I rescently released the guts of a a more complicated module called "MailQ". It's a module that "queues" mail up before sending, however still uses which ever smtp_library (either drupal_mail default or mimemail, etc.) to send the mail.
One can read more about the module here or try out the module at http://drupal.org/project/mailq.

Cross posting from here: http://drupal.org/node/219476

There will be a Drupal site showcase and case study contest at the upcoming Boston Drupalcon.

Submitted sites will be voted online and judged by a panel of community nominated judges and some great prizes will be awarded to the winners! This is a chance to have all your hard work creating amazing Drupal sites recognized.

This is a planning wiki page for those who want to get involved in this.

TODO (Please expand on your section below)

-How performance impacts your business - Michael Myers

• Webserver, File systems, Squid, Architecture tuning ?

• Squid

<

ul>

We are running community web site with medium of 2000-3000 visits per day, according to Google Analytics.
We used dedicated server AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ with 2GB RAM

mysql Ver 14.7 Distrib 4.1.11, for pc-linux-gnu (x86_64)
Here is my.cnf file

We upgraded to: AMD Athlon(tm) 64 X2 Dual Core Processor 6000+, With 8 GB DDR2 RAM

mysql Ver 14.12 Distrib 5.0.32, for pc-linux-gnu (x86_64) using readline 5.2

This article I wrote should appeal to people interested in high performance Drupal sites. Most of the performance gains are achieved by doing things with the Javascript and Drupal's File API.

http://wimleers.com/article/improving-drupals-page-loading-performance

You can expect another article soon, about the same topic, but then putting it to practice. I'll use the CDN integration module (of which I'm also the maintainer) and the included core patches to do it in just a few simple steps.

I'm assuming it's too late, but if not here's an easy performance win for D6:

http://drupal.org/node/215080

<

blockquote>The type column of the {system} table is currently a varchar(255) field. Yet in 99% of all Drupal installations it stores either the text 'module' or the text 'theme' and nothing else. This is already a strong case for making it an int column and defining constants for module and theme. Then, on every page, there is this query:
SELECT filename FROM system WHERE name = 'user' AND type = 'module';

MySQL AB and KernelTrap.org's Jeremy Andrews are partnering up to offer an online presentation titled "Achieving Optimal MySQL Performance For Drupal". Aiming to provide a better understanding of how to properly monitor and tune your MySQL database, the online Webinar will take place on Thursday, January 31st, 2008, at 16:00 UTC (11:00 am EST). The presentation will last 45 minutes, followed by 15 minutes for questions and answers. You can sign up for the free event here.

I've been using yslow to review a few sites and always leave one last step out: move javascript to the bottom of the page

So, on a test site I just moved my

to the bottom of the page.tpl.php right before the print $closure.

Let's say you've got a site that you're taking into a phase that will have lots of traffic.

Let's say you've decided to centralize on a single database machine.

Let's say you have many load balanced web servers on the front end.

Let's say you're considering putting a memcache on each of the front ends, and taking advantage of the fact that it can be configured to write to the DB, and read from the memcache.

What about the issue that as soon as you have a write to the DB, most of your memcaches are now out of date?

http://twtitw.firebus.com/node/10

Trellon is announcing multiple vacancies, effective immediately. Please send resumes including cover letters and salary requirements to jobs@trellon.com. No phone calls please.

Position Title: Technical Lead
Reports To: Director of Development
Term: Permanent, Full Time
Classification: Professional, technical
Travel Involved: Infrequent
Salary: $70,000 - $120,000 / annual
Location: Telecommute

The New York Observer is currently hiring for PHP developers with Drupal experience. We only require knowledge of the LAMP stack in general; Drupal skills are a plus, but can be learned on the job by someone with PHP talent. Also, you should either live in or be willing to relocate to New York City. We value demonstrable programming skills over formal education and will consider intelligent applicants of all skill levels.

Please send a resume and two code samples (we prefer a Drupal sample and an OO PHP sample, but whatever you have is fine) to asmith@observer.com.

As a site goes live and we do performance tweaking and benchmarking one of the common results is adding database indexes. I post this into the CCK group since one of the common sources of slow queries is for queries against the CCK tables. For example, a site that has a block that queries a field table to look for information about that field for every page load can quickly make its way into the slow query log. Adding an index to the field is likely to be the best solution.

Just started reading. Looks good so far.

http://www.johnandcailin.com/blog/john/scaling-drupal-open-source-infras...

I'm cross posting this from here in case anyone else has any information to share: http://drupal.org/node/187464

Hi Fellow Drupalers,
I'm working on a session for BADCAMP (http://badcamp07.org) where we will walk through some cool, popular and high performance Drupal sites. The main focus here is high-profile, big companies, high traffic. What is the biggest and highest traffic Drupal site out there?

Hi,

We need our www.projectstars.com site to run a lot faster than it is at the moment and wondered if anyone with experience could advise us on a good option for upgrading from our current virtual server. We're using memcache, block caching, and general drupal caching but as most of the site is dynamic I'm guessing that having a fast database server would help.

Hi guys,

What you feel about choosing CCK for heavy traffic driven website?

Or is it better to make our own content type?

plus point of using CCK is that it will take very less time to setup & ready to use but my main concern is that can CCK support a website which gets 1-2 million hits a day.

Your inputs will greatly help in taking this decision.

Thanks in advance,
Sumoanand

Introduction

projectstars is a new business network which gives experts the chance to show off their experience, network, and find work. Consisting of 300 enterprise communities, we are the first business network to be giving stock to our members in a 'Blog for Stock' event where every three months the top 100 projectstars receive shares in projectstars inc..

Our requirements

Morris DigitalWorks -- the creators of SavannahNow.com, BlufftonToday.com, and the new Skirt.com -- is currently seeking 2 Drupal developers to work as in-house contractors in Augusta, GA.

Morris DigitalWorks is a division of the Morris Communication Company, LLC, a privately-held international media company.

A quick browse through many public-facing websites at the University of Prince Edward Island reveals a growing trend: UPEI has been converting many of its existing websites to Drupal.

I've done a small patch, which makes fast private file transfers through X-Sendfile possible.

Instead of changing UI as the old one, this one just provides the possibility to specify a handler for file transferring:
http://drupal.org/node/74472#comment-323445

fago

OpenCraft hosted a 3.5 hour seminar that I presented on Drupal performance tuning and optimization. You can find the slides from the presentation there, which can be useful to some of you.

I create a 5.2 and 6.x patch for measuring how much memory a module takes, and also how much the whole Drupal page load takes.

The results are published here: Measuring memory consumption by Drupal bootstrap and modules.

Short version: in Drupal 6, modules are better off than in 5.2, but the bootstrap is a tad larger.

If anyone is interested in replicating the results with more realistic install profiles, please do so and publish your results/findings/recommendations.

Have a strong desire to help make the world a better place? Know what scalability really means and how to deal with millions of users at a time? Can you think of better ways to organize the information on your favorite sites? Do you use the word code as a verb and enjoy sharing your knowledge with others?

A file-based caching mechanism for serving up cached Drupal pages instead of using the database 'cache' table. This approach is generally faster since the web server doesn't need to initialize a database connection and crunch through a bunch of PHP files for an anonymous user hit.

This is in release candidate state.

Hi, Drupal.org has been running project honeypot.

Today it was able to reduce between 5-10% of their traffic by blocking bad IPs. If you are running a high traffic site you might want to implement Project Honeypot. http://www.projecthoneypot.org

Want to live and work in New Zealand?

We are expanding our drupal work, and are looking for more developers who love drupal.

Other Opportunities within catalyst to learn and work with other opensource technologies as well as drupal including php, perl, java, ruby, erlang, more...

send your CV to brenda@catalyst.net.nz. Please include links to modules you've contributed or patches/testing/benchmarking you've done on drupal.org, or another open source project.

http://catalyst.net.nz

CivicSpace runs over 600 Drupal and CiviCRM sites. We can bring our experience working on Drupal high performance issues to help improve the performance of your Drupal sites. We will provide one on one training with your website staff to make assessments of your sites performance and provide you with tools to measure your web sites performance.

1) MySQL database measurement and tuning
2) Apache configuration and tuning
3) PHP configuration and tuning
4) Drupal performance using devel module and other tools
5) Provide patches to improve the performance of your site.

Kieran Lal

Here's a script that was mentioned on the infrastructure list for tuning your MySQL database.

MySQL tuning script

Khalid, Dries and I posted an update on the Drupal Infrastructure today.

Improvements to the Drupal.org infrastructure

Thanks to Gerhard and the OSUOSL team for all the hard work to make these improvements actually happen.

Kieran

7348 Queries

Taking 16 to 46 seconds to complete

SELECT n.nid, n.title, l.comment_count FROM node n INNER JOIN node_comment_statistics l ON n.nid = l.nid WHERE n.type = 'XXX' AND n.status = XXX ORDER BY n.nid DESC LIMIT XXX, XXX;

63 hours total, assuming an average time of 31 seconds.

3574 Queries

Taking 16 to 67 seconds to complete

41 hours total, assuming an average of 41.5 seconds

We're a small, flexible, and energetic company - Gradspot dot com - that recently launched our website and services, and we're looking for a PHP hacker who isn't afraid to get drunk on drupal and join our team.

Hi, if you want to tune your database, a good place to start is finding out what the slowest queries are by total time spent.

Here's a nice tool from the MySQL performance blog.

Add this line to your my.cnf and restart your database server:

log-slow-queries = /var/log/mysql/mysql-slow.log

Add execute permissions to mysql_slow_log_parser, it's a perl script.

Here's an example:

Starting...

2818 Queries

Total time: 2821, Average time: 1.00106458481192

Provide support for database replciation

This is something we would like to see available for Drupal.org.

Cheers,
Kieran

Hi all. I've been working on a new project called Advcache (Advanced Cache) which is a package of caching patches (and a controller module) for Drupal 5. It allows you to cache many things which don't get cached normally, and is intended to increase the performance of Drupal sites, especially for authenticated users. Current caching targets include nodes, comments, taxonomy (trees, terms, vocabularies and node-term bindings), paths, search queries, and forums.