High performance

This group is dedicated to solutions and approaches for high traffic, high performing Drupal sites. As such, it will deal with a lot of information around the rest of a typical Drupal "stack" -- the operating system, web server, database, and PHP tweaks that combine to support the Drupal application.

The Performance and Scalability forum on Drupal.org is where else to look.

Improving Drupal's Search Speed Under Load Conditions

techsoldaten's picture
public
techsoldaten - Wed, 2008-05-14 02:04

Trellon recently released the Xapian search module for Drupal. This replaces Drupal's standard search feature seamlessly (except for a core patch) with an interface to the Xapian search engine. There's a post about it over on Trellon.com.


Developer | The New York Observer

public
netaustin - Thu, 2008-05-08 22:33
Employment type: 
Full time
Employment type: 
Part time
Employment type: 
Contract
Telecommute: 
Allowed

The Observer Media Group, the company that runs Observer.com and Politicker.com, is hiring again!

Using News aggregator or Views/Block

gdtechindia's picture
public
gdtechindia - Thu, 2008-05-01 03:03

I have a quick question for Drupal Performance Experts.

We wish to show stories from a particular section, Say Health, on all our pages.

The plan is to show the latest stories from that section as a Block on every page of the site.

What will be better way is terms of performance ?

Using RSS feed of that section in News aggregator ?
Or making a view to fetch Stories from that section and to display them on the pages ? (i guess this will use more resources)

Please help.

Thanks
Dhaliwal


Structure '08 Innovations in Internet Infrastructure - San Francisco

Amazon's picture
public
Amazon - Sat, 2008-04-26 15:01
Start: 
2008-06-25 08:00 US/Pacific - 2008-06-25 20:00 US/Pacific

http://events.gigaom.com/structure/08/schedule/

Structure 08 is a conference developed by award winning technology writer Om Malik. Structure 08 is created to inform businesses so they can plan their future computing infrastructure needs. With a uniquely privileged vantage brought by our industry-leading speakers and their deep insight, Structure 08 will allow our audience to filter the new technology explosion and determine the right choices. We will highlight the best implementations, case studies, ideas and startups.

Cloud Computing: Infrastructure for Entrepreneurs


Performance Optimization Modules [WIKI]

public
drupalina - Sat, 2008-04-19 06:58

Hi (This is a WIKI -- please edit as you see fit)

I thought I'd create a Wiki listing of all modules that can help beginners and intermediary drupal users (like myself) to optimise their websites' pageloads. Please enter the modules that you are familiar with, and few words of description/review/rating would also be helpful.

South Bay: Birds of a Feather session at MySQL Con followed by Sun & MySQL party (Both free attendance)

sooz's picture
public
sooz - Mon, 2008-04-14 21:44
Start: 
2008-04-16 19:30 America/Los_Angeles

Join Amazon (Kieran Lal) and other Drupal community members and leaders at the Drupal Birds of a Feather session at the MySQL Conference & Expo on Wednesday, April 16th at 7:30pm. The BoF takes place in Ballroom “E” at the Santa Clara Convention Center. Afterwards, Sun will be giving away a Playstation 3 and Sun Fire X2100 M2 Server ($2,495) at their party next door. Wear your Drupal shirt!

More info about the conference:
http://en.oreilly.com/mysql2008

More info about the Sun party:


Drupal Developers (Cebu / Philippines) | THE NEB INTERNATIONAL PTY LIMITED

njclark007's picture
public
njclark007 - Sat, 2008-04-12 12:05
Employment type: 
Full time
Employment type: 
Part time
Employment type: 
Contract
Telecommute: 
Allowed

We are looking for a couple of part or full time php programmers to work on some exciting web projects.

You must have the following skillset:
Strong PHP coding skills (3+ years)
Ability to write SQL Queries on the fly
AJAX Experience
Experience with Drupal theme and module development

Our post if for part or full time as we know many of you already have full time jobs and are looking to pick up some extra side projects. That is acceptable for us as long as we know the situation up front.

Hours are completely flexible, we are just looking for some great developers.


New caching module for Drupal 6

slantview@drupal.org's picture
public
slantview@drupal.org - Mon, 2008-03-31 18:19

So, some people have been asking about memcache for Drupal 6 and my personal modules have been neglected, but meanwhile I decided to come up with something new.

Let me introduce you to Cache Router. http://drupal.org/project/cacherouter

From the project page:

"CacheRouter is a caching system for Drupal allowing you to assign individual cache tables to specific cache technology. CacheRouter also utilizes the page_fast_cache part of Drupal in order to reduce the amount of resources needed for serving pages to anonymous users."


More Search used; What should be done to increase performance

gdtechindia's picture
public
gdtechindia - Thu, 2008-03-20 11:54

We have a website which is mainly based on the Search done by users.
Out of our total traffic is around 20k uniques per day, 40% is only searching on the site for information.

Can someone suggest the best module / system which will help us to increase the website speed.

Due to too many search queries, our server is running under heavy load.

Thanks
Dhaliwal


Database sharding and other high scalability features

TechMan's picture
public
TechMan - Tue, 2008-03-18 17:12

As information over the Internet becomes more and more revolved around communities and as the need for fast, dynamic content increases, simple caching can't do it anymore. In order to really stand out, a new application logic layer, which would interface with the database abstraction layer and provide tools for easy database sharding would be great. The idea is to be able to split data across several tables, but at the same time keep the caching features and the code intact(meaning that in order to use sharding features, one would only have to plot some rules for the data logic and that's it, no need to redo parts of the code or anything). The idea is inspired by the hibernate sharding feature, and by the ever increasing need for scalability.
Later edit: The nicest thing about sharding is that you can have parts of the data in different databases, spread across multiple servers.


updating memcache module to version 6?

justinrandell's picture
public
justinrandell - Tue, 2008-03-11 06:57

hi,

i just submitted a simple patch to try to get the ball rolling on memcache support for drupal 6.

has anyone else started on this? is anyone interested on collaborating on updating the memcache module for drupal 6?

cheers
justin


APC pre-compiled distro

public
deekayen - Mon, 2008-03-10 20:01

This is the result of a chat in IRC one day. It got really positive feedback, but the tone was suggested it would never be done. I don't know why it couldn't actually happen.

MailQ: More control over Mail

public
Souvent22@drupal.org - Fri, 2008-02-22 06:47

Hello. I rescently released the guts of a a more complicated module called "MailQ". It's a module that "queues" mail up before sending, however still uses which ever smtp_library (either drupal_mail default or mimemail, etc.) to send the mail.
One can read more about the module here or try out the module at http://drupal.org/project/mailq.

Boston Drupalcon showcase and case study contest

ChrisB's picture
public
ChrisB - Fri, 2008-02-08 22:08

Cross posting from here: http://drupal.org/node/219476

There will be a Drupal site showcase and case study contest at the upcoming Boston Drupalcon.

Submitted sites will be voted online and judged by a panel of community nominated judges and some great prizes will be awarded to the winners! This is a chance to have all your hard work creating amazing Drupal sites recognized.


DrupalCon Boston 2008 Performance session planning

kbahey's picture
public
kbahey - Tue, 2008-02-05 19:34

This is a planning wiki page for those who want to get involved in this.

TODO (Please expand on your section below)

-How performance impacts your business - Michael Myers

  • Webserver, File systems, Squid, Architecture tuning ?

  • Squid

<

ul>

  • What is Squid?

    • Caching Proxy, originally just a caching proxy
    • Moon-lights as a Reverse Caching Proxy, a.k.a. HTTP Accelerator
    • Can increase speed and decrease load by caching some dynamic content and almost all static content

  • Squid at the Open Source Lab

    <

    ul>

  • Discovered by Scott Kveton


  • Mysql 5.x performance problems after upgrade of drupal version and hardware

    Ainur@drupal.org's picture
    public
    Ainur@drupal.org - Thu, 2008-01-31 12:48

    We are running community web site with medium of 2000-3000 visits per day, according to Google Analytics.
    We used dedicated server AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ with 2GB RAM

    mysql Ver 14.7 Distrib 4.1.11, for pc-linux-gnu (x86_64)
    Here is my.cnf file

    We upgraded to: AMD Athlon(tm) 64 X2 Dual Core Processor 6000+, With 8 GB DDR2 RAM

    mysql Ver 14.12 Distrib 5.0.32, for pc-linux-gnu (x86_64) using readline 5.2


    Improving Drupal's page loading performance

    public
    Wim Leers@drupal.org - Wed, 2008-01-30 22:51

    This article I wrote should appeal to people interested in high performance Drupal sites. Most of the performance gains are achieved by doing things with the Javascript and Drupal's File API.

    http://wimleers.com/article/improving-drupals-page-loading-performance

    You can expect another article soon, about the same topic, but then putting it to practice. I'll use the CDN integration module (of which I'm also the maintainer) and the included core patches to do it in just a few simple steps.

    Are performance optimisations still going into D6?

    robertDouglass's picture
    public
    robertDouglass - Mon, 2008-01-28 15:07

    I'm assuming it's too late, but if not here's an easy performance win for D6:

    http://drupal.org/node/215080

    <

    blockquote>The type column of the {system} table is currently a varchar(255) field. Yet in 99% of all Drupal installations it stores either the text 'module' or the text 'theme' and nothing else. This is already a strong case for making it an int column and defining constants for module and theme. Then, on every page, there is this query:
    SELECT filename FROM system WHERE name = 'user' AND type = 'module';


    Achieving Optimal MySQL Performance For Drupal

    Jeremy@kerneltrap.org's picture
    public
    Jeremy@kerneltr... - Sat, 2008-01-26 04:06

    MySQL AB and KernelTrap.org's Jeremy Andrews are partnering up to offer an online presentation titled "Achieving Optimal MySQL Performance For Drupal". Aiming to provide a better understanding of how to properly monitor and tune your MySQL database, the online Webinar will take place on Thursday, January 31st, 2008, at 16:00 UTC (11:00 am EST). The presentation will last 45 minutes, followed by 15 minutes for questions and answers. You can sign up for the free event here.


    Moving JS to the Bottom of the Page

    greggles's picture
    public
    greggles - Wed, 2008-01-23 18:13

    I've been using yslow to review a few sites and always leave one last step out: move javascript to the bottom of the page

    So, on a test site I just moved my

    <?php
    print $scripts
    ?>

    to the bottom of the page.tpl.php right before the print $closure.


    Multiple Memcache Instances Keeping Pace with each other?

    nally's picture
    public
    nally - Wed, 2008-01-16 20:00

    Let's say you've got a site that you're taking into a phase that will have lots of traffic.

    Let's say you've decided to centralize on a single database machine.

    Let's say you have many load balanced web servers on the front end.

    Let's say you're considering putting a memcache on each of the front ends, and taking advantage of the fact that it can be configured to write to the DB, and read from the memcache.

    What about the issue that as soon as you have a write to the DB, most of your memcaches are now out of date?


    Multiple Positions - Technical lead, lead developer, senior developer | Trellon, LLC

    techsoldaten's picture
    public
    techsoldaten - Wed, 2007-12-19 13:22
    Employment type: 
    Full time
    Telecommute: 
    Allowed

    Trellon is announcing multiple vacancies, effective immediately. Please send resumes including cover letters and salary requirements to jobs@trellon.com. No phone calls please.

    Position Title: Technical Lead
    Reports To: Director of Development
    Term: Permanent, Full Time
    Classification: Professional, technical
    Travel Involved: Infrequent
    Salary: $70,000 - $120,000 / annual
    Location: Telecommute


    PHP Developer | The New York Observer

    public
    netaustin@drupal.org - Tue, 2007-12-11 21:12
    Employment type: 
    Full time
    Telecommute: 
    Not allowed

    The New York Observer is currently hiring for PHP developers with Drupal experience. We only require knowledge of the LAMP stack in general; Drupal skills are a plus, but can be learned on the job by someone with PHP talent. Also, you should either live in or be willing to relocate to New York City. We value demonstrable programming skills over formal education and will consider intelligent applicants of all skill levels.

    Please send a resume and two code samples (we prefer a Drupal sample and an OO PHP sample, but whatever you have is fine) to asmith@observer.com.

    RFC: Best practices for managing indexes (particularly on cck tables)

    greggles's picture
    public
    greggles - Mon, 2007-12-10 21:21

    As a site goes live and we do performance tweaking and benchmarking one of the common results is adding database indexes. I post this into the CCK group since one of the common sources of slow queries is for queries against the CCK tables. For example, a site that has a block that queries a field table to look for information about that field for every page load can quickly make its way into the slow query log. Adding an index to the field is likely to be the best solution.


    Scaling high traffic Drupal sites - A six part series

    Amazon's picture
    public
    Amazon - Fri, 2007-11-16 00:36

    Whats your primary method to gain performance?

    public
    tjholowaychuk - Thu, 2007-11-15 01:45
    Refactor code
    14% (4 votes)
    Caching content
    45% (13 votes)
    Opcode Caching
    21% (6 votes)
    Load balancing server setup
    3% (1 vote)
    More ram / better server CPU etc
    17% (5 votes)
    Total votes: 29

    Popular, high profile, high traffic Drupal sites for BADCAMP and Drupal.org

    ChrisB's picture
    public
    ChrisB - Mon, 2007-10-29 09:54

    I'm cross posting this from here in case anyone else has any information to share: http://drupal.org/node/187464

    Hi Fellow Drupalers,
    I'm working on a session for BADCAMP (http://badcamp07.org) where we will walk through some cool, popular and high performance Drupal sites. The main focus here is high-profile, big companies, high traffic. What is the biggest and highest traffic Drupal site out there?


    Can anyone recommend a server setup?

    projectstars's picture
    public
    projectstars - Thu, 2007-10-25 17:04

    Hi,

    We need our www.projectstars.com site to run a lot faster than it is at the moment and wondered if anyone with experience could advise us on a good option for upgrading from our current virtual server. We're using memcache, block caching, and general drupal caching but as most of the site is dynamic I'm guessing that having a fast database server would help.


    content type for heavy traffic driven website

    public
    sumoanand - Tue, 2007-10-23 10:52

    Hi guys,

    What you feel about choosing CCK for heavy traffic driven website?

    Or is it better to make our own content type?

    plus point of using CCK is that it will take very less time to setup & ready to use but my main concern is that can CCK support a website which gets 1-2 million hits a day.

    Your inputs will greatly help in taking this decision.

    Thanks in advance,
    Sumoanand

    Drupal optimization ninja, experienced with ecommerce, and ideally navigation guru! Facebook a plus... | projectstars inc.

    projectstars's picture
    public
    projectstars - Sun, 2007-10-21 12:19
    Employment type: 
    Contract
    Telecommute: 
    Allowed

    Introduction

    projectstars is a new business network which gives experts the chance to show off their experience, network, and find work. Consisting of 300 enterprise communities, we are the first business network to be giving stock to our members in a 'Blog for Stock' event where every three months the top 100 projectstars receive shares in projectstars inc..

    Our requirements


    Drupal Developer: On site | Morris DigitalWorks

    agentrickard@drupal.org's picture
    public
    agentrickard@dr... - Sat, 2007-10-20 02:19
    Employment type: 
    Contract
    Telecommute: 
    Not allowed

    Morris DigitalWorks -- the creators of SavannahNow.com, BlufftonToday.com, and the new Skirt.com -- is currently seeking 2 Drupal developers to work as in-house contractors in Augusta, GA.

    Morris DigitalWorks is a division of the Morris Communication Company, LLC, a privately-held international media company.


    UPEI.ca goes Drupal

    Ryan Palmer@drupal.org's picture
    public
    Ryan Palmer@dru... - Fri, 2007-10-19 06:16

    A quick browse through many public-facing websites at the University of Prince Edward Island reveals a growing trend: UPEI has been converting many of its existing websites to Drupal.


    Fast private file transfers the second

    fago@drupal.org's picture
    public
    fago@drupal.org - Thu, 2007-10-18 17:55

    I've done a small patch, which makes fast private file transfers through X-Sendfile possible.

    Instead of changing UI as the old one, this one just provides the possibility to specify a handler for file transferring:
    http://drupal.org/node/74472#comment-323445

    fago


    Are you storing your Drupal sites or configs in a version control system? If so, which?

    ccharlton's picture
    public
    ccharlton - Mon, 2007-09-17 21:12
    SVN (Subversion)
    51% (72 votes)
    CVS
    6% (9 votes)
    Other (please comment which)
    7% (10 votes)
    No, but I'd like to!
    24% (34 votes)
    No, I don't store my drupal sites in a version control system of any kind.
    11% (16 votes)
    Total votes: 141

    Presentation: Drupal performance tuning and optimization

    kbahey's picture
    public
    kbahey - Mon, 2007-09-03 08:45

    OpenCraft hosted a 3.5 hour seminar that I presented on Drupal performance tuning and optimization. You can find the slides from the presentation there, which can be useful to some of you.


    Drupal memory consumption by core and modules

    kbahey's picture
    public
    kbahey - Mon, 2007-09-03 08:45

    I create a 5.2 and 6.x patch for measuring how much memory a module takes, and also how much the whole Drupal page load takes.

    The results are published here: Measuring memory consumption by Drupal bootstrap and modules.

    Short version: in Drupal 6, modules are better off than in 5.2, but the bootstrap is a tad larger.

    If anyone is interested in replicating the results with more realistic install profiles, please do so and publish your results/findings/recommendations.


    Drupal Gunslinger | Full Time or Contract | Trellon, LLC

    techsoldaten's picture
    public
    techsoldaten - Thu, 2007-08-23 13:59
    Employment type: 
    Contract
    Telecommute: 
    Allowed

    Have a strong desire to help make the world a better place? Know what scalability really means and how to deal with millions of users at a time? Can you think of better ways to organize the information on your favorite sites? Do you use the word code as a verb and enjoy sharing your knowledge with others?


    Reducing DB load/need with fastpath file system cache

    Amazon's picture
    public
    Amazon - Sat, 2007-08-18 01:39

    A file-based caching mechanism for serving up cached Drupal pages instead of using the database 'cache' table. This approach is generally faster since the web server doesn't need to initialize a database connection and crunch through a bunch of PHP files for an anonymous user hit.

    This is in release candidate state.


    Blocking bad IPs with project honeypot

    Amazon's picture
    public
    Amazon - Sat, 2007-08-18 01:34

    Hi, Drupal.org has been running project honeypot.

    Today it was able to reduce between 5-10% of their traffic by blocking bad IPs. If you are running a high traffic site you might want to implement Project Honeypot. http://www.projecthoneypot.org


    Developer | Catalyst IT

    Shiny's picture
    public
    Shiny - Tue, 2007-08-07 22:42
    Employment type: 
    Full time
    Telecommute: 
    Not allowed

    Want to live and work in New Zealand?

    We are expanding our drupal work, and are looking for more developers who love drupal.

    Other Opportunities within catalyst to learn and work with other opensource technologies as well as drupal including php, perl, java, ruby, erlang, more...

    send your CV to brenda@catalyst.net.nz. Please include links to modules you've contributed or patches/testing/benchmarking you've done on drupal.org, or another open source project.

    http://catalyst.net.nz

    Catalyst logo


    Performance consultant for your site | CivicSpace available for performance tuning your site

    Amazon's picture
    public
    Amazon - Tue, 2007-08-07 08:43
    Employment type: 
    Contract
    Telecommute: 
    Allowed

    CivicSpace runs over 600 Drupal and CiviCRM sites. We can bring our experience working on Drupal high performance issues to help improve the performance of your Drupal sites. We will provide one on one training with your website staff to make assessments of your sites performance and provide you with tools to measure your web sites performance.

    1) MySQL database measurement and tuning
    2) Apache configuration and tuning
    3) PHP configuration and tuning
    4) Drupal performance using devel module and other tools
    5) Provide patches to improve the performance of your site.

    Kieran Lal


    Automated MySQL tuning script

    Amazon's picture
    public
    Amazon - Tue, 2007-08-07 05:56

    Here's a script that was mentioned on the infrastructure list for tuning your MySQL database.

    MySQL tuning script


    Improvements to the Drupal.org infrastructure

    Amazon's picture
    public
    Amazon - Tue, 2007-07-31 03:57

    Khalid, Dries and I posted an update on the Drupal Infrastructure today.

    Improvements to the Drupal.org infrastructure

    Thanks to Gerhard and the OSUOSL team for all the hard work to make these improvements actually happen.

    Kieran


    Which query takes 63 hours of cummulative wait time to process on Drupal.org?

    Amazon's picture
    public
    Amazon - Sat, 2007-07-21 03:08

    7348 Queries

    Taking 16 to 46 seconds to complete

    SELECT n.nid, n.title, l.comment_count FROM node n INNER JOIN node_comment_statistics l ON n.nid = l.nid WHERE n.type = 'XXX' AND n.status = XXX ORDER BY n.nid DESC LIMIT XXX, XXX;

    63 hours total, assuming an average time of 31 seconds.

    3574 Queries

    Taking 16 to 67 seconds to complete

    41 hours total, assuming an average of 41.5 seconds


    PHP Lead Developer - Full Time | Gradspot.com

    public
    stuart - Mon, 2007-07-16 03:37
    Telecommute: 
    Allowed

    We're a small, flexible, and energetic company - Gradspot dot com - that recently launched our website and services, and we're looking for a PHP hacker who isn't afraid to get drunk on drupal and join our team.

    MySQL slow query log analyzer orders by total time spent on queries

    Amazon's picture
    public
    Amazon - Mon, 2007-06-18 04:02

    Hi, if you want to tune your database, a good place to start is finding out what the slowest queries are by total time spent.

    Here's a nice tool from the MySQL performance blog.

    Add this line to your my.cnf and restart your database server:

    log-slow-queries = /var/log/mysql/mysql-slow.log

    Add execute permissions to mysql_slow_log_parser, it's a perl script.

    Here's an example:

    Starting...

    2818 Queries

    Total time: 2821, Average time: 1.00106458481192


    Read from slave database patch to help Drupal.org

    Amazon's picture
    public
    Amazon - Tue, 2007-06-12 17:34

    Provide support for database replciation

    This is something we would like to see available for Drupal.org.

    Cheers,
    Kieran


    Advcache: call for benchmarks

    robertDouglass's picture
    public
    robertDouglass - Thu, 2007-05-31 20:54

    Hi all. I've been working on a new project called Advcache (Advanced Cache) which is a package of caching patches (and a controller module) for Drupal 5. It allows you to cache many things which don't get cached normally, and is intended to increase the performance of Drupal sites, especially for authenticated users. Current caching targets include nodes, comments, taxonomy (trees, terms, vocabularies and node-term bindings), paths, search queries, and forums.


    Syndicate content