Tips with tracing performance issue please

Events happening in the community are now at Drupal community events on www.drupal.org.
tiendoan's picture

When I saw this posting, http://groups.drupal.org/node/20494, I thought déja vu but as it turned out, no such luck, though the end result is the same, or worse.

Situation:

development server: XAMPP 1.6.8 on vista premium, PHP Version 5.2.6, acquia drupal 6.10, eAccelerator 0.9.5.3 (couldn't get binary for APC for this PHP version).

Staging server: Ubuntu 8.10, PHP Version 5.2.6-2ubuntu4.1, APC 3.0.18

Production server: Red Hat Enterprise Linux Server release 5 (Tikanga), PHP Version 5.1.6 (latest release for Red Hat), APC 3.0.19 on a VPS behind a watchguard firewall (at one point, I moved it outside the firewall and made no difference).

Acquia drupal (6.10 based) installed on all systems.

(now the sad part ...) Of course everything (i.e., the fully developed site with contents) runs perfect on the development server and the staging server. The production server is another story altogether. It was a clean install of Redhat, after which I updated the php installation, installed apc, and a clean stock Acquia Drupal.

Immediately the problem appears. Except for the "create content" pages and sub-pages, all other logged-in user pages are unacceptable slow (slow as in > 75 seconds). In the anonymous mode, the home page loads in less than 1.5 s. In an effort to debug this problem, I installed the following modules: drupal firebug, devel, and on the client side firefox browser, installed yslow, drupal for firebug plugin. I enabled performance logging.

Here's a tail of the log:

num Date Path Memory (MB) Milliseconds (Total) Anonymous? # Queries Query Milliseconds
86 04/07/2009 - 14:15 admin/firebug/exec 0.00 22 No 8 6
85 04/07/2009 - 14:15 node/add 0.00 1803 No 44 25
84 04/07/2009 - 14:15 admin/content 0.00 77941 No 177 101
83 04/07/2009 - 14:15 admin/user 0.00 75621 No 177 72
82 04/07/2009 - 14:15 admin/build 0.00 75818 No 177 77

note: the time is in ms.

and here's the APC config (I left it on default)

apc
APC Support enabled
Version 3.0.19
MMAP Support Enabled
MMAP File Mask no value
Locking type pthread mutex Locks
Revision $Revision: 3.154.2.5 $
Build Date Mar 20 2009 14:37:22

Directive Local Value Master Value
apc.cache_by_default On On
apc.coredump_unmap Off Off
apc.enable_cli Off Off
apc.enabled On On
apc.file_update_protection 2 2
apc.filters no value no value
apc.gc_ttl 3600 3600
apc.include_once_override Off Off
apc.max_file_size 1M 1M
apc.mmap_file_mask no value no value
apc.num_files_hint 1000 1000
apc.report_autofilter Off Off
apc.shm_segments 1 1
apc.shm_size 30 30
apc.slam_defense 0 0
apc.stat On On
apc.stat_ctime Off Off
apc.ttl 0 0
apc.user_entries_hint 4096 4096
apc.user_ttl 0 0
apc.write_lock On On

Question: where do I go from here?

Regards
Tien Doan

Comments

So with what you did so far

Alexander Langer@drupal.org's picture

So with what you did so far (I guess you activated Drupal's caching) you already tracked the problem pretty much down to the most probable cause: the database.

Can you give us some "hardware" specs on your VPS and your my.cnf? Just recently I saw a newly installed virtual machine of a big hosting company leaving MySQL's query cache deactivated in the system's default config for unknown reasons. No wonder the machine performed poorly..

Alex

my.cnf

tiendoan's picture

Hi Alexander

Thanks for taking a stab at this.

uname -a returns

Linux mecLOC 2.6.18-92.1.10.el5.xs5.0.0.39xen #1 SMP Thu Aug 7 16:25:59 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

my.cnf file content=

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=******
;Default to using old password format for compatibility with mysql 3.x
;clients (those using the mysqlclient10 compatibility package).
old_passwords=1

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

=====

Based on that, I guess the mysql query cache is not enabled. A quick question: why do you think this is the rsn for the slow down? and why would it only affects the logged in user? I didn't think of it b/c the perf log says that most of the queries were done in very quickly. As a matter of fact, on the long log version, I have no queries that set off the flag.

If Drupal's caching is on,

Alexander Langer@drupal.org's picture

If Drupal's caching is on, Drupal does some lightweight database queries on the cache tables to serve requests for anonymous users. Logged in users usually trigger a lot more and more complex database queries. So if anonymous taffic goes fast it indicates that pure cpu power (or whatever php and apc deliver) are probably not the bottleneck.

Honestly I've never seen a my.cnf that short and if that's all then MySQL's query cache is not activated at all which is the default. You can find out more about the query cache as part of the official MySQL documentation here: http://dev.mysql.com/doc/refman/5.0/en/query-cache-configuration.html

But of course, it's not only about the query cache which is just one of many configrable features.

I can offer you to take a quick look at your machine if you like and in case you trust me ;) Just to make sure my assumption is right and to get your site going quickly.

P.S.:
Gonna go to bed now for some remaining 2-3 hours or so until I head off to the office. I'll be back online no later than 09:30 am CET

Alex

MySQL doesn't appear to be your problem

Jamie Holly's picture

Look at your output up there:

num Date Path Memory (MB) Milliseconds (Total) Anonymous? # Queries Query Milliseconds
83 04/07/2009 - 14:15 admin/user 0.00 75621 No 177 72
82 04/07/2009 - 14:15 admin/build 0.00 75818 No 177 77

Those pages are taking 75 seconds, but the queries are done in only 72-77 milliseconds (so you still got over 75 seconds to account for - being lost in PHP). Just for troubleshooting, I would start with disabling APC and see if that helps. If not, then paste a copy of your php.ini file so we can look through that. Also there are some suggestions in this thread, which was a very similar problem:

http://groups.drupal.org/node/20494


HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

In 75 seconds you could

Alexander Langer@drupal.org's picture

In 75 seconds you could easily run some climate prediction in PHP ;-)

Unfortunately the other thread as well raises more questions than it answers. But looking at top, memory consumption, stracing a test script, etc. isn't a bad hint either. I doubt that the machine is swapping as it would also hurt MySQL but maybe it's all cached..

May it be it's got something to do with disk access? You may (as hoster) be able to give a VPS a fixed amount of cpu power but you can't easily share harddisk performance - or maybe they're even mounted over net... Not sure though, cause I don't run virtual servers...

Alex

There is a very easy way to

Coornail's picture

There is a very easy way to find out why is a php site damn slow:

Install Xdebug on the server and edit the config (/etc/php/apache2-php5/ext/xdebug.ini at me):

xdebug.profiler_enable_trigger = on

Now reload the page with ?XDEBUG_PROFILE=1 attached to the end.
If you're finished you should get a cachegrind.out.blablabla file.
Open it with Kcachegrind (in linux, I don't know what windows guys using).
Now you should see which functions takes the most time, and you can find out what is slow then.

When you're finished disable xdebug, it should never run on a production site.

More xdebug profiler info
Kcachegrind

Yeah, xdebug is lovely. An

moshe weitzman's picture

Yeah, xdebug is lovely. An alternative to kcachegrind that works for all OS is webgrind.

Netzarim's picture

I am still trying to figure this out:

Executed 307 queries in 790.53 milliseconds. Queries taking longer than 5 ms and queries executed more than once, are highlighted. Page execution time was 11247.13 ms.

I am using Apache2, APC w/128mb, memcache, block_cache, etc... Still 12 seconds to load the home page at amazinggroups.com is never going to allow the site to be functional.

I tis like it is hanging some where... CPU utilization jumps to 100% for a long time, and sometimes 2-3 CPUs max out...

This is a Ubuntu 8.04.2 server running as a VM under ESXi with 4 2ghz procs asssigned and 4gb ram. The core system has 8Procs and 16gb ram.

What caching module are you

cfuller12's picture

What caching module are you using? This sounds like a problem I had with using shared = TRUE in cache router. If you have that set up, try changing the shared property to false and see if that helps.

It's all ball bearings these days...

Frontpage loads in 570 to

Alexander Langer@drupal.org's picture

Frontpage loads in 570 to 1200 ms. The rest comes rather slow, but from a lot of external places. Maybe it's just the transatlantic cable..

What's that woopra cookie ns ping thing supposed to do that it loads every other second?
Sometimes I get a "502 bad gateway", too. Firewall?

Have you tried the xdebug as mentioned above?

Alex

Woopra

Netzarim's picture

Woopra is a great tracking addon, I use it on several other sites with no issue, including a Drigg based site.
I've tried to route the external stuff through a CND - hence the nyud.net extension on some domains. I had the main site running though it too for css, js, and images using a mod to the core from 2bitsconsulting - but I disabled it for troubleshooting as it did not appear to help at this point.

Whenever I am doing anything on the site - anonymous or logged in, the CPU load goes really high and hangs there for a long time. Memory usage is not ramping up.

502 error happens when the page is not generated fast enough and the reverse proxy timesout. Right now I am seeing alot of js timeouts in the browser too, I have removed most of the jq stuff I had in the site already.

I am currrently defragging the mysql data drive (it is on another machine).

In a week this server will be moved to a new VM host. I am building a 4 x 8347HE Opteron with 20 SATA2. 10 of them built into a RAID 60, others for b/u and testing. 16GB initially, but can support upto 64GB.

I think this may help some as any I/O binding that may be occurring across the iSCSI feed for the current VMs should be gone.

Current setup has MySQL running on a IBM440 w/8x1.9gz Xeon 16GB under windows 2003 Ent, NFS server housing the Drupal site mapped into the web server which serves the site. This allows me to have multiple frontend webservers and one core of live code. The NFS and Web server1 are currently on another IBM440 w/8x2Ghz 16GB, running ESXi. Web server2, currently disabled is on VMWare Server - running on the same box as MySQL. Reverse proxy is also running as a VM on the same server as MySQL.

Load on the VMs are very low, except for Web Server1, which spikes CPU for extended periods of time on page loads.

All the backend stuff between hardware is connected by a managed Layer2 GB switch. Communication between VMs internally is by a Virtual GB switch.

In theory this config should allow for massive site growth. I have other VMs that host some shared hosting stuff without issue - Joomla based, Mediawiki, static HTML, etc...

I disabled the banner and adsense modules for a while, wondering if they were inducing the lag, but it did not seem to help.

I have xdebug installed and have webgrind in place. Still not seeing it the problem.

Within a short time of Apache restart APC is caching almost 100% of code.

Update - changing cache modules

Netzarim's picture

I am in process of removing the memcache module and authcache module, I am replacing with cacherouter module, which I will configure for memcache. I am still in process of defragging the database drive so things are slow anyway. Also, I am setting up an instance on the local /var/www/XXXXXX folder to see if there may be an issue over the NFS, although bandwidth across there has always been really low.

Proxy/MySQL & Modules

Swampcritter's picture

One thing you need to keep in mind when it is concerning Proxy (i.e. Squid) and MySQL... these are memory resident applications, thus they will eat up as much memory as you throw at them. MySQL is also known to cause constant disk I/O needs for temporary files.

  • Post a list of all the modules your site is currently
  • Post a copy of your my.cnf and a copy of the phpinfo() output

Proxy/MySQL & Modules

Netzarim's picture

MySQL hosted on windows 2003 Ent - my.ini:

port=3306
[mysql]
default-character-set=utf8
[mysqld]
max_allowed_packet = 64M
port=3306
basedir="C:/Program Files/MySQL/MySQL Server 5.0/"
datadir="V:/MySQL Datafiles/Data/"
default-character-set=utf8
default-storage-engine=myisam
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
max_connections=1200
query_cache_size=4096M
table_cache=1520
tmp_table_size=64M
thread_cache_size=16
myisam_max_sort_file_size=100G
myisam_max_extra_sort_file_size=100G
myisam_sort_buffer_size=64M
key_buffer_size=16M
read_buffer_size=2M
read_rnd_buffer_size=2M
sort_buffer_size=16M
innodb_data_home_dir="V:/MySQL Datafiles/"
innodb_additional_mem_pool_size=20M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=1M
innodb_buffer_pool_size=1024M
innodb_log_file_size=50M
innodb_thread_concurrency=32
query_cache_limit=256M
query_cache_min_res_unit=8k
log-warnings
expire_logs_days=14
memlock
low-priority-updates
interactive_timeout=300
wait_timeout=300
query_cache_type=1
query_cache_size=256000
big-tables
innodb_lock_wait_timeout=180
thread_concurrency=32
innodb_file_io_threads=8

apc.ini settings:
extension=apc.so
[apc]
apc.enabled = 1
apc.stat = 1
apc.shm_segments = 1
apc.shm_size = 128
apc.optimization = 1
apc.ttl = 7200
apc.user_ttl = 7200
apc.num_files_hint = 1000
;apc.include_once_override = 0
apc.mmap_file_mask = /tmp/apc.XXXXXX

phpinfo:
PHP Logo
PHP Version 5.2.4-2ubuntu5.5

System Linux xxxxxxx.amazinggroups.com 2.6.24-19-server #1 SMP Wed Aug 20 23:54:28 UTC 2008 i686
Build Date Feb 11 2009 19:49:37
Server API Apache 2.0 Handler
Virtual Directory Support disabled
Configuration File (php.ini) Path /etc/php5/apache2
Loaded Configuration File /usr/local/Zend/etc/php.ini
Scan this dir for additional .ini files /etc/php5/apache2/conf.d
additional .ini files parsed /etc/php5/apache2/conf.d/apc.ini, /etc/php5/apache2/conf.d/curl.ini, /etc/php5/apache2/conf.d/ffmpeg.ini, /etc/php5/apache2/conf.d/gd.ini, /etc/php5/apache2/conf.d/imagick.ini, /etc/php5/apache2/conf.d/memcache.ini, /etc/php5/apache2/conf.d/mhash.ini, /etc/php5/apache2/conf.d/ming.ini, /etc/php5/apache2/conf.d/mysql.ini, /etc/php5/apache2/conf.d/mysqli.ini, /etc/php5/apache2/conf.d/pdo.ini, /etc/php5/apache2/conf.d/pdo_mysql.ini, /etc/php5/apache2/conf.d/pdo_sqlite.ini, /etc/php5/apache2/conf.d/pspell.ini, /etc/php5/apache2/conf.d/recode.ini, /etc/php5/apache2/conf.d/snmp.ini, /etc/php5/apache2/conf.d/sqlite.ini, /etc/php5/apache2/conf.d/tidy.ini, /etc/php5/apache2/conf.d/xdebug.ini, /etc/php5/apache2/conf.d/xmlrpc.ini, /etc/php5/apache2/conf.d/xsl.ini
PHP API 20041225
PHP Extension 20060613
Zend Extension 220060519
Debug Build no
Thread Safety disabled
Zend Memory Manager enabled
IPv6 Support enabled
Registered PHP Streams zip, php, file, data, http, ftp, compress.bzip2, compress.zlib, https, ftps
Registered Stream Socket Transports tcp, udp, unix, udg, ssl, sslv3, sslv2, tls
Registered Stream Filters string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed, convert.iconv.*, bzip2.*, zlib.*

Suhosin logo This server is protected with the Suhosin Patch 0.9.6.2
Copyright (c) 2006 Hardened-PHP Project

Zend logo This program makes use of the Zend Scripting Language Engine:
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
with Xdebug v2.0.3, Copyright (c) 2002-2007, by Derick Rethans

PHP Credits
Configuration
PHP Core
Directive Local Value Master Value
allow_call_time_pass_reference On On
allow_url_fopen On On
allow_url_include Off Off
always_populate_raw_post_data Off Off
arg_separator.input & &
arg_separator.output & &
asp_tags Off Off
auto_append_file no value no value
auto_globals_jit On On
auto_prepend_file no value no value
browscap no value no value
default_charset no value no value
default_mimetype text/html text/html
define_syslog_variables Off Off
disable_classes no value no value
disable_functions no value no value
display_errors On On
display_startup_errors Off Off
doc_root no value no value
docref_ext no value no value
docref_root no value no value
enable_dl Off Off
error_append_string no value no value
error_log no value no value
error_prepend_string no value no value
error_reporting 6135 6135
expose_php On On
extension_dir /usr/lib/php5/20060613+lfs /usr/lib/php5/20060613+lfs
file_uploads On On
highlight.bg #FFFFFF #FFFFFF
highlight.comment #FF8000 #FF8000
highlight.default #0000BB #0000BB
highlight.html #000000 #000000
highlight.keyword #007700 #007700
highlight.string #DD0000 #DD0000
html_errors On On
ignore_repeated_errors Off Off
ignore_repeated_source Off Off
ignore_user_abort Off Off
implicit_flush Off Off
include_path .:/usr/share/php:/usr/share/pear .:/usr/share/php:/usr/share/pear
log_errors Off Off
log_errors_max_len 1024 1024
magic_quotes_gpc Off On
magic_quotes_runtime Off Off
magic_quotes_sybase Off Off
mail.force_extra_parameters no value no value
max_execution_time 600 600
max_input_nesting_level 64 64
max_input_time 300 300
memory_limit 1024M 1024M
open_basedir no value no value
output_buffering no value no value
output_handler no value no value
post_max_size 640M 640M
precision 12 12
realpath_cache_size 16K 16K
realpath_cache_ttl 120 120
register_argc_argv On On
register_globals Off Off
register_long_arrays On On
report_memleaks On On
report_zend_debug On On
safe_mode Off Off
safe_mode_exec_dir no value no value
safe_mode_gid Off Off
safe_mode_include_dir no value no value
sendmail_from no value no value
sendmail_path /usr/sbin/sendmail -t -i /usr/sbin/sendmail -t -i
serialize_precision 100 100
short_open_tag On On
SMTP localhost localhost
smtp_port 25 25
sql.safe_mode Off Off
suhosin.log.phpscript 0 0
suhosin.log.phpscript.is_safe Off Off
suhosin.log.phpscript.name no value no value
suhosin.log.sapi no value no value
suhosin.log.script no value no value
suhosin.log.script.name no value no value
suhosin.log.syslog no value no value
suhosin.log.syslog.facility no value no value
suhosin.log.syslog.priority no value no value
suhosin.log.use-x-forwarded-for Off Off
track_errors Off Off
unserialize_callback_func no value no value
upload_max_filesize 640M 640M
upload_tmp_dir no value no value
user_dir no value no value
variables_order EGPCS EGPCS
xmlrpc_error_number 0 0
xmlrpc_errors Off Off
y2k_compliance On On
zend.ze1_compatibility_mode Off Off

apache2handler
Apache Version Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.5 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.3 Perl/v5.8.8
Apache API Version 20051115
Server Administrator [no address given]
Hostname:Port www.amazinggroups.com:0
User/Group www-data(33)/33
Max Requests Per Child: 0 - Keep Alive: on - Max Per Connection: 100
Timeouts Connection: 300 - Keep-Alive: 15
Virtual Server Yes
Server Root /etc/apache2
Loaded Modules core mod_log_config mod_logio prefork http_core mod_so mod_alias mod_auth_basic mod_authn_file mod_authz_default mod_authz_groupfile mod_authz_host mod_authz_user mod_autoindex mod_cache mod_cgi mod_deflate mod_dir mod_env mod_expires mod_include mod_mem_cache mod_mime mod_negotiation mod_perl mod_php5 mod_rewrite mod_setenvif mod_ssl mod_status mod_suexec

Directive Local Value Master Value
engine 1 1
last_modified 0 0
xbithack 0 0

Apache Environment
Variable Value
HTTP_HOST www.amazinggroups.com
HTTP_USER_AGENT Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8
HTTP_ACCEPT text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
HTTP_ACCEPT_LANGUAGE en-us,en;q=0.5
HTTP_ACCEPT_ENCODING gzip,deflate
HTTP_ACCEPT_CHARSET ISO-8859-1,utf-8;q=0.7,*;q=0.7
HTTP_COOKIE SESSee4838b977517c974bea0dec2ab1736c=3fcd2662549b5a499a1900080f6fa17d; sessionCookie=WB5UN9Q3YTXJS50LX7K7MWZKMAD3686G; wooTracker=VCJLVPL4TQYPDFZNSGYFBRK2S0MUIMAW; __utma=120474904.362955234.1237921482.1239797828.1239800207.59; __utmz=120474904.1239116318.38.5.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=Your%20just%20rewards%20amazinggroups; DRUPAL_LOGGED_IN=Y; SESS02b6899af915a6cc35d9e4a97ebb9712=6e418007053abb400d4a3cff749cbce5; zemantaLatestSeenCookie=%7B%22id%22%3A0%2C%22first%22%3Anull%2C%22last%22%3Anull%2C%22times%22%3A0%2C%22read%22%3Afalse%7D; DRUPAL_UID=1; PLBSID=0.s1; has_js=1; __utmc=120474904
HTTP_X_FORWARDED_FOR 64.29.222.84
HTTP_X_FORWARDED_HOST www.amazinggroups.com
HTTP_X_FORWARDED_SERVER www.amazinggroups.com
HTTP_CONNECTION Keep-Alive
PATH /usr/local/bin:/usr/bin:/bin
SERVER_SIGNATURE

Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.5 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.3 Perl/v5.8.8 Server at www.amazinggroups.com Port 80

SERVER_SOFTWARE Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.5 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.3 Perl/v5.8.8
SERVER_NAME www.amazinggroups.com
SERVER_ADDR 10.3.0.52
SERVER_PORT 80
REMOTE_ADDR 10.3.0.53
DOCUMENT_ROOT /var/www/nfs/ag
SERVER_ADMIN [no address given]
SCRIPT_FILENAME /var/www/nfs/ag/phpinfo.php
REMOTE_PORT 22425
GATEWAY_INTERFACE CGI/1.1
SERVER_PROTOCOL HTTP/1.1
REQUEST_METHOD GET
QUERY_STRING no value
REQUEST_URI /phpinfo.php
SCRIPT_NAME /phpinfo.php

HTTP Headers Information
HTTP Request Headers
HTTP Request GET /phpinfo.php HTTP/1.1
Host www.amazinggroups.com
User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8
Accept text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Cookie SESSee4838b977517c974bea0dec2ab1736c=3fcd2662549b5a499a1900080f6fa17d; sessionCookie=WB5UN9Q3YTXJS50LX7K7MWZKMAD3686G; wooTracker=VCJLVPL4TQYPDFZNSGYFBRK2S0MUIMAW; __utma=120474904.362955234.1237921482.1239797828.1239800207.59; __utmz=120474904.1239116318.38.5.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=Your%20just%20rewards%20amazinggroups; DRUPAL_LOGGED_IN=Y; SESS02b6899af915a6cc35d9e4a97ebb9712=6e418007053abb400d4a3cff749cbce5; zemantaLatestSeenCookie=%7B%22id%22%3A0%2C%22first%22%3Anull%2C%22last%22%3Anull%2C%22times%22%3A0%2C%22read%22%3Afalse%7D; DRUPAL_UID=1; PLBSID=0.s1; has_js=1; __utmc=120474904
X-Forwarded-For 64.29.222.84
X-Forwarded-Host www.amazinggroups.com
X-Forwarded-Server www.amazinggroups.com
Connection Keep-Alive
HTTP Response Headers
X-Powered-By PHP/5.2.4-2ubuntu5.5
Cache-Control max-age=1
Expires Wed, 15 Apr 2009 13:46:45 GMT
Vary Accept-Encoding
Content-Encoding gzip

apc
APC Support enabled
Version 3.1.2
MMAP Support Enabled
MMAP File Mask /tmp/apc.eX4fS4
Locking type pthread mutex Locks
Revision $Revision: 3.196 $
Build Date Apr 1 2009 09:42:10

Directive Local Value Master Value
apc.cache_by_default On On
apc.canonicalize On On
apc.coredump_unmap Off Off
apc.enable_cli Off Off
apc.enabled On On
apc.file_md5 Off Off
apc.file_update_protection 2 2
apc.filters no value no value
apc.gc_ttl 3600 3600
apc.include_once_override Off Off
apc.max_file_size 1M 1M
apc.mmap_file_mask /tmp/apc.eX4fS4 /tmp/apc.eX4fS4
apc.num_files_hint 1000 1000
apc.preload_path no value no value
apc.report_autofilter Off Off
apc.rfc1867 Off Off
apc.rfc1867_freq 0 0
apc.rfc1867_name APC_UPLOAD_PROGRESS APC_UPLOAD_PROGRESS
apc.rfc1867_prefix upload_ upload_
apc.rfc1867_ttl 3600 3600
apc.shm_segments 1 1
apc.shm_size 128 128
apc.stat On On
apc.stat_ctime Off Off
apc.ttl 7200 7200
apc.user_entries_hint 4096 4096
apc.user_ttl 7200 7200
apc.write_lock On On

bcmath
BCMath support enabled

bz2
BZip2 Support Enabled
Stream Wrapper support compress.bz2://
Stream Filter support bzip2.decompress, bzip2.compress
BZip2 Version 1.0.4, 20-Dec-2006

calendar
Calendar support enabled

ctype
ctype functions enabled

curl
cURL support enabled
cURL Information libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.1

date
date/time support enabled
"Olson" Timezone Database Version 0.system
Timezone Database internal
Default timezone America/Chicago

Directive Local Value Master Value
date.default_latitude 31.7667 31.7667
date.default_longitude 35.2333 35.2333
date.sunrise_zenith 90.583333 90.583333
date.sunset_zenith 90.583333 90.583333
date.timezone no value no value

dba
DBA support enabled
Supported handlers cdb cdb_make db4 inifile flatfile

dom
DOM/XML enabled
DOM/XML API Version 20031129
libxml Version 2.6.31
HTML Support enabled
XPath Support enabled
XPointer Support enabled
Schema Support enabled
RelaxNG Support enabled

exif
EXIF Support enabled
EXIF Version 1.4 $Id: exif.c,v 1.173.2.5.2.20 2007/06/10 20:12:45 iliaa Exp $
Supported EXIF Version 0220
Supported filetypes JPEG,TIFF

ffmpeg
ffmpeg support (ffmpeg-php) enabled
ffmpeg-php version 0.5.1
libavcodec version Lavc1d.51.38.0
libavformat version Lavf1d.51.10.0

Directive Local Value Master Value
ffmpeg.allow_persistent 0 0

filter
Input Validation and Filtering enabled
Revision $Revision: 1.52.2.39 $

Directive Local Value Master Value
filter.default unsafe_raw unsafe_raw
filter.default_flags no value no value

ftp
FTP support enabled

gd
GD Support enabled
GD Version 2.0 or higher
FreeType Support enabled
FreeType Linkage with freetype
FreeType Version 2.3.5
T1Lib Support enabled
GIF Read Support enabled
GIF Create Support enabled
JPG Support enabled
PNG Support enabled
WBMP Support enabled

gettext
GetText Support enabled

hash
hash support enabled
Hashing Engines md2 md4 md5 sha1 sha256 sha384 sha512 ripemd128 ripemd160 ripemd256 ripemd320 whirlpool tiger128,3 tiger160,3 tiger192,3 tiger128,4 tiger160,4 tiger192,4 snefru gost adler32 crc32 crc32b haval128,3 haval160,3 haval192,3 haval224,3 haval256,3 haval128,4 haval160,4 haval192,4 haval224,4 haval256,4 haval128,5 haval160,5 haval192,5 haval224,5 haval256,5

iconv
iconv support enabled
iconv implementation glibc
iconv library version 2.7

Directive Local Value Master Value
iconv.input_encoding ISO-8859-1 ISO-8859-1
iconv.internal_encoding ISO-8859-1 ISO-8859-1
iconv.output_encoding ISO-8859-1 ISO-8859-1

imagick
imagick module enabled
imagick module version 2.0.1
imagick classes Imagick, ImagickDraw, ImagickPixel, ImagickPixelIterator
ImageMagick version ImageMagick 6.3.7 02/19/08 Q16 http://www.imagemagick.org
ImageMagick release date 02/19/08
ImageMagick Number of supported formats: 179
ImageMagick Supported formats A, AI, ART, ARW, AVI, AVS, B, BMP, BMP2, BMP3, C, CAPTION, CIN, CIP, CLIP, CMYK, CMYKA, CR2, CRW, CUR, CUT, DCM, DCR, DCX, DFONT, DJVU, DNG, DOT, DPS, DPX, EPDF, EPI, EPS, EPS2, EPS3, EPSF, EPSI, EPT, EPT2, EPT3, EXR, FAX, FITS, FRACTAL, FTS, G, G3, GIF, GIF87, GRADIENT, GRAY, HISTOGRAM, HTM, HTML, ICB, ICO, ICON, INFO, IPL, JNG, JP2, JPC, JPEG, JPG, JPX, K, K25, KDC, LABEL, M, M2V, MAP, MAT, MATTE, MIFF, MNG, MONO, MPC, MPEG, MPG, MRW, MSL, MSVG, MTV, MVG, NEF, NULL, O, ORF, OTB, OTF, PAL, PALM, PAM, PATTERN, PBM, PCD, PCDS, PCL, PCT, PCX, PDB, PDF, PEF, PFA, PFB, PFM, PGM, PGX, PICON, PICT, PIX, PJPEG, PLASMA, PNG, PNG24, PNG32, PNG8, PNM, PPM, PREVIEW, PS, PS2, PS3, PSD, PTIF, PWP, R, RAF, RAS, RGB, RGBA, RGBO, RLA, RLE, SCR, SCT, SFW, SGI, SHTML, SR2, SRF, STEGANO, SUN, SVG, SVGZ, TEXT, TGA, THUMBNAIL, TIFF, TIFF64, TILE, TIM, TTC, TTF, TXT, UIL, UYVY, VDA, VICAR, VID, VIFF, VST, WBMP, WMF, WMZ, WPG, X, X3F, XBM, XC, XCF, XPM, XV, XWD, Y, YCbCr, YCbCrA, YUV

imap
IMAP c-Client Version 2004
SSL Support enabled
Kerberos Support enabled

json
json support enabled
json version 1.2.1

libxml
libXML support active
libXML Version 2.6.31
libXML streams enabled

mbstring
Multibyte Support enabled
Multibyte string engine libmbfl
Multibyte (japanese) regex support enabled
Multibyte regex (oniguruma) version 4.4.4
Multibyte regex (oniguruma) backtrack check On

mbstring extension makes use of "streamable kanji code filter and converter", which is distributed under the GNU Lesser General Public License version 2.1.

Directive Local Value Master Value
mbstring.detect_order no value no value
mbstring.encoding_translation Off Off
mbstring.func_overload 0 0
mbstring.http_input pass pass
mbstring.http_output pass pass
mbstring.internal_encoding no value no value
mbstring.language neutral neutral
mbstring.strict_detection Off Off
mbstring.substitute_character no value no value

mcrypt
mcrypt support enabled
Version 2.5.7
Api No 20021217
Supported ciphers cast-128 gost rijndael-128 twofish arcfour cast-256 loki97 rijndael-192 saferplus wake blowfish-compat des rijndael-256 serpent xtea blowfish enigma rc2 tripledes
Supported modes cbc cfb ctr ecb ncfb nofb ofb stream

Directive Local Value Master Value
mcrypt.algorithms_dir no value no value
mcrypt.modes_dir no value no value

memcache
memcache support enabled
Active persistent connections 0
Version 2.2.4
Revision $Revision: 1.104 $

Directive Local Value Master Value
memcache.allow_failover 1 1
memcache.chunk_size 8192 8192
memcache.default_port 11211 11211
memcache.hash_function crc32 crc32
memcache.hash_strategy standard standard
memcache.max_failover_attempts 20 20

mhash
MHASH support Enabled
MHASH API Version 20060101

mime_magic
mime_magic support invalid magic file, disabled

Directive Local Value Master Value
mime_magic.debug Off Off
mime_magic.magicfile /usr/share/file/magic.mime /usr/share/file/magic.mime

ming
Ming SWF output library enabled
Version 0.3.0

mysql
MySQL Support enabled
Active Persistent Links 1
Active Links 1
Client API version 5.0.51a
MYSQL_MODULE_TYPE external
MYSQL_SOCKET /var/run/mysqld/mysqld.sock
MYSQL_INCLUDE -I/usr/include/mysql
MYSQL_LIBS -L/usr/lib -lmysqlclient

Directive Local Value Master Value
mysql.allow_persistent On On
mysql.connect_timeout 60 60
mysql.default_host no value no value
mysql.default_password no value no value
mysql.default_port no value no value
mysql.default_socket no value no value
mysql.default_user no value no value
mysql.max_links Unlimited Unlimited
mysql.max_persistent Unlimited Unlimited
mysql.trace_mode Off Off

mysqli
MysqlI Support enabled
Client API library version 5.0.51a
Client API header version 5.0.51a
MYSQLI_SOCKET /var/run/mysqld/mysqld.sock

Directive Local Value Master Value
mysqli.default_host no value no value
mysqli.default_port 3306 3306
mysqli.default_pw no value no value
mysqli.default_socket no value no value
mysqli.default_user no value no value
mysqli.max_links Unlimited Unlimited
mysqli.reconnect Off Off

openssl
OpenSSL support enabled
OpenSSL Version OpenSSL 0.9.8g 19 Oct 2007

pcre
PCRE (Perl Compatible Regular Expressions) Support enabled
PCRE Library Version 7.4 2007-09-21

Directive Local Value Master Value
pcre.backtrack_limit 100000 100000
pcre.recursion_limit 100000 100000

PDO
PDO support enabled
PDO drivers mysql, sqlite, sqlite2

pdo_mysql
PDO Driver for MySQL, client library version 5.0.51a

pdo_sqlite
PDO Driver for SQLite 3.x enabled
PECL Module version 1.0.1 $Id: pdo_sqlite.c,v 1.10.2.6.2.2 2007/03/23 14:30:00 wez Exp $
SQLite Library 3.4.2

posix
Revision $Revision: 1.70.2.3.2.16 $

pspell
PSpell Support enabled

recode
Recode Support enabled
Revision $Revision: 1.37.2.1.2.3 $

Reflection
Reflection enabled
Version $Id: php_reflection.c,v 1.164.2.33.2.45 2007/08/20 17:01:22 sebastian Exp $

session
Session Support enabled
Registered save handlers files user memcache sqlite
Registered serializer handlers php php_binary wddx

Directive Local Value Master Value
session.auto_start Off Off
session.bug_compat_42 On On
session.bug_compat_warn On On
session.cache_expire 180 180
session.cache_limiter nocache nocache
session.cookie_domain no value no value
session.cookie_httponly Off Off
session.cookie_lifetime 0 0
session.cookie_path / /
session.cookie_secure Off Off
session.entropy_file no value no value
session.entropy_length 0 0
session.gc_divisor 100 100
session.gc_maxlifetime 1440 1440
session.gc_probability 0 0
session.hash_bits_per_character 4 4
session.hash_function 0 0
session.name PHPSESSID PHPSESSID
session.referer_check no value no value
session.save_handler files files
session.save_path /var/lib/php5 /var/lib/php5
session.serialize_handler php php
session.use_cookies On On
session.use_only_cookies Off Off
session.use_trans_sid 0 0

shmop
shmop support enabled

SimpleXML
Simplexml support enabled
Revision $Revision: 1.151.2.22.2.35 $
Schema support enabled

snmp
NET-SNMP Support enabled
NET-SNMP Version 5.4.1

soap
Soap Client enabled
Soap Server enabled

Directive Local Value Master Value
soap.wsdl_cache 1 1
soap.wsdl_cache_dir /tmp /tmp
soap.wsdl_cache_enabled 1 1
soap.wsdl_cache_limit 5 5
soap.wsdl_cache_ttl 86400 86400

sockets
Sockets Support enabled

SPL
SPL support enabled
Interfaces Countable, OuterIterator, RecursiveIterator, SeekableIterator, SplObserver, SplSubject
Classes AppendIterator, ArrayIterator, ArrayObject, BadFunctionCallException, BadMethodCallException, CachingIterator, DirectoryIterator, DomainException, EmptyIterator, FilterIterator, InfiniteIterator, InvalidArgumentException, IteratorIterator, LengthException, LimitIterator, LogicException, NoRewindIterator, OutOfBoundsException, OutOfRangeException, OverflowException, ParentIterator, RangeException, RecursiveArrayIterator, RecursiveCachingIterator, RecursiveDirectoryIterator, RecursiveFilterIterator, RecursiveIteratorIterator, RecursiveRegexIterator, RegexIterator, RuntimeException, SimpleXMLIterator, SplFileInfo, SplFileObject, SplObjectStorage, SplTempFileObject, UnderflowException, UnexpectedValueException

SQLite
SQLite support enabled
PECL Module version 2.0-dev $Id: sqlite.c,v 1.166.2.13.2.9 2007/05/19 17:58:22 iliaa Exp $
SQLite Library 2.8.17
SQLite Encoding UTF-8

Directive Local Value Master Value
sqlite.assoc_case 0 0

standard
Regex Library Bundled library enabled
Dynamic Library Support enabled
Path to sendmail /usr/sbin/sendmail -t -i

Directive Local Value Master Value
assert.active 1 1
assert.bail 0 0
assert.callback no value no value
assert.quiet_eval 0 0
assert.warning 1 1
auto_detect_line_endings 0 0
default_socket_timeout 60 60
safe_mode_allowed_env_vars PHP_ PHP_
safe_mode_protected_env_vars LD_LIBRARY_PATH LD_LIBRARY_PATH
url_rewriter.tags a=href,area=href,frame=src,input=src,form=,fieldset= a=href,area=href,frame=src,input=src,form=,fieldset=
user_agent no value no value

sysvmsg
sysvmsg support enabled
Revision $Revision: 1.20.2.3.2.6 $

tidy
Tidy support enabled
libTidy Release 6 November 2007
Extension Version 2.0 ($Id: tidy.c,v 1.66.2.8.2.23 2007/05/04 17:11:05 nlopess Exp $)

Directive Local Value Master Value
tidy.clean_output no value no value
tidy.default_config no value no value

tokenizer
Tokenizer Support enabled

wddx
WDDX Support enabled
WDDX Session Serializer enabled

xdebug
xdebug support enabled
Version 2.0.3

Supported protocols Revision
DBGp - Common DeBuGger Protocol $Revision: 1.125.2.4 $
GDB - GNU Debugger protocol $Revision: 1.87 $
PHP3 - PHP 3 Debugger protocol $Revision: 1.22 $

Directive Local Value Master Value
xdebug.auto_trace Off Off
xdebug.collect_includes On On
xdebug.collect_params 0 0
xdebug.collect_return Off Off
xdebug.collect_vars Off Off
xdebug.default_enable On On
xdebug.dump.COOKIE no value no value
xdebug.dump.ENV no value no value
xdebug.dump.FILES no value no value
xdebug.dump.GET no value no value
xdebug.dump.POST no value no value
xdebug.dump.REQUEST no value no value
xdebug.dump.SERVER no value no value
xdebug.dump.SESSION no value no value
xdebug.dump_globals On On
xdebug.dump_once On On
xdebug.dump_undefined Off Off
xdebug.extended_info On On
xdebug.idekey no value no value
xdebug.manual_url http://www.php.net http://www.php.net
xdebug.max_nesting_level 100 100
xdebug.profiler_aggregate Off Off
xdebug.profiler_append Off Off
xdebug.profiler_enable On On
xdebug.profiler_enable_trigger On On
xdebug.profiler_output_dir /tmp /tmp
xdebug.profiler_output_name cachegrind.out.%p cachegrind.out.%p
xdebug.remote_autostart Off Off
xdebug.remote_enable Off Off
xdebug.remote_handler dbgp dbgp
xdebug.remote_host localhost localhost
xdebug.remote_log no value no value
xdebug.remote_mode req req
xdebug.remote_port 9000 9000
xdebug.show_exception_trace Off Off
xdebug.show_local_vars Off Off
xdebug.show_mem_delta Off Off
xdebug.trace_format 0 0
xdebug.trace_options 0 0
xdebug.trace_output_dir /tmp /tmp
xdebug.trace_output_name trace.%c trace.%c
xdebug.var_display_max_children 128 128
xdebug.var_display_max_data 512 512
xdebug.var_display_max_depth 3 3

xml
XML Support active
XML Namespace Support active
libxml2 Version 2.6.31

xmlreader
XMLReader enabled

xmlrpc
core library version xmlrpc-epi v. 0.51
php extension version 0.51
author Dan Libby
homepage http://xmlrpc-epi.sourceforge.net
open sourced by Epinions.com

xmlwriter
XMLWriter enabled

xsl
XSL enabled
libxslt Version 1.1.22
libxslt compiled against libxml Version 2.6.31
EXSLT enabled
libexslt Version 1.1.22

zip
Zip enabled
Extension Version $Id: php_zip.c,v 1.1.2.38 2007/08/06 22:02:32 bjori Exp $
Zip version 2.0.0
Libzip version 0.7.1

zlib
ZLib Support enabled
Stream Wrapper support compress.zlib://
Stream Filter support zlib.inflate, zlib.deflate
Compiled Version 1.2.1.1
Linked Version 1.2.3.3

Directive Local Value Master Value
zlib.output_compression Off Off
zlib.output_compression_level -1 -1
zlib.output_handler no value no value

Additional Modules
Module Name
sysvsem
sysvshm

Environment
Variable Value
APACHE_PID_FILE /var/run/apache2.pid
APACHE_RUN_USER www-data
PATH /usr/local/bin:/usr/bin:/bin
PWD /
APACHE_RUN_GROUP www-data
LANG C
SHLVL 1
_ /usr/sbin/apache2

PHP Variables
Variable Value
_REQUEST["SESSee4838b977517c974bea0dec2ab1736c"] 3fcd2662549b5a499a1900080f6fa17d
_REQUEST["sessionCookie"] WB5UN9Q3YTXJS50LX7K7MWZKMAD3686G
_REQUEST["wooTracker"] VCJLVPL4TQYPDFZNSGYFBRK2S0MUIMAW
_REQUEST["__utma"] 120474904.362955234.1237921482.1239797828.1239800207.59
_REQUEST["__utmz"] 120474904.1239116318.38.5.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=Your just rewards amazinggroups
_REQUEST["DRUPAL_LOGGED_IN"] Y
_REQUEST["SESS02b6899af915a6cc35d9e4a97ebb9712"] 6e418007053abb400d4a3cff749cbce5
_REQUEST["zemantaLatestSeenCookie"] {"id":0,"first":null,"last":null,"times":0,"read":false}
_REQUEST["DRUPAL_UID"] 1
_REQUEST["PLBSID"] 0.s1
_REQUEST["has_js"] 1
_REQUEST["__utmc"] 120474904
_COOKIE["SESSee4838b977517c974bea0dec2ab1736c"] 3fcd2662549b5a499a1900080f6fa17d
_COOKIE["sessionCookie"] WB5UN9Q3YTXJS50LX7K7MWZKMAD3686G
_COOKIE["wooTracker"] VCJLVPL4TQYPDFZNSGYFBRK2S0MUIMAW
_COOKIE["__utma"] 120474904.362955234.1237921482.1239797828.1239800207.59
_COOKIE["__utmz"] 120474904.1239116318.38.5.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=Your just rewards amazinggroups
_COOKIE["DRUPAL_LOGGED_IN"] Y
_COOKIE["SESS02b6899af915a6cc35d9e4a97ebb9712"] 6e418007053abb400d4a3cff749cbce5
_COOKIE["zemantaLatestSeenCookie"] {"id":0,"first":null,"last":null,"times":0,"read":false}
_COOKIE["DRUPAL_UID"] 1
_COOKIE["PLBSID"] 0.s1
_COOKIE["has_js"] 1
_COOKIE["__utmc"] 120474904
_SERVER["HTTP_HOST"] www.amazinggroups.com
_SERVER["HTTP_USER_AGENT"] Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8
_SERVER["HTTP_ACCEPT"] text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
_SERVER["HTTP_ACCEPT_LANGUAGE"] en-us,en;q=0.5
_SERVER["HTTP_ACCEPT_ENCODING"] gzip,deflate
_SERVER["HTTP_ACCEPT_CHARSET"] ISO-8859-1,utf-8;q=0.7,*;q=0.7
_SERVER["HTTP_COOKIE"] SESSee4838b977517c974bea0dec2ab1736c=3fcd2662549b5a499a1900080f6fa17d; sessionCookie=WB5UN9Q3YTXJS50LX7K7MWZKMAD3686G; wooTracker=VCJLVPL4TQYPDFZNSGYFBRK2S0MUIMAW; __utma=120474904.362955234.1237921482.1239797828.1239800207.59; __utmz=120474904.1239116318.38.5.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=Your%20just%20rewards%20amazinggroups; DRUPAL_LOGGED_IN=Y; SESS02b6899af915a6cc35d9e4a97ebb9712=6e418007053abb400d4a3cff749cbce5; zemantaLatestSeenCookie=%7B%22id%22%3A0%2C%22first%22%3Anull%2C%22last%22%3Anull%2C%22times%22%3A0%2C%22read%22%3Afalse%7D; DRUPAL_UID=1; PLBSID=0.s1; has_js=1; __utmc=120474904
_SERVER["HTTP_X_FORWARDED_FOR"] 64.29.222.84
_SERVER["HTTP_X_FORWARDED_HOST"] www.amazinggroups.com
_SERVER["HTTP_X_FORWARDED_SERVER"] www.amazinggroups.com
_SERVER["HTTP_CONNECTION"] Keep-Alive
_SERVER["PATH"] /usr/local/bin:/usr/bin:/bin
_SERVER["SERVER_SIGNATURE"]

Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.5 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.3 Perl/v5.8.8 Server at www.amazinggroups.com Port 80

_SERVER["SERVER_SOFTWARE"] Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.5 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.3 Perl/v5.8.8
_SERVER["SERVER_NAME"] www.amazinggroups.com
_SERVER["SERVER_ADDR"] 10.3.0.52
_SERVER["SERVER_PORT"] 80
_SERVER["REMOTE_ADDR"] 10.3.0.53
_SERVER["DOCUMENT_ROOT"] /var/www/nfs/ag
_SERVER["SERVER_ADMIN"] [no address given]
_SERVER["SCRIPT_FILENAME"] /var/www/nfs/ag/phpinfo.php
_SERVER["REMOTE_PORT"] 22425
_SERVER["GATEWAY_INTERFACE"] CGI/1.1
_SERVER["SERVER_PROTOCOL"] HTTP/1.1
_SERVER["REQUEST_METHOD"] GET
_SERVER["QUERY_STRING"] no value
_SERVER["REQUEST_URI"] /phpinfo.php
_SERVER["SCRIPT_NAME"] /phpinfo.php
_SERVER["PHP_SELF"] /phpinfo.php
_SERVER["REQUEST_TIME"] 1239803204
_SERVER["argv"]

Array
(
)

SERVER["argc"] 0
_ENV["APACHE_PID_FILE"] /var/run/apache2.pid
_ENV["APACHE_RUN_USER"] www-data
_ENV["PATH"] /usr/local/bin:/usr/bin:/bin
_ENV["PWD"] /
_ENV["APACHE_RUN_GROUP"] www-data
_ENV["LANG"] C
_ENV["SHLVL"] 1
_ENV["
"] /usr/sbin/apache2

Drupal & Modules:
Drupal 6.10

Modules
Account reminder 6.x-1.1
ACL 6.x-1.0-beta4
AdSense 6.x-1.0
Advanced help 6.x-1.2
Affinity 6.x-1.2
Aggregation 6.x-1.8
Amazon 6.x-1.0-beta5
Avatar Selection 6.x-1.5
Backup and Migrate 6.x-1.2
Block Cache Alter 6.x-1.2
Blog Add-ons 6.x-1.1
CAPTCHA 6.x-1.0-rc2
CCK Fieldgroup Tabs 6.x-1.x-dev (2008-Oct-01)
Charts 6.x-1.0-alpha5
Checkbox Validate 6.x-1.1
Comment Notify 6.x-1.2
Comment Page 6.x-1.1
Content Construction Kit (CCK) 6.x-2.2
Content Access 6.x-1.1
Javascript Countdown Timer 6.x-2.19
Creative Commons Lite 6.x-1.2
CSS Gzip 6.x-1.0
Date 6.x-2.1
DB Maintenance 6.x-1.1
Devel 6.x-1.16
DrupalIt 6.x-1.4-beta2
Email Change Confirmation 6.x-1.2
Error Codes 6.x-1.3
Frequently Asked Questions 6.x-1.8
FAQ_Ask 6.x-1.2
Favorite Nodes 6.x-1.x-dev (2009-Apr-11)
FCKeditor - WYSIWYG HTML editor 6.x-1.3
FeedBurner 6.x-1.0-beta3
Fivestar 6.x-1.13
Flash Node 6.x-3.1
FlashVideo 6.x-1.4
SWF Tools 6.x-2.4
Front Page 6.x-1.2
Global Redirect 6.x-1.2
Google Analytics 6.x-2.2
Helpers 6.x-1.1
Hovertips and Clicktips 6.x-1.x-dev (2008-May-15)
Image 6.x-1.0-alpha4
ImageAPI 6.x-1.5
ImageCache 6.x-2.0-beta8
IMCE 6.x-1.2
IMCE Crop 6.x-1.x-dev (2009-Feb-10)
IMCE Mkdir 6.x-1.x-dev (2009-Feb-10)
Imce CCK Image 6.x-1.0-beta2
Image Assist 6.x-1.0
Incoming 6.x-1.x-dev (2008-Aug-14)
Javascript Aggregator 6.x-1.1
Legal 6.x-2.2-beta4
Lightbox2 6.x-1.9
Mail API 6.x-2.x-dev (2008-Sep-25)
Mailhandler 6.x-1.5
Memcache API and Integration 6.x-1.2
Mugshot 6.x-1.x-dev (2008-Aug-14)
Node Convert 6.x-1.2
Open Flash Chart API 6.x-2.10
Openads 6.x-1.0
Password policy 6.x-1.0-alpha1
Pathauto 6.x-1.1
Path Cache 6.x-1.2
Persistent Login 6.x-1.4-beta4
PHPIDS 6.x-1.7
Please register 6.x-1.3
Printer, e-mail and PDF versions 6.x-1.4
Private 6.x-1.0
Quick Tabs 6.x-1.0-beta1
Quotes 6.x-1.30
reCAPTCHA 6.x-1.1
Recipe 6.x-1.0-beta1
Register with Picture 6.x-1.0
Scheduler 6.x-1.3
Schema 6.x-1.3
Scripture Filter 6.x-1.0
Search 404 6.x-1.4
Service links 6.x-1.0
Session Restore 6.x-1.1-beta
Signatures for Forums 6.x-1.0
Site Documentation 6.x-1.2
Smileys 6.x-1.0-alpha5
Subscriptions 6.x-1.0-beta5
System information 6.x-1.0-beta4
jQuery UI Tabs 6.x-1.0
Tagadelic 6.x-1.2
Taxonomy Manager 6.x-1.0-beta2
Token 6.x-1.11
Tool Tips 6.x-1.1
userplus 6.x-2.2
Video Filter 6.x-2.5
Views 6.x-2.5
Voting API 6.x-2.0-rc2
Image watermark 6.x-1.x-dev (2008-Jun-23)
Woopra 6.x-1.1
Wordfilter 6.x-1.x-dev (2009-Apr-14)
Workspace 6.x-1.3
XML Sitemap 6.x-0.x-dev (2008-Nov-18)
Zemanta 6.x-1.1

I had some other modules in play, but had already removed them. I was unable to get cacherouter to work and went back to memcache as that seemed to perform the best to this point. Despite the php.ini file location, Zend Optimizer is disabled as it does not work with APC.

I know we came to the point

Alexander Langer@drupal.org's picture

I know we came to the point that DB performance seems not to be the issue here, but nevertheless there's two things that standout:

Your query_cache_size=4096M is way too high. Check how much of the query cache is used and I'm pretty sure you'll find out it's a three digit number at worst. MySQL's query cache caches already checked, optimized and compiled querys, but propably will never have 4G of pure query code from different queries. Also a too bloated cache may prove inefficient as it consumes too much cpu power and maybe memory bandwith just to administer it.

Second thing is your key_buffer_size=16M is very low, based on my assumption that all your MyISAM tables' index sizes are way bigger than 16M. You have InnoDB configured, do you already use it for at least some of your tables? If not, save the memory and let the system use it elsewhere when needed. If your MyISAM index space is bigger than your key buffer size MySQL will have to reload index data from disk which is slooow. If you can afford it, use something close to your MyISAM index space in size.

A good resource on MySQL optimizations, besides some stuff on 2bits.com is the MySQLPerformanceBlog.com .

Alex

MySQL modified.

Netzarim's picture

I just made some adjustments based on recommended settings from 2bits.com.

Yes, some tables, not all are using innodb. I have made table type adjustments based on how the tables are accessed. ie. if the table is subject to a lot of updates, they get to be innodb so the app isnt waiting on table locks during peak load.

using tempfs to alleviate MySQL temporary file IO

carteriii's picture

On the assumption that you have sufficient memory to handle any temporary files created by MySQL, you might consider my solution which was to create/mount a tempfs directory and then tell MySQL to use that location for its temporary files. Thus MySQL thinks it is still creating file-based temporary files, but it's really RAM. I have only modest knowledge of the inner workings of tempfs and MySQL so I don't know whether this is a good, great, or horrible solution, but it appeared to work well, handle all the corner cases (like MySQL always using temporary files for tables that include long columns) and was pretty easy.

If anyone can offer a more detailed/thorough analysis of this approach and the tradeoffs, I'd love to hear it.

It's okay to use tmtpfs for

Alexander Langer@drupal.org's picture

It's okay to use tmtpfs for this as long as you have enough RAM you don't need anywhere else and you don't run some custom scripts creating HUGE tmp tables that may not fit into your tmpfs drive (which takes only as much RAM as it needs until reaching it's configured limit, btw).

Problem is however, that MySQL 5 has a bug in not-so-recent versions where it ignores the temp_dir setting and instead uses it's default path ( like /var/lib/mysql ). So make sure you're up to date if you want to use this.

Alex

markus_petrux's picture

If not already on latest 5.1, then I would upgrade as there were several improvements and bug fixes.

Memcache API

Swampcritter's picture

Since you have memcache API enabled, are you running with one memcache server or a cluster? Also how much is memory is allocated to each memcache daemon/session?

Another module you might need to consider is called Boost (http://drupal.org/project/boost), which works rather well for static content for anonymous traffic.

memcache

Netzarim's picture

I have one instance of memcache running on the webserver locally. It is running with 256M.

I had some problems with boost in the past and disabled it. I read that the bug was fixed, but the site is really designed to be interactive with subscribers - or at least that is the intention.

Still, I should not be seeing 10 second+ page loads with only one user hitting the site:
Executed 562 queries in 2089.45 milliseconds. Queries taking longer than 5 ms and queries executed more than once, are highlighted. Page execution time was 17081.16 ms.

I was wondering if there is a problem in the apache2/php5 configuration that is causing this, but apc.php and phpinfo.php load right up.

I do see some lag at getting banners, I wonder if there is a way to have the adsense and openx modules load those last or at least let the page continue loading while their fetched. That might account for a couple of seconds. I am removing the cdn from the openx for now...

How ever, I have fully disabled both those modules in the past and really saw no difference.

Can you make a recent

Alexander Langer@drupal.org's picture

Can you make a recent cachegrind file available for download so we can take a look at the xdebug data?

Alex

Webgrind access

Netzarim's picture

I have place webgrind off the root for the moment, http://www.amazinggroups.com/webgrind

I get the bad gateway when

Alexander Langer@drupal.org's picture

I get the bad gateway when trying to (re)load data in webgrind. :-/

Alex

Hang..

Netzarim's picture

I saw for all 4 procs at 100% for a bit and ssh disconnected me form the host. It has cleared and is accessible again. I am looking at webgrind too, from a remote location. It doesnt take many users hitting the site right now to over load the system, which is ridiculous. I almost to the point of building a new VM and starting over on the webserver side.

I had to reboot... It locked...

I can only imagine how

Alexander Langer@drupal.org's picture

I can only imagine how frustrating this must be.

It seems as if the VM or at least processing PHP is just .. eh.. slow! Loading the files from NFS doesn't seem to be an issue (the scripts easily get cached by the operating system anyway) but even if without any bytecode cache your vm on that iron should perform an order of magnitude better than it does.

Maybe you could use the here mentioned bench.php to benchmark your system in comparison to your dev system and whatever alese you have at hand: http://sebastian-bergmann.de/archives/634-PHP-GCC-ICC-Benchmark.html
Bench.php is not bound to storage I/O or DB performance. Running it from a browser may at least narrow things down a bit. Maybe compare it to the performance gained without APC. Maybe it's your PHP module, maybe it's some weird untraceable VM stuff (lucky me, I don't use production server VMs)..

Alex

I ran bench...

Netzarim's picture

simple 1.407 simplecall 60.244 simpleucall 65.185 simpleudcall 63.914 mandel 5.247 mandel2 4.157
( ! ) Fatal error: Maximum function nesting level of '100' reached, aborting! in /mnt/nfs/home/ag/bench.php on line 98
Call Stack

Time Memory Function Location

1 0.2653 57732 {main}( ) ../bench.php:0
2 203.3195 102340 ackermann( ) ../bench.php:394
3 203.3195 102340 Ack( ) ../bench.php:103
4 203.3195 102340 Ack( ) ../bench.php:99
5 203.3196 102340 Ack( ) ../bench.php:99
6 203.3196 102396 Ack( ) ../bench.php:99
7 203.4663 126500 Ack( ) ../bench.php:99
......
94 203.5748 138504 Ack( ) ../bench.php:99
95 203.5749 138504 Ack( ) ../bench.php:99
96 203.5749 138504 Ack( ) ../bench.php:99
97 203.5749 138504 Ack( ) ../bench.php:99
98 203.5749 138504 Ack( ) ../bench.php:99
99 203.5749 138504 Ack( ) ../bench.php:99

Here's what I get on a

Alexander Langer@drupal.org's picture

Here's what I get on a rather old AMD64 single core dedicated server (Debian Etch, PHP 5.2):

simple 0.364 simplecall 0.423 simpleucall 0.614 simpleudcall 0.771 mandel 1.200 mandel2 1.554 ackermann(7) 0.791 ary(50000) 0.077 ary2(50000) 0.062 ary3(2000) 0.616 fibo(30) 2.234 hash1(50000) 0.129 hash2(500) 0.108 heapsort(20000) 0.495 matrix(20) 0.392 nestedloop(12) 0.641 sieve(30) 0.319 strcat(200000) 0.081 ------------------------ Total 10.871

I did several runs in a row and they all took about 10.5 to 11 seconds without considerable other stuff going on on the machine. The test itself obviously does not benefit from multiple cores and bytecode caching (as parsing time isn't measured). It's all about the pure speed of execution and although your test broke up (maybe some suhosin limits??) it says your machine is darn slow.

Question:
Is it only PHP that's slow, or does it go deeper intom the system? You may check that by benching with other tools that do not use PHP.

Alex

Disable xdebug or set

Coornail's picture

Disable xdebug or set xdebug.max_nesting_level higher with ini_set().

xdebug off results

Netzarim's picture

simple 1.452 simplecall 10.000 simpleucall 11.662 simpleudcall 12.253 mandel 4.784 mandel2 3.848

I think it is PHP as everything else runs fast... Oddly, acp.php opens right up.... In fact, so did the management page for eaccelerator when I used it and webgrind. My managmeent tools open right up, but they have their own apache instance.

I am prone to keeping patches current, I wonder if one of the upgrades created an incompatability with one of the modules. I run apt-get update, apt-get upgrade probably once every two weeks or more.

I have all the parts ordered for the new ESXi host. I will begin building it, then build a new host VM on the new platform, and transfer the site to it. The new system is using 8347HE opterons, so 64bit and using local drives in a raid 60 should yield some kind of improvement over the current situation.

High performance

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: