Posted by superfedya on October 10, 2011 at 11:59am
Hi,
I just look at my mysql statistic and I see that there is some problem with table locking.
SHOW STATUS LIKE 'Table%'
Table_locks_immediate 3412356
Table_locks_waited 2798
But how can I known which table exactly has this problem? I already known that session and watchdog tables must be in InnoDB, maybe something else too? Maybe voting api or cache tables?
All attached my tables list.
Thank you for any advise!
| Attachment | Size |
|---|---|
| 2011-10-10 07-52-51.png | 11.94 KB |
| 2011-10-10 07-53-21.png | 10.91 KB |
| 2011-10-10 07-53-50.png | 8.51 KB |
| 2011-10-10 07-54-27.png | 10.99 KB |
| 2011-10-10 07-55-01.png | 2.01 KB |
| 2011-10-10 07-55-26.png | 11.65 KB |
| 2011-10-10 07-55-44.png | 3.8 KB |

Comments
Make all tables InnoDB (the
Make all tables InnoDB (the whole D6/Pressflow Database!)!
When I moved to Pressflow, I dumped my D6 MySQL database to a SQL file,
then edited all entries of:
ENGINE=MyISAM DEFAULT CHARSET=utf8;to
ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;Look for any
ENGINE=InnoDB AUTO_INCREMENT=xxtoo.See: http://www.dynamiteheads.com/blog/jakub-suchy/migrating-your-site-pressf...
and
http://highervisibilitywebsites.com/convert-your-mysql-database-myisam-i...
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
Make all tables InnoDB But
But InnoDB is slower. I don't wanna hurt the performance.
But InnoDB is slower?
Where did you get that 'interesting' information? (I am 'somewhat stunned, I have done months of research and benchmarks on this MySQL level).
See: http://stackoverflow.com/questions/1428678/is-innodb-sorting-really-that...
and
http://tag1consulting.com/InnoDB_Performance_Tuning
Base line is this; a well tuned InnoDB can offer much more than a the same level of MyISAM.
If you study and use the correct InnoDB methods you will get good results, I did!
If you have a low-end server, I would stay with MyISAM, it works well enough for basic needs.
If you have more than 2Gb on a Linux Server (Ubuntu/Debian in your case), then plan well the move to InnoDB! It is worth it.
When you are ready to 'hit' the InnoDB performance factor, then read this:
http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimi...
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
There are cases where InnoDB
There are cases where InnoDB is slower than MyISAM (ie: doing a count(*) without a where clause). Other examples are alter tables, especially on large datasets. While this isn't an everyday operation, it can cause a noticeable performance degradation on some module updates.
Having said that, doing a mixture of tables really is fine. Tables that see a lot of writes should become InnoDB to stop the locks. I wouldn't base Watchdog off of that. If Watchdog is causing a problem then I would look at changing storage on that to some other mechanism (log files, couchdb, etc.). On one site I manage I just hijacked the dblog module and had it not write certain things, like new comments being posted.
One site I run does a mixture of the two storage engines. We once saw over 65,000 page views, 13,000 authenticated users and over 700 comments posted in 1 hour. That was all on a single quad-core server (Xeon 2.5g) with 4gb ram. During that time the server load peaked at 1.65, so things ran great. The only other extra we had was the static content coming off a CDN.
One thing about InnoDB - you need to stay up on it. It does require attention and tuning, much more so than MyISAM.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
A problem with mixing the
A problem with mixing the table types is that you need to maintain two buffers in RAM. Essentially you need to allocate decent portions of RAM to MyISAM and InnoDB. Finding that line of what is good for both of them is much harder than giving InnoDB 70-80% of RAM on a dedicated MySQL server or 30% on a server that also runs PHP for instance.
Especially if your db fits in RAM in which case InnoDB with a large buffer pool will perform nearly all the queries that don't create temporary tables in RAM. That is very fast.
I don't have brilliant evidence but when using both table engine pools I have had joins that go across the boundary perform worse also. All of the sites with decent levels of traffic I work on are full InnoDB for the Drupal schema. This is many sites/environments that do millions of PV per day.
Full Fat Things ( http://fullfatthings.com ), my Drupal consultancy that makes sites fast.
Thanks!dbtuner:InnoDBIs
Thanks!
dbtuner:
InnoDBIs InnoDB enabled?: YES
% innoDB log size: 25
InnoDB log file size is not an appropriate size, in relation to the InnoDB buffer pool. Consider changing either\ninnodb_log_file_size or innodb_buffer_pool_size
(innodb_log_file_size / innodb_buffer_pool_size * 100 >=0)
(268435456 / 1073741824 * 100 25>=0)
My config is wrong? Did I forgot something? Any advise? (quad core + 8gb ram):
innodb_buffer_pool_size = 1024Minnodb_log_file_size = 256M
innodb_flush_log_at_trx_commit = 0
innodb_log_buffer_size = 4M
innodb_flush_method=O_DIRECT
innodb_thread_concurrency = 8
innodb_file_per_table = 1
Thanks!
Check this article!
http://highervisibilitywebsites.com/convert-your-mysql-database-myisam-i...
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
It is true? There are many
It is true?
There are many systems, Drupal for example, that are very much optimized for that particular engine. This is not to say that they perform poorly on InnoDB, but they are not optimized for it. For example, while many of Drupal's core queries are well indexed and use the primary key (thus benefiting from InnoDB's primary key clustering), some could be improved. The node table has a primary key on (nid,vid). Having this index is a good idea, but it is a two-integer index and there are eleven secondary indexes based on it. This doesn't mean much when you use MyISAM, but under InnoDB it means each of those secondary indexes has two-integer sized leaves identifying the primary key.You're welcome to do some
You're welcome to do some benchmarks and open some issues on d.o. But I think you'll find that there is a net positive to having the compound primary key - that the queries that take advantage of it are by far the most plentiful.
In my experience MyISAM is great if you just want to set it up for a low traffic site and forget it. But if your site is going to be encountering any issues with performance or scalability then InnoDB is most definitely the better choice. Overall better performance for Drupal, and far more flexibility in how to address your performance issues.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
You are referring to this
You are referring to this article: http://stackoverflow.com/questions/1082194/when-should-you-choose-to-use...
It just means allocate more memory for InnoDB.. (in the right places)!
Some of these 'node table -multiple index' issues have been addressed in Pressflow 6.22.
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
I transferred a 700+ mb table
I transferred a 700+ mb table to innodb and now, when I run http://drupal.org/project/db_maintenance my CPU loading raise to 65% and I got 504 timeout...
504 Error
If you are using Nginx Proxy to Apache
the HTTP 504 TIMEOUT is likely coming from Nginx acting a front-end Proxy to Apache.
In this case, the 504 is caused by a proxy timeout, or Nginx having to wait too long for the expected Apache PHP response.
** Basically this an issue with whatever serves your PHP (Nginx or Apache).
In this Nginx proxy to Apache, the Apache server is waiting for MySQL process to finish fetching the data, but it’s not finished. So Nginx waits for Apache, and Apache is 'hung' waiting for MySQL. This process will keep loading while a new connection arrives, in the end it is time deadlock = 504 !
If mysql is the problem, you can check it with this command (Linux):
watch "mysqladmin -u username -ppassword processlist"Note that username should be your primary MySQL user [root], and password should be that user's
password. There is no space between the -p in the command and the password.
If you're using fastcgi:
Proxy timeouts are well, for proxies, not for FastCGI...
For FastCGI there's fastcgi_read_timeout which is affecting the fastcgi process response timeout.
eg:
fastcgi_send_timeout 1800;fastcgi_read_timeout 1800;
fastcgi_connect_timeout 1800;
Comment: You have something terribly wrong if you need to raise this limit.
Try this: Assuming your using PHP-FPM:
Setup The php-fpm Slow Log
Once your timeout settings are in place the problem is effectively isolated to some runaway code that only stops when it hits one of the timeouts.
The next step is identifying where the runaway code is. That’s were the php-fpm slow log may help trace the problem.
By configuring the request_slowlog_timeout and setting up a slowlog file, php-fpm will dump (write) a php backtrace into to the slow log file whenever a script hits the normal timeout value.
set the request_slowlog_timeout to 25 seconds in your Apache / Nginx .conf (which ever serves the PHP)
request_slowlog_timeout = 25sset a location for the slowlog file in www.conf
slowlog = /var/log/php5-fpm.slow.log
Now when you restart everything and hit the page causing the 504 error you’ll get a helpful backtrace in your slow log file. The backtrace will show you the specific line of PHP code that was being executed when it hit the typical 25 second limit.
Check if your php is 'sigfaulting'
If you are using php 5.3, increase the backlog.
If you are using php 5.2, backport the patch to increase the backlog size from 128.
Also, use a unix socket rather than a TCP socket. unix:/tmp/php5-cgi.sock (or the relevant path)
CPU loading raise to 65%
This is likely related to PHP server problems [errors]; and to me, it seems very related to the Proxy 504 timeout.
At this point I think PHP is hitting a 'brick wall' = 'Deadlock'
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
Thanks, ill upgrade my mysql
Thanks, ill upgrade my mysql to 5.5 with some patch, it's 3 times faster for innodb.
Ill verify php too.
Is OK for 8gb server?
innodb_buffer_pool_size = 4096M
innodb_log_file_size = 2048M
Is OK for 8gb
1) On a dedicated MySQL Server innodb_buffer_pool_size can be as high as 90% of Server memory.
On a Server shared with PHP, Nginx, Apache, etc; 30% - 50% of server memory is better.
So in your case 4Gb / 8Gb for innodb_buffer_pool_size is good.
2) The innodb_log_file_size for InnoDB is probably 'a bit large'; mine is 384Mb (of 8Gb Server Total)
Another important InnoDB 'my.cnf' setting is:
innodb_buffer_pool_instances = 8Where '8' represents the number of CPU cores x 2
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
On a dedicated MySQL Server
It's only useful to make it as large as your active data set. Assuming that there is only one site on this server, and that the size of your database is likely < 1GB, there will be no benefit of having a buffer pool larger than about 1.5GB.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Allow some 'headroom' for data set growth
If you are on linux and using only innodb tables, you can tell how large your database is by looking at the ibdata file in /var/lib/mysql/.
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
Thanks you! Now
Thanks you!
Now is:
innodb_buffer_pool_size = 4096Minnodb_log_file_size = 384M
innodb_flush_log_at_trx_commit = 0
innodb_log_buffer_size = 4M
innodb_flush_method=O_DIRECT
innodb_buffer_pool_instances = 8
innodb_thread_concurrency = 8
innodb_file_per_table = 1
After that:
After that: innodb_buffer_pool_instances = 8 mysql failed to running.
innodb_buffer_pool_size in GB
innodb_buffer_pool_size in GB + number of CPUs)/2
What does the mysqld.log show?
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
4 other 'edge-cutting' settings to consider MySQL 5.5+
superfedya Your current settings look good!
Four more 'edge-cutting' InnoDB my.cnf settings you might want
to try: (assuming a recent version of MySQL 5.5+)
innodb_flush_log_at_trx_commit = 2transaction-isolation = READ-COMMITTED
1) The 1st item; 'innodb_flush_log_at_trx_commit = 2'
In this case, the MySQL log buffer is written out to the file at each commit, but the flush to disk operation is not performed on it.
Notes: The flushing on the log file takes place once per second(varies with process scheduling needs).
There is a slight risk (cost); you can lose at most 'one second worth of transactions' in a crash.
2) READ-COMMITTED: With the READ-COMMITTED isolation level, the phenomenon of dirty read is avoided, because any uncommitted changes is not visible to any other transaction, until the change is committed. Within this isolation level each SELECT uses its own snapshot of the committed data that was committed before the execution of the SELECT. This has improved with MySQL 5.5.
3) Mmap is a new MyISAM feature available with MySQL 5.1+.
Typically improves MyISAM write/read performance ~6%.
myisam_use_mmap = 14) There is a new MySQL5.1+ option
'innodb_autoinc_lock_mode'= "Less InnoDB Locking (compared to previous MySQL 5.0)"Just to summarize, we can 'almost' remove auto_increment locking in InnoDB.
We need to set both innodb_autoinc_lock_mode=2 and binlog_format=row.
New Summary:
[mysqld]innodb_autoinc_lock_mode=2
binlog_format=row
innodb_flush_log_at_trx_commit = 2
tx_isolation=READ-COMMITTED
myisam_use_mmap = 1
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)