2

We have a server running LAMP stack on CentOS release 5.6 (Final) MySQL is having high CPU usage constantly which causes server to lag as shown

top - 13:50:47 up 80 days, 13:53,  1 user,  load average: 4.02, 3.44, 3.19
Tasks: 129 total,   2 running, 127 sleeping,   0 stopped,   0 zombie
Cpu0  : 24.7%us, 19.8%sy,  0.2%ni, 54.5%id,  0.7%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu1  : 23.0%us,  9.4%sy,  0.1%ni, 67.0%id,  0.3%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   4034120k total,  3317400k used,   716720k free,   124324k buffers
Swap:  8385920k total,     9788k used,  8376132k free,  2004692k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26266 mysql     15   0 2472m 464m 4448 S 158.6 11.8 109:00.59 mysqld
30276 apache    16   0  310m  16m 3724 S 12.1  0.4   0:00.85 httpd
30205 apache    16   0  309m  17m 3860 S  8.6  0.4   0:01.81 httpd

The server has 4GB RAM, tables are stored in MyISAM. Database is about 6665MB. It has a replicated database (readonly) on another server

MySQL version:

 +----------------+
 | 5.1.57-ius-log |
 +----------------+

Here's our my.cnf (IP information Xed out). [mysqld] # General datadir = /home/mysql tmpdir = /var/lib/mysqltmp socket = /var/lib/mysql/mysql.sock skip-external-locking = 1 skip-name-resolve open-files-limit = 20000 # Cache thread-cache-size = 16 table-open-cache = 2048 table-definition-cache = 512 query-cache-size = 32M query-cache-limit = 1M # Per-thread Buffers sort-buffer-size = 1M read-buffer-size = 1M read-rnd-buffer-size = 8M join-buffer-size = 1M # Temp Tables tmp-table-size = 64M max-heap-table-size = 64M # Networking back-log = 100 max-connections = 200 max-connect-errors = 10000 max-allowed-packet = 16M interactive-timeout = 600 wait-timeout = 600 #Storage Engines innodb = FORCE # MyISAM key-buffer-size = 64M myisam-sort-buffer-size = 128M # InnoDB innodb-buffer-pool-size = 16M innodb_log_files_in_group = 2 innodb-log-buffer-size = 4M # Replication server-id = 1 bind-address = XX.XX.XXX.XXX log-bin = /var/lib/mysqllogs/bin-log binlog-do-db = rewards1 binlog-ignore-db = mysql relay-log-index=master-relay-bin.index relay-log = /var/lib/mysqllogs/relay-log #Logging slow-query-log = 1 slow-query-log-file = /var/log/slow-log #MysqlTuner skip-innodb max_connections = 200 wait_timeout = 500 interactive_timeout = 500 query_cache_size = 64M join_buffer_size = 5M tmp_table_size = 256M max_heap_table_size = 256M table_cache = 4096 key_buffer_size = 2048M [mysqld_safe] log-error = /var/log/mysqld.log

We've seen similar problem as described in MySQL high CPU usage (MyISAM table indexes) which I've adjust the key_buffer_size to 2048M so it's greater than MyISAM index mysql> sELECT SUM(index_length) ndxsize FROM information_schema.tables WHERE engine='MyISAM'; +------------+ | ndxsize | +------------+ | 1876895744 | +------------+ MySQL continues to have high %CPU, our system admin suggested that we add more RAM to the box, will this solve the problem?

Any helps/suggestions are appreciated!!

Thank you!!!

Edited for show status, it seems that show status only shows session status, changed it to run global status mysql> show global status; +--------------------------------+--------------+ | Variable_name | Value | +--------------------------------+--------------+ | Aborted_clients | 15 | | Aborted_connects | 101 | | Binlog_cache_disk_use | 0 | | Binlog_cache_use | 0 | | Bytes_received | 4820268802 | | Bytes_sent | 104046088856 | | Com_admin_commands | 2 | | Com_assign_to_keycache | 0 | | Com_alter_db | 0 | | Com_alter_db_upgrade | 0 | | Com_alter_event | 0 | | Com_alter_function | 0 | | Com_alter_procedure | 0 | | Com_alter_server | 0 | | Com_alter_table | 0 | | Com_alter_tablespace | 0 | | Com_analyze | 0 | | Com_backup_table | 0 | | Com_begin | 0 | | Com_binlog | 0 | | Com_call_procedure | 0 | | Com_change_db | 1376751 | | Com_change_master | 0 | | Com_check | 0 | | Com_checksum | 2 | | Com_commit | 0 | | Com_create_db | 0 | | Com_create_event | 0 | | Com_create_function | 0 | | Com_create_index | 1 | | Com_create_procedure | 0 | | Com_create_server | 0 | | Com_create_table | 133 | | Com_create_trigger | 0 | | Com_create_udf | 0 | | Com_create_user | 0 | | Com_create_view | 0 | | Com_dealloc_sql | 0 | | Com_delete | 31518 | | Com_delete_multi | 0 | | Com_do | 0 | | Com_drop_db | 0 | | Com_drop_event | 0 | | Com_drop_function | 0 | | Com_drop_index | 0 | | Com_drop_procedure | 0 | | Com_drop_server | 0 | | Com_drop_table | 133 | | Com_drop_trigger | 0 | | Com_drop_user | 0 | | Com_drop_view | 0 | | Com_empty_query | 0 | | Com_execute_sql | 0 | | Com_flush | 0 | | Com_grant | 0 | | Com_ha_close | 0 | | Com_ha_open | 0 | | Com_ha_read | 0 | | Com_help | 0 | | Com_insert | 140425 | | Com_insert_select | 137 | | Com_install_plugin | 0 | | Com_kill | 0 | | Com_load | 0 | | Com_load_master_data | 0 | | Com_load_master_table | 0 | | Com_lock_tables | 0 | | Com_optimize | 0 | | Com_preload_keys | 0 | | Com_prepare_sql | 0 | | Com_purge | 0 | | Com_purge_before_date | 0 | | Com_release_savepoint | 0 | | Com_rename_table | 0 | | Com_rename_user | 0 | | Com_repair | 0 | | Com_replace | 5805 | | Com_replace_select | 0 | | Com_reset | 0 | | Com_restore_table | 0 | | Com_revoke | 0 | | Com_revoke_all | 0 | | Com_rollback | 3391 | | Com_rollback_to_savepoint | 0 | | Com_savepoint | 0 | | Com_select | 7088783 | | Com_set_option | 1226039 | | Com_show_authors | 0 | | Com_show_binlog_events | 0 | | Com_show_binlogs | 0 | | Com_show_charsets | 0 | | Com_show_collations | 0 | | Com_show_column_types | 0 | | Com_show_contributors | 0 | | Com_show_create_db | 0 | | Com_show_create_event | 0 | | Com_show_create_func | 0 | | Com_show_create_proc | 0 | | Com_show_create_table | 2 | | Com_show_create_trigger | 0 | | Com_show_databases | 24 | | Com_show_engine_logs | 0 | | Com_show_engine_mutex | 0 | | Com_show_engine_status | 0 | | Com_show_events | 0 | | Com_show_errors | 0 | | Com_show_fields | 17299 | | Com_show_function_status | 0 | | Com_show_grants | 0 | | Com_show_keys | 0 | | Com_show_master_status | 0 | | Com_show_new_master | 0 | | Com_show_open_tables | 0 | | Com_show_plugins | 0 | | Com_show_privileges | 0 | | Com_show_procedure_status | 0 | | Com_show_processlist | 131 | | Com_show_profile | 0 | | Com_show_profiles | 0 | | Com_show_slave_hosts | 0 | | Com_show_slave_status | 0 | | Com_show_status | 10990 | | Com_show_storage_engines | 0 | | Com_show_table_status | 13 | | Com_show_tables | 23 | | Com_show_triggers | 0 | | Com_show_variables | 3400 | | Com_show_warnings | 0 | | Com_slave_start | 0 | | Com_slave_stop | 0 | | Com_stmt_close | 0 | | Com_stmt_execute | 0 | | Com_stmt_fetch | 0 | | Com_stmt_prepare | 0 | | Com_stmt_reprepare | 0 | | Com_stmt_reset | 0 | | Com_stmt_send_long_data | 0 | | Com_truncate | 133 | | Com_uninstall_plugin | 0 | | Com_unlock_tables | 0 | | Com_update | 713677 | | Com_update_multi | 133 | | Com_xa_commit | 0 | | Com_xa_end | 0 | | Com_xa_prepare | 0 | | Com_xa_recover | 0 | | Com_xa_rollback | 0 | | Com_xa_start | 0 | | Compression | OFF | | Connections | 1381196 | | Created_tmp_disk_tables | 500962 | | Created_tmp_files | 17174 | | Created_tmp_tables | 1323921 | | Delayed_errors | 0 | | Delayed_insert_threads | 0 | | Delayed_writes | 0 | | Flush_commands | 1 | | Handler_commit | 0 | | Handler_delete | 38013 | | Handler_discover | 0 | | Handler_prepare | 0 | | Handler_read_first | 326121 | | Handler_read_key | 141753461 | | Handler_read_next | 2628857102 | | Handler_read_prev | 65624374428 | | Handler_read_rnd | 55126444 | | Handler_read_rnd_next | 9990315545 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 10330960 | | Handler_write | 22915553 | | Key_blocks_not_flushed | 0 | | Key_blocks_unused | 1424358 | | Key_blocks_used | 297815 | | Key_read_requests | 3988011306 | | Key_reads | 294476 | | Key_write_requests | 9047373 | | Key_writes | 439479 | | Last_query_cost | 0.000000 | | Max_used_connections | 81 | | Not_flushed_delayed_rows | 0 | | Open_files | 1623 | | Open_streams | 0 | | Open_table_definitions | 531 | | Open_tables | 1093 | | Opened_files | 2242361 | | Opened_table_definitions | 1076 | | Opened_tables | 1655 | | Prepared_stmt_count | 0 | | Qcache_free_blocks | 10115 | | Qcache_free_memory | 36606896 | | Qcache_hits | 6401379 | | Qcache_inserts | 6086477 | | Qcache_lowmem_prunes | 539333 | | Qcache_not_cached | 1001986 | | Qcache_queries_in_cache | 17468 | | Qcache_total_blocks | 45584 | | Queries | 18400635 | | Questions | 18400635 | | Rpl_status | NULL | | Select_full_join | 29962 | | Select_full_range_join | 0 | | Select_range | 2141059 | | Select_range_check | 0 | | Select_scan | 2084788 | | Slave_open_temp_tables | 0 | | Slave_retried_transactions | 0 | | Slave_running | OFF | | Slow_launch_threads | 4 | | Slow_queries | 4 | | Sort_merge_passes | 8587 | | Sort_range | 1924836 | | Sort_rows | 117048520 | | Sort_scan | 1366104 | | Ssl_accept_renegotiates | 0 | | Ssl_accepts | 0 | | Ssl_callback_cache_hits | 0 | | Ssl_cipher | | | Ssl_cipher_list | | | Ssl_client_connects | 0 | | Ssl_connect_renegotiates | 0 | | Ssl_ctx_verify_depth | 0 | | Ssl_ctx_verify_mode | 0 | | Ssl_default_timeout | 0 | | Ssl_finished_accepts | 0 | | Ssl_finished_connects | 0 | | Ssl_session_cache_hits | 0 | | Ssl_session_cache_misses | 0 | | Ssl_session_cache_mode | NONE | | Ssl_session_cache_overflows | 0 | | Ssl_session_cache_size | 0 | | Ssl_session_cache_timeouts | 0 | | Ssl_sessions_reused | 0 | | Ssl_used_session_cache_entries | 0 | | Ssl_verify_depth | 0 | | Ssl_verify_mode | 0 | | Ssl_version | | | Table_locks_immediate | 16998029 | | Table_locks_waited | 236687 | | Tc_log_max_pages_used | 0 | | Tc_log_page_size | 0 | | Tc_log_page_waits | 0 | | Threads_cached | 10 | | Threads_connected | 7 | | Threads_created | 5861 | | Threads_running | 6 | | Uptime | 250234 | | Uptime_since_flush_status | 250234 | +--------------------------------+--------------+ 249 rows in set (0.00 sec)

Show processlist during huge lag spike:

 mysql> show processlist;
 +---------+--------------+-----------+----------+---------+------+--------------+-------+
 | Id      | User         | Host      | db       | Command | Time | State | Info                                                                                     |
 +---------+--------------+-----------+----------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
 | 1421036 | rewards_user | localhost | rewards1 | Sleep   |  478 |              | NULL                                                                                                 |
 | 1421054 | rewards_user | localhost | rewards1 | Sleep   |  478 |              | NULL                                                                                                 |
 | 1421084 | rewards_user | localhost | rewards1 | Sleep   |  474 |              | NULL                                                                                                 |
 | 1421188 | rewards_user | localhost | rewards1 | Sleep   |  456 |              | NULL                                                                                                 |
 | 1421296 | rewards_user | localhost | rewards1 | Sleep   |  435 |              | NULL                                                                                                 |
 | 1421298 | rewards_user | localhost | rewards1 | Sleep   |  435 |              | NULL                                                                                                 |
 | 1421334 | rewards_user | localhost | rewards1 | Sleep   |  430 |              | NULL                                                                                                 |
 | 1421335 | rewards_user | localhost | rewards1 | Sleep   |  430 |              | NULL                                                                                                 |
 | 1421342 | rewards_user | localhost | rewards1 | Sleep   |  426 |              | NULL                                                                                                 |
 | 1421405 | rewards_user | localhost | rewards1 | Sleep   |  415 |              | NULL                                                                                                 |
 | 1421459 | rewards_user | localhost | rewards1 | Sleep   |  409 |              | NULL                                                                                                 |
 | 1421494 | rewards_user | localhost | rewards1 | Sleep   |  404 |              | NULL                                                                                                 |
 | 1421513 | rewards_user | localhost | rewards1 | Sleep   |  399 |              | NULL                                                                                                |

after mysql recovers

mysql> show processlist;
+---------+--------------+-----------+----------+---------+------+-------+------------------+
| Id      | User         | Host      | db       | Command | Time | State | Info             |
+---------+--------------+-----------+----------+---------+------+-------+------------------+
| 1423101 | rewards_user | localhost | rewards1 | Query   |    0 | NULL  | show processlist |
| 1427422 | rewards_user | localhost | rewards1 | Sleep   |    0 |       | NULL             |
+---------+--------------+-----------+----------+---------+------+-------+------------------+
2 rows in set (0.00 sec)

Thanks again!! appreciated :)

Community
  • 1
wayne
  • 21
  • 1
  • 3
  • Could you show the output of SHOW STATUS command from mysql? – Tero Kilkanen Mar 22 '14 at 10:05
  • Added the result for SHOW STATUS. Thanks! – wayne Mar 22 '14 at 17:16
  • It looks like this SHOW STATUS was taken just after server restart? The server should be running for a while and serving normal requests before SHOW STATUS has good information. – Tero Kilkanen Mar 22 '14 at 17:19
  • I might have restarted mysql yesterday due to the heavy lag. How long should I wait till next mysql status measure? Thanks! – wayne Mar 23 '14 at 00:13
  • A couple of hours shoule be enough. – Tero Kilkanen Mar 23 '14 at 09:35
  • Added the result for another SHOW STATUS. Thanks!! – wayne Mar 23 '14 at 18:17
  • Thanks for the SHOW STATUS. Unfortunately I didn't notice anything odd there. I see that you have MySQL slow log enabled, (/var/log/slow-log). Does it have anything inside it? You could take a look at the queries there and see if they can be optimized. – Tero Kilkanen Mar 23 '14 at 20:38
  • Slow log only picked up about 3~5 slow queries everyday, that's why I suspect that it's not the primary cause of this. Can it be caused by a lot of small queries? if it's the case, is there any ways that I can verify this? will more RAM fix this? Thanks – wayne Mar 24 '14 at 04:41
  • Actually the SHOW STATUS looks quite odd. The numbers starting with Com_ are counts of different type of SQL queries the server has processed. According to this, only one SELECT query has been made, which doesn't look right at all. – Tero Kilkanen Mar 24 '14 at 10:50
  • Hi, this is definitely weird, I manually ran a checksum and a select query, and the com_checksum and com_select recorded them. – wayne Mar 24 '14 at 16:34
  • It seems that show status only shows the session status, I reran it with SHOW GLOBAL STATUS, updated the results in the post, good catch! – wayne Mar 24 '14 at 16:44
  • we just had another lag spike, I captured the SHOW PROCESSLIST, it seems to be similar issue described in http://dba.stackexchange.com/questions/41601/mysql-high-cpu-usage-myisam-table-indexes I've added the SHOW PROCESSLIST; results to the post – wayne Mar 24 '14 at 19:10
  • on 2nd thought tho..the sleeping process could be caused by the constant cpu load, the cpu load stays high even after mysql recovered. – wayne Mar 24 '14 at 20:00
  • Sorry for the SHOW STATUS / SHOW GLOBAL STATUS mixup. Your handler_read_* values are quite high. This indicates that your queries cause lots of table scans. This can be fixed by proper indexing of columns. – Tero Kilkanen Mar 25 '14 at 01:51

0 Answers0