2

I've 4 web servers, 2 database servers (read & write), Redis and a load balancer. I'm using PHP 7.0 & Apache2 & MySQL 5.7, however when I do load tests with 3000 users which surf at the same time, the servers crash one by one.

The Apache servers are strong, each one has 32GB ram and sadly all of them reach to the maximum usage of RAM.

We've made a lot of changes to the code, however, it does not feel right to me, my hosting providers can't help me further config my server as they say this is the best they can do, and here I need your help - what are we missing? which configurations in the PHP / Apache / MySQL should be set?

I would clarify that the site works well for let's say 300 clients, but as more and more clients visit, the response time reaches to 50 seconds per request which is really weird, how only 400 clients make my server to respond so slowly?

I'm really despaired and confused, what should we look at?

Thanks a lot.

MyLibary
  • 121
  • 2
  • 1
    Do you actually have 3000 concurrent users? That kind of traffic is going to destroy some of the most robust systems... Look into cutting php/mysql out as much as possible. the more static files you can serve the better. – Daniel Widrick Jun 20 '18 at 15:35
  • 1
    We had even 6000 last summer. How do 4 servers with load balancer not handle such requests? I start to get errors 503 when only 1000 people are on the server, which makes me wonder even more. – MyLibary Jun 20 '18 at 15:39
  • 1
    Fairly typical Apache problems are the wrong MPM and using mod_php instead of php fpm. In general to address issues arising during perf testing you need to start by collecting detailed monitoring and logs on all aspects of your system . If the issue is not immediately obvious you can you can then write a great question with sufficient useful details that we can help you answer. – HBruijn Jun 20 '18 at 15:55
  • 1
    @HBruijn so what ideal MPM should I use if I change to PHP fpm? Unfortunately, I did not find any errors, only out of memory errors. Also, should the use of ngnix instead solve the problem for me? – MyLibary Jun 20 '18 at 16:13
  • 1
    The message there is: don't use prefork – HBruijn Jun 20 '18 at 19:46
  • 1
    @HBruijn so mpm event along with PHP fpm is the ideal way to go? – MyLibary Jun 20 '18 at 19:56
  • 1
    That usually gives a significant performance increase compared to prefork MPM and mod_php, yes. But run your banchmark, change settings and repeat your benchmark to see what tuning works best for you. – HBruijn Jun 20 '18 at 20:00
  • 1
    Sorry, last question, as I can't use ngnix, is the combination of mpm event with PHP fpm is the best for heavy load with a big amount of clients ? – MyLibary Jun 20 '18 at 20:02
  • 1
    Does your PHP application use a database? – Gerard H. Pille Jun 21 '18 at 13:24
  • 1
    Yep, MySQL and Redis – MyLibary Jun 21 '18 at 16:36
  • @MyLibary Additional information request, please. Post on pastebin.com or here. A) complete (not edited) my.cnf-ini
    Text results of: B) SHOW GLOBAL STATUS; C) SHOW GLOBAL VARIABLES; D) complete MySQLTuner report E) SHOW ENGINE INNODB STATUS; Optional very helpful information, if available includes - htop, top & mytop for most active apps, ulimit -a for a linux/unix list of limits, iostat -x when system is busy for an idea of IOPS by device, df -h for a linux/unix free space list by device, cat /proc/meminfo includes VMallocUused
    for server tuning analysis.     – Wilson Hauck Jul 02 '18 at 20:02

1 Answers1

2

We had even 6000 last summer. How do 4 servers with load balancer not handle such requests?

So what changed? To answer the question, almost always by being bogged down in backend code processing, and sometimes by things like tcp socket exhaustion, and other times it is just simple pool/etc limits between apache and php.

Follow this process:

  1. Eliminate PHP from the stack. Serve only small static files for your testing
  2. Observe results and tune apache/OS to accommodate the desired connections
  3. Reintroduce PHP. The server will be unable to match static file serving performance
  4. Profile your PHP code, identify slowdowns and resolve them
  5. Finally: Cache whatever PHP results you possibly can.

"How do I get nondescript php application to meet x benchmark?" is a very broad question but the process is generally, get your frontend static serving to beat your requirements, then add your backend code to the mix. Either ix slowness in the backend code or eliminate it.

Once you start hitting thousands of concurrent users.. You're going to brush up against certain limits (filesystem speed, memory, cpu speed, communication latency, inefficient wordpress code and slow database calls as well as a ton of context switching on the cpu) that are going to compound and quickly result in exponential degradation of service.

Daniel Widrick
  • 3,488
  • 2
  • 13
  • 27