How do you feel when your client WhatsApps a screenshot of his website that says 502 bad gateway NGINX error? Nothing can be worse than this (to start your day), isn’t it? While I wish you shouldn’t be starting your day like that, I’ll explain how to recover if you were put in that situation.
Before we discuss the issue & its solution, I’d like to remind that my hosting environment is powered by NGINX web server with an FPM PHP handler.
Reason for 502 Bad Gateway NGINX error
I started debugging the error from NGINX error log (/var/log/nginx/nginx.log) and found the below error message.
2019/12/11 04:30:43 [error] 27570#27570: *48599 connect() to unix:/var/run/php-fpm/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 17.58.98.98, server: techglimpse.com, request: "GET /pure-css3-shapes/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm/php-fpm.sock:", host: "techglimpse.com"
It’s evident that the error message is related to the PHP handler and that redirected me to look into PHP-FPM error log at /var/log/php-fpm/error.log and noticed the below error message:
[08-Dec-2019 03:30:01] NOTICE: error log file re-opened [10-Dec-2019 21:00:16] WARNING: [pool www] server reached pm.max_children setting (50), consider raising it
Now it is clear that the PHP-FPM master process is unable to spawn new child processes due to pm.max_children reaching the limit, causing 502 bad gateway error.
How to fix the error: server reached pm.max_children setting, consider raising it
The solution is to increase the pm.max_children limit based on the server specifications. Note, don’t blindly increase the limit, because if the website receives huge traffic, then the workers are never recycled, the RAM usage will grow indefinitely over time and the server will be in out of memory condition.
Therefore, pm.max_children should be increased carefully and gradually while monitoring swap usage.
pm.max_children = ((total RAM in MB) - (RAM usage by MySQL and others taken together in MB)) / 80
Here 80MB is average weight of a PHP-FPM worker process.
Based on the above calculation, increase pm.max_children value in the respective domain configuration file of PHP-FPM. In my case: /etc/php-fpm.d/techglimpse.conf
pm = ondemand pm.max_children = 200
Now restart PHP-FPM and NGINX services
systemctl restart php-fpm nginx
Filter all the domains that have been affected with pm.max_children and apply the above settings to only the affected ones.
How to find the memory used by applications
Run the below command to find the memory used by respective applications.
#ps -o pid,user,%mem,command ax | sort -b -k3 -r PID USER %MEM COMMAND 1355 mysql 14.5 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid 543 root 0.3 /usr/lib/systemd/systemd-journald 9254 nginx 0.3 php-fpm: pool www 9251 nginx 0.3 php-fpm: pool www
In the above example, MySQL uses nearly 14% of the total memory.
