I was working on a web application for my client and its website load time was around 2-3 secs on a sandbox environment. However, the application load time was unexpectedly high (around 20 to 30 seconds) on the production machine. The web server was behind the NAT hiding its IP address from the outside world. Also, the application was based on WordPress and its version and plugins were up-to-date. For instance, the WordPress admin panel took around 30 secs to load and the AJAX requests built into the application were having similar load time. It took a couple of days to understand the reason behind this issue and fortunately, I was able to fix the same.
I started debugging the issue one by one as follows…..
Troubleshoot 1: Disabled all the plugins and tested. Still, there was no change in the load time of the website.
Troubleshoot 2: Created few static HTML/PHP files and accessed them. To my surprise, it was loading faster with a loading time of a few milliseconds!!! So the issue was with the application.
Troubleshoot 3: The web server is behind the NAT and here’s the diagram that explains the network flow.
It is a method of mapping a public IP address into private IP address on a network hardware. NAT is used, due to depletion of IPv4.
The WordPress Plugin page load time was unusually high compared to other pages in the admin panel. After googling around, found an article that said, the WordPress plugin page does heavy lifting work compared to other pages. It lists all the plugins and also polls the WordPress repository for plugin updates. Hence it usually takes a long time, but can be worse when it needs to go through NAT, firewall restrictions, and bad DNS.
Troubleshoot 4: Keeping the above point in mind, I used the Chrome developer tool to monitor the webpage loading time and found TTFB (Time to First Byte) was more than 20 secs. While some of the elements were loading faster, the AJAX request and the assets with self-referential URLs were blocking the entire website loading. Well, the hint is here.
WordPress uses Self-Referential URLs to load its own templates/CSS, scripts, and images. For e.g., techglimpse.com/css/style.css. Since the webserver is behind NAT; firewalls were configured to hide the real IP address of the server and that had the worst side effect on WordPress – to access its own URLs, the traffic is routed through the external interface, which is where the DNS resolves to. The above image explains this scenario and well, this was the reason behind the increased load time of the application.
Fixing high load time of WordPress & AJAX requests behind NAT
The fix is very simple, as I just need to let the server know that its IP address is readily available in the /etc/hosts file and does not need to refer to the address on the firewall.
Step 1: Add the server’s IP (the private IP) in /etc/hosts file mapping to its domain.
<private_ip> techglimpse.com
Step 2: In /etc/nsswitch.conf makes sure the below entry is in the same sequence:
#hosts: db files nisplus nis dns hosts: files dns myhostname
Step 3: Add localhost address as one of the nameservers in /etc/resolv.conf
nameserver 127.0.0.1
Hurray! Now the entire web application was loading at a supersonic speed!
Well, I have used the nameserver 127.0.0.1, as the application does not require to resolve domains other than techglimpse.com. But if you need to resolve other domains, then nameserver 127.0.0.1 will cause issues.
