No space left on device: Couldn’t create the rewrite-map mutex: Configuration Failed [Apache]

Since my webserver (apache) wasn’t responding, I restarted the apache service. Most of the time restarting works, this time it didn’t! The error message was “No space left on device: Couldn’t create the rewrite-map mutex: Configuration Failed”. Let’s try to diagnose the issue and find the solution.

Analyse why Apache restart not working?

Step 1: Though restarted apache, it didn’t start. Command systemctl status httpd  didn’t get proper error message.

# systemctl status httpd
Jul 12 10:21:19 tools systemd[1]: Starting The Apache HTTP Server...
Jul 12 10:21:19 tools httpd[341877]: [Wed Jul 12 10:21:19.952105 2023] [so:warn] [pid 341877] AH01574: module rewrite_module is already loaded, skipping
Jul 12 10:21:19 tools systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jul 12 10:21:19 tools kill[341879]: kill: cannot find process ""
Jul 12 10:21:20 tools systemd[1]: httpd.service: control process exited, code=exited status=1
Jul 12 10:21:20 tools systemd[1]: Failed to start The Apache HTTP Server.
Jul 12 10:21:20 tools systemd[1]: Unit httpd.service entered failed state.
Jul 12 10:21:20 tools systemd[1]: httpd.service failed.

Step 2: Used the command Journalctl -xe to get more error message. Tried few solutions from stackoverflow, such as restart the server, disable SELinux etc., but all in vain!

Jul 12 10:20:50 tools polkitd[1071]: Registered Authentication Agent for unix-process:341785:417934979 (system bus name :1.111577 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent
Jul 12 10:20:50 tools polkitd[1071]: Unregistered Authentication Agent for unix-process:341785:417934979 (system bus name :1.111577, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_IN.UTF-8) (disconnected from bus)
Jul 12 10:20:56 tools polkitd[1071]: Registered Authentication Agent for unix-process:341813:417935562 (system bus name :1.111578 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent
Jul 12 10:20:56 tools systemd[1]: Starting The Apache HTTP Server...

Step 3: Looked at error_log file /var/log/httpd/error_logThis actually provided me the exact error: “No space left on device: AH00023: Couldn’t create the rewrite-map mutex AH00016: Configuration Failed“.

[Wed Jul 12 10:17:41.524207 2023] [core:emerg] [pid 336293] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex AH00016: Configuration Failed
[Wed Jul 12 10:18:46.026923 2023] [core:emerg] [pid 336579] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex AH00016: Configuration Failed
[Wed Jul 12 10:20:11.492132 2023] [core:emerg] [pid 338286] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex AH00016: Configuration Failed
[Wed Jul 12 10:20:56.788551 2023] [core:emerg] [pid 341819] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex AH00016: Configuration Failed
[Wed Jul 12 10:21:19.955492 2023] [core:emerg] [pid 341877] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex AH00016: Configuration Failed
[Wed Jul 12 10:25:40.754027 2023] [core:emerg] [pid 342478] (28)No space left on device: AH00023: Couldn't create the rewrite-map mutex AH00016: Configuration Failed

Typically this error occurs when there is insufficient disk space or inodes available on the server to create necessary resources for Apache’s URL rewriting mechanism.

How to solve rewrite-map mutex apache error?

Step 4: Check Disk space and inodes using the commands below.

# df -h

# df -i

If your disk space or inodes are exhausted, you will need to free up some space or inodes by deleting unnecessary files or increasing your server’s storage capacity. However, in my case this wasn’t the issue.

Step 5: Check Apache’s Configuration files.

This wasn’t also an issue in my case, as it was working earlier and nobody updated the configuration files.

Step 6: The error probably occurred due to the operating system’s semaphore limit being reached, and Apache failing to perform proper cleanup as explained in stackoverflow article. Below are the quick steps to solve:

# ipcs -sl

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

ipcs -ls command shows the Semaphore Limits set in the OS.

# ipcs -us

------- Semaphore Status --------
used arrays = 132
allocated semaphores = 132

ipcs -us command lists currently used semaphores. It’s evident that the cause of the issue was exceeding the semaphore array limit, as we can clearly see that we are using more semaphore arrays than allowed.

Step 7: Remove the semaphore held by apache user

Below command removes all the semaphores filtered out owned by apache user.

# ipcs -s | awk -v user=apache '$3==user {system("ipcrm -s "$2)}'

# ipcs -us

------ Semaphore Status --------
used arrays = 0
allocated semaphores = 0

Step 8: Now restart apache

# systemctl restart httpd

Hooray! It worked.