Fixing 'too many open files' error

Once in a while we stumble upon 'too many open files' error in log files especially for java application servers. What this means is that you have hit a limit on number of open file descriptors (fd)/file handles assigned for the process or user the application server is running as.

Let's use tomcat for our example.

So to get the number of fd's used by tomcat we run

$ pgrep tomcat

12345

$ sudo ls /proc/12345/fd | wc -l

2040

Don't use `lsof | wc -l` to get the number of fd's. `lsof` lists much more than the number of fd's in use. If you look at the FD column you'll see memory mapped files, current working directory, root directoy, the text segment of the running process etc listed which don't utilize file descriptors.

To see the fd limit run `ulimit` as the application user

$ sudo -u tomcat sh -c 'ulimit -n'

2048

One can also check for soft limit(-S) and hard limit(-H)

$ sudo -u tomcat sh -c 'ulimit -Hn'

2048

To see the system wide fd limit

$ cat /proc/sys/fs/file-max

148922

Or

$ sysctl fs.file-max

fs.file-max = 148922

To find the current system wide fd usage

$ cat /proc/sys/fs/file-nr

1184 0 148922

Or

$ sysctl fs.file-nr

fs.file-nr = 1184 0 148922

The first field shows the number of allocated fd's, second field gives the number of allocated but unused fd's and the last field shows the maximum number of fd's. So in the above example 1184 fd's are allocated and all are in use.

So once you hit a limit the next step, obviously, is to increase it a bit. So let's do that.

For example to change the limit for tomcat user append the below lines to /etc/security/limits.conf

tomcat      soft    nofile  1024

tomcat  hard    nofile  3072

We can also put

ulimit -n 3072

in the shell script (startup wrapper script or rc script) that invokes the application server to change the limit for that process as all processes spawned from the shell will inherit the limit.

To increase the system wide limit we have to change the kernel runtime parameter fs.file-max and save it in /etc/sysctl.conf

# echo 'fs.file-max = 200000' >> /etc/sysctl.conf

# sysctl -p

# sysctl fs.file-max

fs.file-max = 200000