Filesystem out of inodes

Before version 0.9.0 redmine was storing session data in files on the tmp directory of the server. These files are usually of small size but a lot of them are produced quite fast on a production system. By default, redmine never deletes these files. A cron job could be used to delete session files older than a month or so, but to configure that cron job, you have to know of the problem first.

The easiest way to find out is to wait until you start getting “No space left on device” messages on the system log.

There are actually two reasons for such a thing to happen. The filesystem could have no more blocks available (the usual case) or the filesystem could be out of inodes (the not so usual case).

Inodes are filesystem data structures that store basic information about a regular file, directory, or other file system object. For each file, the filesystem occupies at least one inode.

The default inode ratio on an ubuntu server as configured in /etc/mke2fs.conf is one inode for every 16Kbytes of data. So creating a lot files of smaller size could lead to having enough space on the filesystem but no more inodes to create new files or directories.

You can see the amount of free inodes using df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            1251712 1251712       0  100% /
tmpfs                 129512       2  129510    1% /lib/init/rw
udev                  129512     488  129024    1% /dev
tmpfs                 129512       3  129509    1% /dev/shm
/dev/sda6            1251712  148360 1103352   12% /usr
/dev/sda7            35291136   21862 35269274    1% /home
/dev/sda8            35241984   31055 35210929    1% /var

Unfortunately, the system libraries of linux do not give out distinct error messages for a filesystem being out of blocks or being out of inodes. The message is always “No space left on device”.

Starting from version 0.9.0, redmine stores session data in cookies on the client side, so this problem will not occur any more.