In a Linux system and pretty much all systems – log files are crucial when it comes to examining and troubleshooting errors. They provide important clues as to what could have gone wrong with various system services prior to failure.
Any service installed on your Linux systems such as Apache web server or MySQL database server generates log files that are usually stored in the /var/log directory. If you check the contents of this directory, you will see contents similar to what we have below:
Over time, as additional information gets logged, the log files increase in size and take up more space on your hard drive. Before you even know it, the log files will have ballooned in size, gobbling up much of your hard drive space, and if you are not careful, you can easily run out of disk space.
With that in mind, it becomes prudent to keep the log files to a manageable size and delete old log entries that hog precious disk space. And this is where the log rotation comes in.
What is log rotation ?
Log rotation is a process that creates new log files and archives & removes old ones to save on disk space. The process renames a current log file. For example, apport.log becomes apport.log.1 and a new apport.log log file is created to log new log entries. Older log files are usually compressed and appear as apport.log.2.gz, apport.log.3.gz, apport.log.4.gz, and so on.
The log rotation process is facilitated using a utility called logrotate. This is a tool that facilitates the rotation of log files and archival & removal of old ones to free up disk space. In summary, logrotate accomplishes the following:
- Creation of new log files after rotating old ones.
- Archival of old log files.
- Purging of older log files that have been rotated to save on space.
Log rotation is usually activated when the size of log files grows and exceeds a certain limit.
How logrotate utility works
Before we examine the workings of the logrotate utility, ensure that logrotate is installed on your system. To do that, issue the command:
For Debian / Ubuntu System:
$ sudo apt-get install logrotate -y
For CentOS / RHEL / Fedora System:
$ sudo yum install logrotate -y or $ sudo dnf install logrotate -y
Run below command to check logrotate version,
linuxtechi@ubuntu-server:~$ logrotate --version logrotate 3.14.0 Default mail command: /usr/bin/mail Default compress command: /bin/gzip Default uncompress command: /bin/gunzip Default compress extension: .gz Default state file path: /var/lib/logrotate/status ACL support: yes SELinux support: yes linuxtechi@ubuntu-server:~$
From the output, we can clearly see that we have logrotate version 3.14.0. By default, logrotate comes preinstalled in modern Linux distributions and hence no need to install it.
Logrotate configuration files
Logrotate runs daily as a cron job, going through various log files, rotating them, and purging older log files as defined in the configuration file. There are two main configuration sources that you need to pay close attention to:
/etc/logrotate.conf – This is the main configuration file for the logrotate tool. It contains default settings and facilitates log rotation for non-system package logs. More notably, it uses an ‘include‘ directive for pulling configurations located in the ‘/etc/logrotate.d‘ directory. Let’s have a look at the configuration file.
$ cat /etc/logrotate.conf
From the configuration shown, the /etc/logrotate.conf file rotates log files on a weekly basis as indicated on line 3.
- Line 7 indicates that the root user and the adm group own the log files.
- Line 10 indicates that only 4 weeks’ worth of log files are backed up after which older ones will be purged or removed to create more disk space.
- Line 13 instructs the creation of a new log file after the rotation of the current log file.
- The include statement on line 22 pulls the configurations of application files that are listed in the /etc/logrotate.d directory.
/etc/logrotate.d – This is a directory that contains logrotate configuration of installed packages whose log files require log rotation. Typically, you are also likely to find configuration files of system tools such as apt & dpkg (For Debian systems), rsyslog, ufw, and cups-daemon. Here’s what you’d find:
linuxtechi@ubuntu-server:~$ ls -l /etc/logrotate.d/ total 60 -rw-r--r-- 1 root root 120 Sep 5 2019 alternatives -rw-r--r-- 1 root root 126 Dec 4 20:25 apport -rw-r--r-- 1 root root 173 Apr 9 11:21 apt -rw-r--r-- 1 root root 91 Apr 1 10:49 bootlog -rw-r--r-- 1 root root 130 Jan 21 2019 btmp -rw-r--r-- 1 root root 181 Feb 17 08:19 cups-daemon -rw-r--r-- 1 root root 112 Sep 5 2019 dpkg -rw-r--r-- 1 root root 329 Feb 4 2019 nginx -rw-r--r-- 1 root root 94 Feb 8 2019 ppp -rw-r--r-- 1 root root 501 Mar 7 2019 rsyslog -rw-r--r-- 1 root root 677 Nov 29 02:08 speech-dispatcher -rw-r--r-- 1 root root 119 Mar 30 21:49 ubuntu-advantage-tools -rw-r--r-- 1 root root 178 Jan 21 22:16 ufw -rw-r--r-- 1 root root 235 Apr 13 23:37 unattended-upgrades -rw-r--r-- 1 root root 145 Feb 19 2018 wtmp linuxtechi@ubuntu-server:~$
Let’s take a look at the configuration file of the dpkg package manager tool.
$ cat -n /etc/logrotate.d/dpkg
- monthly: This instructs rotation of log files once in a month
- rotate 12: 12 old log files are backed up.
- compress: This implies that rotated files are to be compressed using the default gzip compression with log files having a .gz file extension.
- Create 644 root root: Creates a new log file as soon as log rotation is completed with octal file permissions of 644 with user and group ownership of root.
- missingok: The directive suppresses error messages in the event of a missing log file.
- notifempty: This ignores file rotation if the log file is empty.
Create a sample logrotate configuration file
Let’s assume that we have an application running as linuxtechi user and is generating log files that are stored in the /home/linuxtechi/logs directory. We need to set the log files to rotate on a weekly basis.
But first, we are going to create a logrotate configuration file at the home directory as shown:
$ vim /home/linuxtechi/logrotate.conf
Next, we are going to paste the configuration shown:
/home/linuxtechi/logs/*.log {
weekly
missingok
rotate 14
compress
create
}Let’s put this into context:
The log files will be rotated on a weekly basis, with suppression of any error messages if any of the log files are missing. 14 log files will be backed up within the course of the month with the creation of a new log file after rotation of the current log file.
Now we are going to create a logs directory that will contain the log files of the application and then create a log file called app.log.
linuxtechi@ubuntu-server:~$ mkdir logs && cd logs linuxtechi@ubuntu-server:~/logs$ touch app.log linuxtechi@ubuntu-server:~/logs$ ls app.log linuxtechi@ubuntu-server:~/logs$
Now, we will run the logrotate command to create a logrotate state file in the home directory in order to verify if the log entries have been created or not.
$ logrotate /home/linuxtechi/logrotate.conf --state /home/linuxtechi/logrotate-state --verbose
You will get output similar to this:
From the output, the log file was not rotated for the simple reason that rotation occurs weekly and the log file is barely an hour old.
Examine the logrotate file to verify if there was any information recorded about the log rotation run.
linuxtechi@ubuntu-server:~$ cat logrotate-state logrotate state -- version 2 "/home/linuxtechi/logs/app.log" 2020-5-24-17:0:0 linuxtechi@ubuntu-server:~$
From the output, we can see that the logrotate utility acknowledged when last it considered the log file for rotation, and the timestamp is printed.
Now we shall force logrotate to rotate the log file – which it otherwise wouldn’t for now since the time interval specified has not yet been exceeded – using the –force flag as shown.
$ logrotate /home/linuxtechi/logrotate.conf --state /home/linuxtechi/logrotate-state --verbose --force
If you head back to the logs directory, you will observe an additional log file that has been rotated and compressed as shown.
linuxtechi@ubuntu-server:~$ cd logs/ linuxtechi@ubuntu-server:~/logs$ ls app.log app.log.1.gz linuxtechi@ubuntu-server:~/logs$
Compress and rotate the log files based on the size
Sometimes, log files can grow bigger and gobble up space even before the specified time interval for rotation, whether daily, weekly or monthly.
One way to resolve the issue is to specify the maximum size of the file which when exceeded, rotation of the log file will be triggered. To achieve this, specify the maxsize option in the logrotate file.
For example, to trigger rotation when the file size grows more than 40 Megabytes, include the option:
maxsize 40M
Let’s assume we create a custom log rotation file for our application under /etc/logrotate.d folder,
linuxtechi@ubuntu-server:~$ cd /etc/logrotate.d/
linuxtechi@ubuntu-server:/etc/logrotate.d$ sudo vi custom-app
/home/linuxtechi/logs/app-access.log
{
daily
missingok
size 40M
rotate 4
compress
create
}Save and Close the file,
The suffix M denotes Megabytes, For kilobytes use k while G indicates the size in Gigabytes.
The statement implies that log files bigger than 40 Megabytes will be rotated without any regard for the time interval of rotation. This implies that for a log file that is due for rotation after 1 hour, it will be rotated before the specified interval if it exceeds the threshold of 40MB.
Let’s dump some data into the log file of our custom application and then see how logrotate will rotate the log files,
linuxtechi@ubuntu-server:~$ dd if=/dev/zero of=/home/linuxtechi/logs/app-access.log bs=1M count=25 25+0 records in 25+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.0422015 s, 621 MB/s linuxtechi@ubuntu-server:~$ du -sh /home/linuxtechi/logs/app-access.log 25M /home/linuxtechi/logs/app-access.log linuxtechi@ubuntu-server:~$
Use ‘-d’ option in logrotate command to perform dry run of logrotate on the log file, run the following command,
$ logrotate -d /etc/logrotate.d/custom-app
As we can see in the output, logrotate will not rotate the log file as size is not more than 40MB.
Now Let’s make log file size more than 40MB,
linuxtechi@ubuntu-server:~$ cd logs/ linuxtechi@ubuntu-server:~/logs$ dd if=/dev/zero of=app-access.log bs=1M count=45 45+0 records in 45+0 records out 47185920 bytes (47 MB, 45 MiB) copied, 0.136979 s, 344 MB/s linuxtechi@ubuntu-server:~/logs$ du -sh app-access.log 46M app-access.log linuxtechi@ubuntu-server:~/logs$
Now, try to run logrotate command with ‘-d’ option again,
Above output confirms that logroate will rotate the log file as size is more than 40MB.
In addition to enhancing the rotation of log files based on size, it’s prudent to ensure that the logrotate configuration file is called on a regular frequency using a cron job. This is especially critical for log files that balloon in size so rapidly and risk filling your disk space.
There are two ways to go about this:
You can copy the logrotate script from /etc/cron.daily directory to the /etc/cron.hourly location. This will switch the log rotation to an hourly basis as opposed to a daily one.
Another way is to specify a cron job in the /etc/crontab file as shown
*/10 * * * * /etc/cron.daily/logrotate
This will trigger the rotation after every 10 minutes.
Using the crontab to call the logrotate script in addition to specifying the maximum size using the maxsize directive, makes for a perfect combination in ensuring that your log files are rotated in good time to avoid filling up your hard drive.
For additional options with logrotate tool, visit the man pages as shown:
Conclusion
In this guide, we have shed light on the importance of log files, the menace that they can cause if left to increase in size, and how the logrotate tool can help manage the size of log files on your system. We also looked at some of the available options for use in logrotate configuration.
