Skip to main content

How to manage log files


The Fleek Network node streams text messages to the standard output and error in Linux. Given that disk space is a limited resource for most systems, the number of text data can be a cause of concern. Managing the file sizes by means of rotation and compression can help.

A conventional Fleek Network Service setup (as per the documentation and tool recommendations), have the text data stored in a special directory for storing logs called /var/log. The /var/log directory contains logs from various applications running on the operating system, the operating system itself, and others.

For the purpose of storing network node operation logs, it defaults to the /var/log/lightning directory where the files output.log (stdout) and diagnostic.log (stderr) is located and accumulated. As the node runs, the size of the files increase, as it aggregates all the output generated by the Fleek Network service operations, such as info, errors, etc.

A Node Operator can configure the system to rotate, compress, and set maximum size of these files to safeguard any concerns or issues that can get out of control, such as causing disk space to become full quickly.

In this guide, we'll look into some options available to manage the logs. First, we'll look into journald.conf which controls where to store journal data (the journal is a component of systemd that handles all the messages in a Systemd enabled system). Afterwards, we'll look into Logrotate an application to help us manage automatic rotation and compression of log files.

In essence, the journal and the logs duplicate the same information, and we want to make sure that we set measures to control it.


Journal is a component of Systemd, a centralized location for all messages logged by different components in a systemd-enabled Linux system. The systemd journal will happily run in parallel with the standard type log files in /var/log/* where the Fleek Network Systemd Unit Service outputs Standard Output and Standard Error in the location /var/log/lightning/*.log.

Here, we are going to learn how to configure the journald.conf service configuration file for the system-wide settings (meaning that it applies to all services), but firstly we're going to learn some commands to help us troubleshoot when necessary.


Disk utilization checkup

To check how much disk space is used by Systemd log files, run the command below:

sudo journalctl --disk-usage

It provides information of how much disk space is utilized by the log files in your system.

Clearing logs manually

The best way to clear log files is done automatically by the journald.conf configuration file, discussed in Configuration file. In the ideal world, you shouldn't have to manually delete the log files, but this can be useful to know about when troubleshooting.

To flush the log files run the command below:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-time=1s

The flush and rotate flag is used, as vaccum-time only clears archived logs and not active ones. It'll flush:

  • Move /run/log/journal to /var/log/journal
  • Rotate (this flag archives logs and retains it)

Since it'll only keep the past 1-second-long files, it'll effectively clear everything.

Follow or tail logs of service

sudo journalctl -u <SERVICE-NAME>.service -f

For example, for a conventional native install (if you haven't followed the conventions, make the appropriate tweaks to fit your needs).

sudo journalctl -u lightning.service -f

The Docker service

sudo journalctl -u docker-lightning.service -f

Show all service entries

Show all journal entries, which can be fairly long:

sudo journalctl

Configuration file

The default configuration file is located at /etc/systemd/journald.conf. This is the main file that journal reads the configuration from, for the system instances that are controlled by root.

In addition to the main configuration file there are a few other locations that take higher precedence and override the main configuration file. To learn more about journald read here.

To keep the guide short we are going to use the main location /etc/systemd/journald.conf.

Create the directory and config file

If the /etc/systemd/journald.conf file doesn't exist, create it by:

Create the journald.conf file:

sudo touch /etc/systemd/journald.conf

Configuration settings

The Systemd provides many options for you to manage the log files and by combining these parameters you can limit the disk space used by the journal files.

A list of the available options are here.

Here is a quick description of the options we're going to use for our example:

  • Storage, controls where to store journal data
  • SystemMaxUse, specifies the maximum disk space that can be used by the journal in persistent storage
  • SystemMaxFileSize, controls how large individual journal files can grow to in persistent storage before being rotated
  • RuntimeMaxUse, control how much disk space the journal may use up at most

You should open the /etc/systemd/journald.conf file in your favorite text editor and put:


Here, we set 1G and 100M, which means 1 Gigabyte and 100 Megabytes. You can also use K for Kbytes, amongst others.

After the changes, you have to restart the journald after updating the file. To restart use the command:

sudo systemctl restart systemd-journald

You can verify the integrity of the log files by running:

sudo journalctl --verify


Logrotate is a tool to manage the log files created by a system processes. It can automatically compress, rename, remove logs and more for your convenience and save your system's resources. Log files can be handled timely, or when it grows too large.


The logrotate tools is available by default on Ubuntu.


We're using Ubuntu for our guide to keep it simple. If you are using a different and support operating system make sure you install Logrotate before proceeding.

You can check if logrotate is installed by executing:

logrotate --version

At the time this guide was written, we got the following output:

logrotate 3.21.0

Default mail command: /usr/bin/mail
Default compress command: /bin/gzip
Default uncompress command: /bin/gunzip
Default compress extension: .gz
Default state file path: /var/lib/logrotate/status
ACL support: yes
SELinux support: yes

If you run an earlier or older version, changes or tweaks might apply.

Explore the Configuration files

The configuration files we'll be exploring today are found in the following locations in Ubuntu:

  • /etc/logrotate.conf, is the main configuration settings file. It includes the statement to pull in configuration files from other directories, e.g. the /etc/logrotate.d

  • /etc/logrotate.d/, a directory where packages drop log rotation information


We use Ubuntu for our guide, if you are on a different distro, you have to determine the file configuration file locations

The content of the file /etc/logrotate.conf should be similar to:

# see "man logrotate" for details

# global options do not affect preceding include directives

# rotate log files weekly

# use the adm group by default, since this is the owning group
# of /var/log/.
su root adm

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones

# use date as a suffix of the rotated file

# uncomment this if you want your log files compressed

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may also be configured here.

The configuration settings if anything like the above, tell us that the rotation happens weekly, keeping 4 weeks worth of backlogs, etc.

To learn more about other configuration options consult the logrotate manual page.

Create the Lightning logrotate configuration file

Let's create a Logrotate configuration file for Fleek Network Lightning Service. Create the file by running the command:

sudo touch /etc/logrotate.d/lightning

Lightning Configuration Settings

Open the recently created file /etc/logrotate.d/lightning in your favorite text editor.

Add the following lines to the file:

/var/log/lightning/*.log {
rotate 5
size 50M

Remember to save the file before exiting the text editor. You can test the configuration file by running the command:

sudo logrotate /etc/logrotate.conf --debug

The configuration file above declares that for the log files in the /var/log/lightning directory, the log files are rotated 5 times daily before being removed, as long they grow bigger than 50 megabytes in file size or empty. Old versions of log files are compressed with gzip.


Bear in mind that this configuration file also inherit the default behavior, e.g. the create as set in the main configuration file /etc/logrotate.conf.

Feel free to customize the settings to your liking by checking the documentation in the logrotate manual page.

Cron job


Depending on the operating system, you have to set up a cron job to execute logrotate with the configuration file daily. Since we are using Ubuntu for our example, a daily cron job runner is already set up for us. If you are on a different Distro/OS make the required amends.

Verify that the /etc/cron.daily/logrotate exists and includes the execution of logrotate with the configuration argument /etc/logrotate.conf.

/usr/sbin/logrotate /etc/logrotate.conf

If you have modified the location of the binary or main configuration file, make sure this is set correctly to your custom locations.

To summarize, the logrotate /etc/logrotate.conf is executed and as logrotate.conf goes through its list of commands, it calls include /etc/logrotate.d. It means that any scripts in /etc/logrotate.d are executed, such as the Lightning Configuration Settings.


The guide starts by warning us about the stream text messages that the Fleek Network emits by default. As the text data is aggregated and stored in the file system it can lead to fill up the limited available disk space quickly, causing issues to the operation of the system.

To help control it, the journald is introduced, by explaining its role as a centralized message system, that runs alongside the application logs. Then, have it configured to limit the maximum file size, amongst other system-wide settings.

Finally, logrotate is discussed in helping us manage the Fleek Network Lightning application log files by setting it to automatically compress, rename, remove logs for the system admin convenience and saving system's resources.

Helder Oliveira
Helder OliveiraSoftware Developer + DXGot questions? Find us on discord!