Journal size

[svatosFZU]

Hi,
I am using a journal in HyperQueue (https://it4innovations.github.io/hyperqueue/v0.19.0/deployment/server/#resuming-stoppedcrashed-server). What does it actually contain? My journal is 9GB at this point:

[svatosm@login1.barbora ~]$ ll hqJournal 
-rw-rw-r--. 1 svatosm svatosm 9147535765 Nov 15 06:30 hqJournal

which is nearly half of my home quota. I tried to decreased its size by using job forget command. Right now, there are only couple of hundreds of jobs existing in the HyperQueue:

[svatosm@login1.barbora ~]$ /home/svatosm/hq-v0.19.0-linux-x64/hq job list
+----+------+-------+-------+
| ID | Name | State | Tasks |
+----+------+-------+-------+
There are 783 jobs in total. Use `--all` to display all jobs.

Yet, the size is still 9GB. Is there a way to decrease journal size?

[spirali]

Right now, there is no way to decrease journal file size. I will try to implement a command for this. Do you need journal prunning while server running or it is ok for you to have prunning only when the server is offline?

[svatosFZU]

Well, with the journal I can stop the server, deal with the journal file, then start the server and all jobs will be still there, right? If so, then it is OK to do it offline.

[spirali]

I have implemented "hq journal prune" in #786. It was actually easier to do it on a live server, so you do not need to stop it.
Once it is merged, you can try it in a next nightly build

[svatosFZU]

Thanks. As it was completed, since which version this would be available? Also, I see there several commands (flush, prune). Could they be documented in https://it4innovations.github.io/hyperqueue/v0.19.0/deployment/server/#resuming-stoppedcrashed-server ?

[spirali]

You can find it tomorrow in the nightly (that has compatible journal format with v0.20.0). It will be part of v0.21.0. There is already some documentation in the commits that will be published when v0.21.0 is released.

In short:

• flush is mostly for internal purposes. You do not need to call it.
• prune is what you need. It will remove finished jobs and non-running workers from the journal. (Note: It does not modify the memory of the server, only journal file, so you may still want to call "job forget" to cleanup memory of the server).