Purging Old Job Agent Logs

bmatt

I've been searching around documentation and the community, but I can't seem to find anything about purging old job logs? We have accumulated nearly 200GB of job data that I don't think we really need any longer. It appears to be linked to a DB and provided via the RM server UI, but no way to clean them out there. I assume if I just delete the data we'd have some UI errors following.

# pwd

/opt/rapidminer/rapidminer-server/rapidminer-server-home/job-agent-home/data

# du -shx jobs

167G    jobs

mmichel

Hi bmatt,

welcome to the RapidMiner Community!
The automatic purging of logs and job data will be released soon - so stay tuned
In the meantime you could manually delete the jobs directory. Obviously the consequence is that the UI will not longer be able to display the logs of the old executions, but I guess this is fine from your point of view.

Cheers,
Marcel

sgenzer

cc @mmichel

bmatt

What I've found is there are jars copied to jobs as they run, they are not cleaned up afterwards. Particularly Radoop jobs have a 44MB hive jar that is copied every time. I create a python script to just purge these jars on inactive jobs that I can run on demand for now. That freed up about 165GB.

bmatt

Thanks! The script I made purges the jars which shouldn't, and doesn't seem to, impact the display of the old logs/etc.

mmichel

Thanks! The script I made purges the jars which shouldn't, and doesn't seem to, impact the display of the old logs/etc.

We are also working on a new execution strategy so that this jar duplication is not longer required. But thanks for pointing it out!