Purging Old Job Agent Logs

bmattbmatt Member Posts: 8 Contributor II
I've been searching around documentation and the community, but I can't seem to find anything about purging old job logs? We have accumulated nearly 200GB of job data that I don't think we really need any longer. It appears to be linked to a DB and provided via the RM server UI, but no way to clean them out there. I assume if I just delete the data we'd have some UI errors following.

# pwd
/opt/rapidminer/rapidminer-server/rapidminer-server-home/job-agent-home/data
# du -shx jobs
167G    jobs

Tagged:

Best Answer

Answers

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
  • bmattbmatt Member Posts: 8 Contributor II
    What I've found is there are jars copied to jobs as they run, they are not cleaned up afterwards. Particularly Radoop jobs have a 44MB hive jar that is copied every time. I create a python script to just purge these jars on inactive jobs that I can run on demand for now. That freed up about 165GB.
  • bmattbmatt Member Posts: 8 Contributor II
    Thanks! The script I made purges the jars which shouldn't, and doesn't seem to, impact the display of the old logs/etc.
  • mmichelmmichel Employee, Member Posts: 129 RM Engineering
    Thanks! The script I made purges the jars which shouldn't, and doesn't seem to, impact the display of the old logs/etc.
    We are also working on a new execution strategy so that this jar duplication is not longer required. But thanks for pointing it out!
Sign In or Register to comment.