Options

How to manage job agent stored information?

kaymankayman Member Posts: 662 Unicorn
edited December 2018 in Help

Hi all, seems the new job agent likes to store a lot of information since I had to remove a combined 15GB of data collected after installation of RM8 server earlier this month. As far as I can see all of the log data is stored in single packages and while this can be handy I would prefer to keep my disc management a bit more under control.

 

For now I created a cron job to simply dump every folder older than x days on a recurrent schedule but are there other ways to control this? It is a bit excessive and I want to avoid a crash because my system runs out of disk space before the warnings reach me.

Tagged:

Answers

  • Options
    homburghomburg Moderator, Employee, Member Posts: 114 RM Data Scientist

    Hi kayman, 

     

    it is possible to reduce log level within process parameters as well as via editing the corresponding parameter in config.properties of the jobagent. What is the exact issue in your case - do you experience extensive logging for a single process or do you run a huge number of processes causing many recordes to be created?

     

    Cheers,

    Helge 

  • Options
    kaymankayman Member Posts: 662 Unicorn

    More the second one, our server is kind of continuously running smaller processes, leading to a lot of packages which in itself are not that big but the combined amount sums up pretty quickly. The actual process log data is already limited in our workflows (for most it's just put to off), but the agent is adding quite some in addition it seems

  • Options
    kaymankayman Member Posts: 662 Unicorn

    Hi @homburg,

    Could you elaborate a bit more on how I would need to adjust the config files? I don't seem to find the right answer and I ran into problems again yesterday when one of our processes created a 1,5GB log file. Granted, the process was pretty heavy and included quite some operational steps but this is a bit on the excessive site.

     

    How can I avoid all is stored by default? In essence I am only interested in what went wrong, not in what went well. 

     

    How can I store my log files on a different location? My system drive (containing my server setup) is fairly small, my data drive is huge. So what settings do I need to apply to have the server store my job agent data onto a different location? Would you have an example or can someone else provide this?

     

    Thanks in advance!

     

     

  • Options
    Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    Hi @kayman,

     

    Within the file config.properties in the config folder of your job-agent you can set the logging level of the job-agent.

    Looks like this:

    ##
    # Logging level for Job Agent
    #
    logging.level.com.rapidminer = info

    I would try and set the level from "info" to "error" or "warn". I think "warn" includes "error" but could also be the other way round.

     

    Happy Mining,

    Edin

Sign In or Register to comment.