What is the correct syntax to use Execute program on Linux?

kaymankayman Member Posts: 662 Unicorn
edited November 2018 in Help

Hi there,

 

I currently have a CRON trigger that executes a script on a daily base on my server, but I want to be able to do the same from within Rapidminer on demand.

 

As far as I understand the Execute Command operator should allow me to do that, but as nothing seems to happen (also no errors thrown) I am not sure I use it correctly.

 

My command is a curl command that loads a json file to my elastic search environment. Running this as a standard bash or sh file works fine, all my data is loaded to elastic, but running the same command in the operator just does nothing. I get a green tick indicating the process ran fine, but no data is transfered.

 

My curl command is as follows :

 

curl -s -H "Content-Type: application/x-ndjson" -XPOST http://localhost:9200/_bulk --data-binary "@mydata.json"

 

And this is my process :

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="productivity:execute_program" compatibility="7.6.001" expanded="true" height="103" name="Execute Program" width="90" x="112" y="34">
<parameter key="command" value="curl -s -H &quot;Content-Type: application/x-ndjson&quot; -XPOST http://localhost:9200/_bulk --data-binary &quot;@mydata.json&quot;"/>
<parameter key="working_directory" value="/data/elastic/imports"/>
<list key="env_variables"/>
</operator>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
</process>
</operator>
</process>

Any idea if I am using it correctly?

or other tips?

 

Answers

  • pschlunderpschlunder Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, RMResearcher, Member Posts: 96 RM Research

    Hi @kayman,

    since `curl` uses the current path as the destination (if not specified differently) it might already work. But don't expect the downloaded file at the output port of the Execute Script Operator. Try Execute Script with `pwd` to output the current path of the working directory the process is executed in and check manually if the downloaded file exists there.

     

    If this doesn't work, please try executing your command without the `-s` option and post the log here.

     

    Regards,

    Philipp

  • kaymankayman Member Posts: 662 Unicorn

    Hi @pschlunder, thanks already for looking into this.

     

    Unfortunatly ommiting the -s variable does not make much of a difference. I am aware there will be no output, the basic idea is just to fire the job in the background so I do not have to wait for my next cron cycle to take care of the load. I'm just abusing rapidminer as a trigger :-)

     

    Attached the log, bit it does not seem to tell a lot. It just confirms there was no problem noticed, and it ended as soon as it started, basically doing nothing with the actual command. Running the same command directly in my cmd shell works as expected.

     

    If I play around with the command, like entering an incorrect path or so it does return an error, but it looks like the --data-binary part is happily ignored. Adding, removing or using an incorrect filename for the --data-binary part is making no difference at all.

     

    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    INFO: Process context is: Macros: [_ra_user : admin, _ra_jobID : 9933]; Input: []; Output: []
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINE: Initialising process setup.
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    INFO: No filename given for result file, using stdout for logging results!
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINE: Process initialised.
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINE: Defining context macro: _ra_user = admin.
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINE: Defining context macro: _ra_jobID = 9933.
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    INFO: Process //_LOCAL/***/JSON2ES starts
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINE: Process:
    Process[0] (Process)
    subprocess 'Main Process'
    +- Execute Program[0] (Execute Program)
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINE: Starting application 1 of operator Process
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINER: Process called 1st time with input:
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINE: Executing subprocess Process.Main Process. Execution order is: [Execute Program (Execute Program)]
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINE: Starting application 1 of operator Execute Program
    Feb 06, 2018 1:40:28 PM <unknown> <unknown>
    FINER: Execute Program called 1st time with input:
    in-/-
    through 1-/-
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    INFO: Program exited succesfully.
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    WARNING: com.rapidminer.operator.CommandLineOperator.errors_occurred
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    FINE: Completed application 1 of operator Execute Program
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    FINER: Execute Program returned with output:
    out-/-
    err-/-
    through 1-/-
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    FINEST: Execute Program: execution time was 270 ms
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    FINE: Completed application 1 of operator Process
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    FINER: Process returned with output:
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    FINEST: Process: execution time was 271 ms
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    FINE: Process:
    Process[1] (Process)
    subprocess 'Main Process'
    +- Execute Program[1] (Execute Program)
    Feb 06, 2018 1:40:29 PM <unknown> <unknown>
    INFO: Process //_LOCAL/***/JSON2ES finished successfully after 0 s
  • SGolbertSGolbert RapidMiner Certified Analyst, Member Posts: 344 Unicorn

    Could it be that you need root-level access to run the script correctly? Cron-jobs are run by root, while RM Server is usually run by an user account.

    sgenzer
  • kaymankayman Member Posts: 662 Unicorn

    Good point...

    Not sure tbh, if I run the script as is (so apart from the cron level) I can run it without sudo with my user account so I'd expect RM server to be able to run the same. 

  • SGolbertSGolbert RapidMiner Certified Analyst, Member Posts: 344 Unicorn

    We discussed it yerterday with Philipp, and we have doubts about the "@my_file.json". Have you tried using the full path for the file (i.e. "/home/user_name/my_file.json")?

  • kaymankayman Member Posts: 662 Unicorn

    Yeah, tried that one also. If I deliberatly mistype the actual path the script throws an error, it just seems to ignore the existence of the json file. Even if I mistype that one it simply will say there is no problem. So wheter the json file is there or not does not make any difference. Looks like the final step in the process, taking the json and loading it to Elastic is simply not starting and it doesn't cause a problem.

Sign In or Register to comment.