Run Rapidminer Server 8+ process from command line

cesar_ortizcesar_ortiz Member Posts: 10 Contributor I
edited December 2018 in Help

I have up and running a Rapidminer Server 8.1. Created a process that connects to AWS EMR through Radoop. Everything works fine, in Studio and in the Server. I've already succesfully run the Studio process through the command line with "rapidminer-batch.sh" but I want to know if it is possible to run a process stored in the server repository through the command line, in this case Ubuntu.

 

Any ideas?

Thanks in advance

Tagged:

Best Answer

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist
    Solution Accepted

    Hi @cesar_ortiz,

     

    I think there is a misunderstanding here. In the REST description on swagger hub (https://app.swaggerhub.com/apis/RapidMiner/JobService/0.1.0#/jobs/createUsingPOST) you can click on Model (see screenshot).

    There you can see that the string for the parameter process should be the "base64 encoded process XML".

     

    You can i.e. use an online available base64 encoder for that.

     

    Example process just generating data:

    <?xml version="1.0" encoding="UTF-8"?><process version="8.1.003">
    <context>
    <input/>
    <output/>
    <macros/>
    </context>
    <operator activated="true" class="process" compatibility="8.1.003" expanded="true" name="Process">
    <process expanded="true">
    <operator activated="true" class="generate_data" compatibility="8.1.003" expanded="true" height="68" name="Generate Data" width="90" x="112" y="34"/>
    <connect from_op="Generate Data" from_port="output" to_port="result 1"/>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_result 1" spacing="0"/>
    <portSpacing port="sink_result 2" spacing="0"/>
    </process>
    </operator>
    </process>

    Base64 encoded:

    PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48cHJvY2VzcyB2ZXJzaW9uPSI4LjEuMDAzIj4NCiAgPGNvbnRleHQ+DQogICAgPGlucHV0Lz4NCiAgICA8b3V0cHV0Lz4NCiAgICA8bWFjcm9zLz4NCiAgPC9jb250ZXh0Pg0KICA8b3BlcmF0b3IgYWN0aXZhdGVkPSJ0cnVlIiBjbGFzcz0icHJvY2VzcyIgY29tcGF0aWJpbGl0eT0iOC4xLjAwMyIgZXhwYW5kZWQ9InRydWUiIG5hbWU9IlByb2Nlc3MiPg0KICAgIDxwcm9jZXNzIGV4cGFuZGVkPSJ0cnVlIj4NCiAgICAgIDxvcGVyYXRvciBhY3RpdmF0ZWQ9InRydWUiIGNsYXNzPSJnZW5lcmF0ZV9kYXRhIiBjb21wYXRpYmlsaXR5PSI4LjEuMDAzIiBleHBhbmRlZD0idHJ1ZSIgaGVpZ2h0PSI2OCIgbmFtZT0iR2VuZXJhdGUgRGF0YSIgd2lkdGg9IjkwIiB4PSIxMTIiIHk9IjM0Ii8+DQogICAgICA8Y29ubmVjdCBmcm9tX29wPSJHZW5lcmF0ZSBEYXRhIiBmcm9tX3BvcnQ9Im91dHB1dCIgdG9fcG9ydD0icmVzdWx0IDEiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzb3VyY2VfaW5wdXQgMSIgc3BhY2luZz0iMCIvPg0KICAgICAgPHBvcnRTcGFjaW5nIHBvcnQ9InNpbmtfcmVzdWx0IDEiIHNwYWNpbmc9IjAiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzaW5rX3Jlc3VsdCAyIiBzcGFjaW5nPSIwIi8+DQogICAgPC9wcm9jZXNzPg0KICA8L29wZXJhdG9yPg0KPC9wcm9jZXNzPg0K

    Happy Mining,

    Edin

     

    image.png

     

     

     

Answers

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,503 RM Data Scientist

    Hi,

     

    there is no CLI for the server repo. But you can use the JobService API to create a process execution: https://app.swaggerhub.com/apis/RapidMiner/JobService/0.1.0#/jobs/createUsingPOST

     

    Best,

    Martin

     

    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • cesar_ortizcesar_ortiz Member Posts: 10 Contributor I

    Thank you for your replay,

     

    I followed your adviced and checked out the REST API. I can create a JWT as the documentation suggests with:

    curl -u username:password http://localhost:8080/internal/jaxrest/tokenservice

    but when i wanna summit a job I always get code 401. (being aa.bb.cc the 100+-character long idToken i've got from the first step)

    curl -X POST "http://localhost:8080/executions/jobs" -H "accept: application/json" -H "Authorization: aa.bb.cc" -H "Content-Type: application/json" -d "{ \"queueName\": \"default\", \"process\": \"test\", \"location\": \"/test\"}"
    {"timestamp":1522187911798,"status":401,"error":"Unauthorized","message":"Unauthorized","path":"/executions/jobs"}

     

    Is the location relative to the location in the server repository?

    Do I need to make any changes in the server?

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    Hi @cesar_ortiz,

     

    Some things to check:

    1. Is the the JWT token still valid?
    2. Has the user for which you created the JWT token permissions to the Process location & the Job Agent?

    Happy Mining,

    Edin

  • cesar_ortizcesar_ortiz Member Posts: 10 Contributor I

    Well, I try to connect immediately after I create the token, unless those are valid only for a few seconds.

    The user for which I created the token is the admin, which have access to everything

     

    Here is a caption of the permissions of my process:

    Screen Shot 2018-04-09 at 1.43.14 PM.png

     

    Thank you for your response 

     

     

  • Marco_BoeckMarco_Boeck Administrator, Moderator, Employee, Member, University Professor Posts: 1,993 RM Engineering

    Hi,

     

    the "Authorization" header needs to be prefixed with "bearer ". After that, append the entire JWT token. For example, for my local test server it currenty looks like that:

    Authorization: bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJyYXBpZG1pbmVyLmNvbSIsInN1YiI6ImFkbWluIiwiZ3JwIjpbImFkbWluIiwiYWRtaW5pc3RyYXRvciIsImFuYWx5c3QiLCJleGVjdXRlIiwicmVwb3J0IGVkaXRvciIsInJlcG9ydCBtYW5hZ2VyIiwicmVwb3J0IHZpZXdlciIsInNjaGVkdWxlciIsInNlcnZpY2UiLCJ1c2VycyJdLCJhZG1pbiI6dHJ1ZSwiYXVkIjoicmFwaWRtaW5lciIsImV4cCI6MTUyMzM0ODk5OCwiaWF0IjoxNTIzMzQ4Njk4fQ.dxqOqsvgwRXIEkscYb_k9q-mrcfkpln6tgTtQXeDtek


    Regards,

    Marco

  • cesar_ortizcesar_ortiz Member Posts: 10 Contributor I

    Thanks Marco,

     

    Now with "bearer" as a prefix the processes are sent to the server and I get this as response:

     

    {"id":"824ea00b-aaa1-4d12-a521-8e8e4a66","state":"PENDING","location":"/test","ownerId":"admin","createdAt":1525735389338,"queueName":"DEFAULT"}

     

    But now I get a new error:

     

    com.rapidminer.tools.XMLException: Cannot parse document: Content is not allowed in prolog.

    Apparently is an encoding issue with the xml. Tried changing my code to this

     

    curl -X POST "http://localhost:8080/executions/jobs" -H "accept: application/json;charset=utf-8" -H "Authorization: bearer eyJhbGciOiJIUzI16IkpXVCJ9.eyJpc3MiOiJyYXFkbWluIiwiZ3JwIjpbImFkbWluIiwiYWRtaW5pc3RyYXRvciIsImFuYWx5c3QiLcG9ydCBtYW5hZ2VyIiwicmVwb3J0IHZpZXdlciIsInNjaGVkdWxlciIsInNlcnZpY2UiLCJ1c2VycyJdLCJhZG1pbiI6dHJ1ZSwiYXVkIjoicmFwaWRtaW5lciIsImV4cCI6MTUyNTczMTk0OCwiaWF0IjoxNTI1NzMxNjQ4fQ.43g_939QDu-TA-CdbxmQHV2Ow_63fw" -H "Content-Type: application/json;charset=utf-8" -d "{ \"queueName\": \"default\", \"process\": \"test\", \"location\": \"/test\"}"

    And other combinations with iso-8859-1 and utf-16 with the same luck. Also tried to change the encoding in the Rapidminer Studio.

     

    Any ideas?

  • Nils_WoehlerNils_Woehler Member Posts: 463 Maven

    Hi @cesar_ortiz,

     

    the REST API expects the process XML to be provided within the "process" parameter. Currently, you only provide "test" which is not a valid process XML :)

    The location is only needed to resolve relative paths (in case your process uses those). If you do not interact with the repository at all you can omit the location.

     

    Best,

    Nils

  • cesar_ortizcesar_ortiz Member Posts: 10 Contributor I

    But my process is named "test" and is stored in the root directory of the server repository

     

    Screen Shot 2018-05-09 at 4.24.49 PM.png

     

    By the way, this is my process. It's really simple, just a proof of concept

     

    <?xml version="1.0" encoding="UTF-8"?><process version="8.1.003">
    <context>
    <input/>
    <output/>
    <macros/>
    </context>
    <operator activated="true" class="process" compatibility="8.1.003" expanded="true" name="Process">
    <process expanded="true">
    <operator activated="true" class="radoop:radoop_nest" compatibility="8.1.001" expanded="true" height="68" name="Radoop Nest" width="90" x="45" y="34">
    <parameter key="connection" value="Radoop"/>
    <parameter key="sample_size" value="200000"/>
    <enumeration key="tables_to_reload"/>
    <process expanded="true">
    <operator activated="true" class="radoop:retrieve" compatibility="8.1.001" expanded="true" height="68" name="Retrieve" width="90" x="45" y="34">
    <parameter key="use_default_database" value="false"/>
    <parameter key="database" value="bigdata"/>
    <parameter key="table" value="table1"/>
    <parameter key="filter_clause" value="WHERE 1 = 1;"/>
    </operator>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_output 1" spacing="0"/>
    </process>
    </operator>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_result 1" spacing="0"/>
    </process>
    </operator>
    </process>
  • cesar_ortizcesar_ortiz Member Posts: 10 Contributor I

    It's alive!!!

     

    Thank you all for your help :smileyhappy:

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,503 RM Data Scientist

    Hi @cesar_ortiz,

    this is great news! 

     

    Would you mind to write a few lines on how you did it? I would love to have this documented somewhere. There is so much knowledge to be shared in the community. We would, of course, help you wherever needed and push this via our social media channels.

     

    Best,

    Martin

    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • cesar_ortizcesar_ortiz Member Posts: 10 Contributor I

    Well, to launch a server process from command line first you need to create a JWT with:

     

    curl -u username:password http://localhost:8080/internal/jaxrest/tokenservice

    Then, you encode the process xml file to base64 with some online tool or other solution, and you'll get something like this:

     

     

    JK94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48cHJvY2VzcyB2ZXJzaW9uPSI4LjEuMDAzIj4NCiAgPGNvbnRleHQ+DQogICAgPGlucHV0Lz4NCiAgICA8b3V0cHV0Lz4NCiAgICA8bWFjcm9zLz4NCiAgPC9jb250ZXh0Pg0KICA8b3BlcmF0b3IgYWN0aXZhdGVkPSJ0cnVlIiBjbGFzcz0icHJvY2VzcyIgY29tcGF0aWJpbGl0eT0iOC4xLjAwMyIgZXhwYW5kZWQ9InRydWUiIG5hbWU9IlByb2Nlc3MiPg0KICAhawDxwcm9jZXNzIGV4cGFuZGVkPSJ0cnVlIj4NCiAgICAgIDxvcGVyYXRvciBhY3RpdmF0ZWQ9InRydWUiIGNsYXNzPSJnZW5lcmF0ZV9kYXRhIiBjb21wYXRpYmlsaXR5PSI4LjEuMDAzIiBleHBhbmRlZD0idHJ1ZSIgaGVpZ2h0PSI2OCIgbmFtZT0iR2VuZXJhdGUgRGF0YSIgd2lkdGg9IjkwIiB4PSIxMTIiIHk9IjM0Ii8+DQogICAgICA8Y29ubmVjdCBmcm9tX29wPSJHZW5lcmF0ZSBEYXRhIiBmcm9tX3BvcnQ9Im91dHB1dCIgdG9fcG9ydD0icmVzdWx0IDEiLz4NCigICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzb3VyY2VfaW5wdXQgMSIgc3BhY2luZz0iMCIvPg0KICAgICAgPHBvcnRTcGFjaW5nIHBvcnQ9InNpbmtfcmVzdWx0IDEiIHNwYWNpbmc9IjAiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzaW5rX3Jlc3VsdCAyIiBzcGFjaW5nPSIwIi8+DQogICAgPC9wcm9jZXNzPg0KICA8L29wZXJhdG9yPg0KPC9wcm9jZXNzPg

    And the line to launch the process will look like this:

    curl -X POST "http://localhost:8080/executions/jobs" -H "accept: application/json;charset=utf-8" -H "Authorization: bearer eyJhbGciOiJzI16IkpXVCJ9.eyJpcMiOiJyYXFkbWluIiwiZ3JwIjpbImFkbWluIiwiYWRtaW5pcYXRvciIsImFuYWx5c3QiLcG9ydCBtYW5hZ2VyIiwicmVwb3J0IHZpZXdlciIsInNjaGVkdWxlciIsInNlcnZpY2UiLCJ1c2VycyJdLCJhZG1pbiI6dHJ1ZSwiYXVkIjoicmFwaWRtaW5lciIsImV4cCI6MTUyNTczMTk0OCwiaWF0IjoxNTI1NzMQ4fQ.43g_939QDu-TA-CdbxmQHV2Ow_63fw" -H "Content-Type: application/json;charset=utf-8" -d "{ \"queueName\": \"default\", \"process\": \"JK94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48cHJvY2VzcyB2ZXJzaW9uPSI4LjEuMDAzIj4NCiAgPGNvbnRleHQ+DQogICAgPGlucHV0Lz4NCiAgICA8b3V0cHV0Lz4NCiAgICA8bWFjcm9zLz4NCiAgPC9jb250ZXh0Pg0KICA8b3BlcmF0b3IgYWN0aXZhdGVkPSJ0cnVlIiBjbGFzcz0icHJvY2VzcyIgY29tcGF0aWJpbGl0eT0iOC4xLjAwMyIgZXhwYW5kZWQ9InRydWUiIG5hbWU9IlByb2Nlc3MiPg0KICAhawDxwcm9jZXNzIGV4cGFuZGVkPSJ0cnVlIj4NCiAgICAgIDxvcGVyYXRvciBhY3RpdmF0ZWQ9InRydWUiIGNsYXNzPSJnZW5lcmF0ZV9kYXRhIiBjb21wYXRpYmlsaXR5PSI4LjEuMDAzIiBleHBhbmRlZD0idHJ1ZSIgaGVpZ2h0PSI2OCIgbmFtZT0iR2VuZXJhdGUgRGF0YSIgd2lkdGg9IjkwIiB4PSIxMTIiIHk9IjM0Ii8+DQogICAgICA8Y29ubmVjdCBmcm9tX29wPSJHZW5lcmF0ZSBEYXRhIiBmcm9tX3BvcnQ9Im91dHB1dCIgdG9fcG9ydD0icmVzdWx0IDEiLz4NCigICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzb3VyY2VfaW5wdXQgMSIgc3BhY2luZz0iMCIvPg0KICAgICAgPHBvcnRTcGFjaW5nIHBvcnQ9InNpbmtfcmVzdWx0IDEiIHNwYWNpbmc9IjAiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzaW5rX3Jlc3VsdCAyIiBzcGFjaW5nPSIwIi8+DQogICAgPC9wcm9jZXNzPg0KICA8L29wZXJhdG9yPg0KPC9wcm9jZXNzPg\", \"location\": \"/test\"}"

     Authorization: bearer is the JWT and \"process\" is the base64 xml

     

     

Sign In or Register to comment.