Options

forgetful password

sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
edited November 2018 in Help
hi...I notice that when I am running a process where it is doing Enrich Data by Webservice many times (i.e. >1000 or so), RapidMiner "forgets" the authentication password that I told it to "remember" when I entered it the first time.  It seems sort of random when it does it, but it is rather annoying - particularly if I leave it running overnight and find out in the morning that after 4000 API calls, it now forgets the password and asks me to enter it again.

By the way, the API calls I am doing are using the new IBM Bluemix service.  It's pretty cool.

Any ideas?

Scott

Answers

  • Options
    Marco_BoeckMarco_Boeck Administrator, Moderator, Employee, Member, University Professor Posts: 1,995 RM Engineering
    Hi,

    we are talking about HTTP Basic auth, correct? If so, I highly suggest using the method below to pass the credentials instead of having to rely on the internal Studio authentication cache (which is not intended for this sort of thing): 

    https://username:password@www.example.com/webcallback?foo=bar
    The credentials are protected by SSL in case of a https call, so this is actually safe to do.

    Regards,
    Marco
  • Options
    sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    oh that very well do the trick.  Thanks very much.

    Scott
  • Options
    kludikovskykludikovsky Member Posts: 30 Maven

    is there anything special I need to consider if I'd like to do this with MailChimp?

    When I try to access MailChimps API I get an error in the brwoser as well as in Rapidminer when using the Base Auth as described here, even as this method is described in the documentation.

    The error is:

    {"type":"http://developer.mailchimp.com/documentation/mailchimp/guides/error-glossary/","title":"API Key Missing","status":401,"detail":"Your request did not include an API key.","instance":"4646a67f-8cb1-417a-914b-6fabf3539eb3"}

    The call I used 

    https://kll:686XXXXXXXXXXXXXXXXXXXXXXXXXXXXX-us16@us16.api.mailchimp.com/3.0/lists/

    But when I use the RESTED extension in Chrome where I can separately enter the user/password credentials it works as expected.

     image.png

    Any hints?


  • Options
    kludikovskykludikovsky Member Posts: 30 Maven

    See BUG-Report: https://community.rapidminer.com/t5/RapidMiner-Studio-Forum/BUG-Enrich-Data-by-Webservice-Web-Mining-Does-not-pass-request/td-p/44049/jump-to/first-unread-message 

  • Options
    sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hi @kludikovsky - so this actually works fine.  Here's a sample call (API key deleted) with the JSON unparsed:

     

    <?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="generate_data_user_specification" compatibility="7.6.001" expanded="true" height="68" name="Generate Data by User Specification" width="90" x="179" y="238">
            <list key="attribute_values"/>
            <list key="set_additional_roles"/>
          </operator>
          <operator activated="true" class="web:enrich_data_by_webservice" compatibility="7.3.000" expanded="true" height="68" name="Enrich Data by Webservice" width="90" x="313" y="238">
            <parameter key="query_type" value="Regular Expression"/>
            <list key="string_machting_queries"/>
            <list key="regular_expression_queries">
              <parameter key="foo" value=".*"/>
            </list>
            <list key="regular_region_queries"/>
            <list key="xpath_queries"/>
            <list key="namespaces"/>
            <list key="index_queries"/>
            <list key="jsonpath_queries"/>
            <parameter key="url" value="https://us7.api.mailchimp.com/schema/3.0/Root.json"/>
            <list key="request_properties">
              <parameter key="content-type" value="application/json"/>
              <parameter key="Authorization" value="apikey xxxxxxxxxxxxxxx-us7"/>
            </list>
          </operator>
          <connect from_op="Generate Data by User Specification" from_port="output" to_op="Enrich Data by Webservice" to_port="Example Set"/>
          <connect from_op="Enrich Data by Webservice" from_port="ExampleSet" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>

    Scott

     

     

     

  • Options
    kludikovskykludikovsky Member Posts: 30 Maven

    Hi @sgenzer, 

    I can repeat your process, even as there authentication.

    I just c did a copy&paste into a new canvas and got a result. So I did no even need a API-key

    But the protocol has no sign of the extra data sent:

    Paket #3
    GET /schema/3.0/Root.json HTTP/1.1
    User-Agent: Java/1.8.0_141
    Host: us7.api.mailchimp.com
    Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
    Connection: keep-alive


    Paket #4
    HTTP/1.1 200 OK
    Server: openresty
    Content-Type: application/json
    Vary: Accept-Encoding
    Date: Wed, 29 Nov 2017 16:32:35 GMT
    Content-Length: 8760
    Connection: keep-alive

    {
      "$schema": "http://json-schema.org/draft-04/schema#",
      "type": "object",
      "title": "API Root",
      "description": "This resource is nothing more than links to other resources available through the API.",
      "properties": {
        "account_id": {

    Could you try this please with /lists/. 

  • Options
    sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hello @kludikovsky - yes I see that.  I was just accessing the schema.  OK please be patient - I will investigate.  :)

     

    Scott

     

  • Options
    sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    OK got it :)

     

    <?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="web:get_webpage" compatibility="7.3.000" expanded="true" height="68" name="Get Page" width="90" x="45" y="34">
            <parameter key="url" value="https://us7.api.mailchimp.com/3.0/lists"/>
            <parameter key="user_agent" value="rapidminer"/>
            <parameter key="accept_cookies" value="all"/>
            <list key="query_parameters"/>
            <list key="request_properties">
              <parameter key="Authorization" value="apikey xxxxxxxxx-us7"/>
            </list>
            <parameter key="override_encoding" value="true"/>
            <parameter key="encoding" value="UTF-8"/>
          </operator>
          <connect from_op="Get Page" from_port="output" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>

    Scott

  • Options
    kludikovskykludikovsky Member Posts: 30 Maven

    @sgenzer confirm that it works with the "Get Page".

     

    There are the request information included as can be seen in the trace:

    GET /3.0/lists HTTP/1.1
    User-Agent: rapidminer
    Authorization: apikey 68xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us16
    Host: us16.api.mailchimp.com
    Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
    Connection: keep-alive
    Cookie: _AVESTA_ENVIRONMENT=prod


    Paket #9
    HTTP/1.1 200 OK
    Server: openresty
    Content-Type: application/json; charset=utf-8
    Vary: Accept-Encoding
    X-Request-Id: 3f6c979e-f8f4-412a-bea0-8562753f2d06
    Link: <https://us16.api.mailchimp.com/schema/3.0/Lists/Collection.json>; rel="describedBy"
    Content-Length: 17816
    Date: Wed, 29 Nov 2017 17:27:20 GMT
    Connection: keep-alive
    Set-Cookie: _AVESTA_ENVIRONMENT=prod; path=/

    {"lists":[{"id":"5c771cab77","web_id":107653,"name":

    While the same with "Enrich Data by Webservice" don't supply the information:

    GET /3.0/lists/ HTTP/1.1
    User-Agent: Java/1.8.0_141
    Host: us16.api.mailchimp.com
    Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
    Connection: keep-alive


    Paket #23
    HTTP/1.1 401 Unauthorized
    Server: openresty
    Content-Type: application/problem+json; charset=utf-8
    Content-Length: 229
    X-Request-Id: 24037518-be5a-4db7-8c7b-38c33c63e998
    Link: <https://us16.api.mailchimp.com/schema/3.0/ProblemDetailDocument.json>; rel="describedBy"
    Date: Wed, 29 Nov 2017 17:37:00 GMT
    Connection: keep-alive
    Set-Cookie: _AVESTA_ENVIRONMENT=prod; path=/

    {"type":"http://developer.mailchimp.com/documentation/mailchimp/guides/error-glossary/","title":"API Key Missing","status":401,"detail":"Your request did not include an API key.","instance":"24037518-be5a-4db7-8c7b-38c33c63e998"}

    So this would conclude that the request properties are not passed on. Which possibly explains some other malfunction I have read on the web.

    Thxs Scott.

     

  • Options
    sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    yes exactly. Unfortunately the "Enrich Data via Webservice" has a lot of trouble with authentication - it's rather out-of-date at this point. I am finding that I see more success with "Get Page", and if that does not work, then "Execute Program" with some cURL statements or calling a shell script.

     

    FWIW you may see a new extension dedicated completely to consuming the most popular APIs natively in RapidMiner (like we do already for Twitter) sometime in the future...  :)

     

    Scott

     

     

Sign In or Register to comment.