Enrich data from Web Service - Xpath Access

Answers

• Options

  sgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

  September 2017

  Hi @kludikovsky - so the first thing I do when trying to decode XML or JSON is first look at the array myself and see what's going on.  I do this by setting the parameter "query type" to "Regular Expression" and just adding one attribute called "foo" with a RegEx of ".*"  Then I paste it into a nice XML or JSON viewer (codebeautify.org is a decent one), and only afterwards start building my XML or JSON paths in the operator. 

   

  So back to your question...my first piece of advice is to make sure you're getting a true XPath.  From what I can see, you're getting the whole HTML from some website instead of an XML data array.  So is this what you want to do?  Or rather do you want to extract the content from the site?  If the latter, you can use "Extract Content" from the Web Mining extension or many other tools (e.g. Unescape HTML) and go from there.

   

  For additional help on XML/JSON path expressions, my go-to site is http://goessner.net/articles/JsonPath/

   

  Scott

  0
• Options

  kludikovsky Member Posts: 30 Maven

  September 2017

  hi @sgenzer,

   

  I would have used the XPath if I hadn't analysed the document before.

  I took several approaches - including the Xpath from the browser - and modified them.

  The examples should point to the same location, as long as my understanding of the XPath syntax is correct.

   

  And there is another question:

  With the Extract Content, how yould you extract attributes from an element, in this case e.g. a title="xxx"?)

  I'd like to get specific elements and not text blocks.

   

  Kurt

   

  0
• Options

  sgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

  September 2017

  hi @kludikovsky - so there are many ways to extract content from HTML block code in RapidMiner.  Extract Content is useful if you are trying to pull out the content of a web page and not something tagged like an href or whatever.  In this case, my personal preference in this situation is to parse the code using various string operations in the Generate Attributes function operator.  To get the title of that web page, I would do something like this:

   

  <?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
    <context>
      <input/>
      <output/>
      <macros/>
    </context>
    <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
      <process expanded="true">
        <operator activated="true" class="web:get_webpage" compatibility="7.3.000" expanded="true" height="68" name="Get Page" width="90" x="45" y="34">
          <parameter key="url" value="http://www.firmenabc.at/result.aspx?what=haniger+benesch+versicherungs+makler+gmbh&amp;amp;where=&amp;amp;exact=false&amp;amp;inTitleOnly=false&amp;amp;l=&amp;amp;si=0&amp;amp;iid=&amp;amp;sid=-1&amp;amp;did=&amp;amp;cc="/>
          <list key="query_parameters"/>
          <list key="request_properties"/>
        </operator>
        <operator activated="true" class="text:documents_to_data" compatibility="7.5.000" expanded="true" height="82" name="Documents to Data" width="90" x="179" y="34">
          <parameter key="text_attribute" value="text"/>
          <parameter key="add_meta_information" value="false"/>
        </operator>
        <operator activated="true" class="generate_attributes" compatibility="7.6.001" expanded="true" height="82" name="Generate Attributes" width="90" x="313" y="34">
          <list key="function_descriptions">
            <parameter key="suf" value="suffix(text,length(text)-index(text,&quot;link title=&quot;)-12)"/>
            <parameter key="repl" value="replaceAll(suf,&quot;\&quot;&quot;,&quot;&quot;)"/>
            <parameter key="title" value="prefix(repl,index(repl,&quot; &quot;))"/>
          </list>
        </operator>
        <operator activated="true" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes" width="90" x="447" y="34">
          <parameter key="attribute_filter_type" value="single"/>
          <parameter key="attribute" value="title"/>
        </operator>
        <connect from_op="Get Page" from_port="output" to_op="Documents to Data" to_port="documents 1"/>
        <connect from_op="Documents to Data" from_port="example set" to_op="Generate Attributes" to_port="example set input"/>
        <connect from_op="Generate Attributes" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
        <connect from_op="Select Attributes" from_port="example set output" to_port="result 1"/>
        <portSpacing port="source_input 1" spacing="0"/>
        <portSpacing port="sink_result 1" spacing="0"/>
        <portSpacing port="sink_result 2" spacing="0"/>
      </process>
    </operator>
  </process>
  

  There are probably a half dozen other operators to do this content extraction - from the Text Processing extension, the Web Mining extension, or just using core components like I did above.   Take your pick.

   

  Scott

  0
• Options

  SGolbert RapidMiner Certified Analyst, Member Posts: 344 Unicorn

  May 2018

  I reopen this post because I've been trying to use XPaths without success. I have simple XPath expressions that work with Scrapy, like '//h2/a/text()' but doesn't in operators like Cut Documents or Extract Information.

   

  I can find the expressions relatively easy with the Scrapy shell and my web browser (inspect element...). But if I can't use them in RM, they are useless. Any kind of guide on the matter?

  0