"Web mining operator"

17900713r

Hello ,

I am new to  rapidminer and I am trying to use the cut document in the web mining operator.I would like to know how to construct a regular expression that will one select online reviews.You help will be much appreciated.I have been trying to follow the tutorials rapidminer suggested but they arent helping much.I am trying to scrap reviews from Amazon .

sgenzer

hello @17900713r welcome to the community. I don't believe we actually have a go-to resource for how to create RegEx expressions, but there are many others online. I personally use https://regexr.com/.

 

Scott

 

17900713r

Thank you for your quick reply.May I please know where to locate my whole process code.I would like to paste it here.

17900713r

Below is my XML process.

<?xml version="1.0" encoding="UTF-8"?><process version="8.1.003">
  <operator activated="true" class="concurrency:loop" compatibility="8.1.003" expanded="true" height="82" name="Loop" width="90" x="45" y="34">
    <parameter key="number_of_iterations" value="5"/>
    <parameter key="iteration_macro" value="iteration"/>
    <parameter key="reuse_results" value="false"/>
    <parameter key="enable_parallel_execution" value="true"/>
    <process expanded="true">
      <operator activated="true" class="web:process_web_modern" compatibility="7.3.000" expanded="true" height="68" name="Process Documents from Web" width="90" x="112" y="85">
        <parameter key="url" value="https://www.rottentomatoes.com/m/the_dark_knight/reviews/?page=%{iteration}"/>
        <list key="crawling_rules"/>
        <parameter key="max_crawl_depth" value="1"/>
        <parameter key="retrieve_as_html" value="false"/>
        <parameter key="enable_basic_auth" value="false"/>
        <parameter key="add_content_as_attribute" value="false"/>
        <parameter key="max_pages" value="4"/>
        <parameter key="max_page_size" value="1000"/>
        <parameter key="delay" value="200"/>
        <parameter key="max_concurrent_connections" value="100"/>
        <parameter key="max_connections_per_host" value="100"/>
        <parameter key="user_agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"/>
        <parameter key="ignore_robot_exclusion" value="true"/>
        <process expanded="true">
          <operator activated="true" class="text:cut_document" compatibility="8.1.000" expanded="true" height="68" name="Cut Document" width="90" x="179" y="34">
            <parameter key="query_type" value="XPath"/>
            <list key="string_machting_queries"/>
            <parameter key="attribute_type" value="Nominal"/>
            <list key="regular_expression_queries"/>
            <list key="regular_region_queries"/>
            <list key="xpath_queries">
              <parameter key="seg" value="//h:table[@class='table table-striped']/h:tr"/>
            </list>
            <list key="namespaces"/>
            <parameter key="ignore_CDATA" value="true"/>
            <parameter key="assume_html" value="true"/>
            <list key="index_queries"/>
            <list key="jsonpath_queries"/>
            <process expanded="true">
              <connect from_port="segment" to_port="document 1"/>
              <portSpacing port="source_segment" spacing="0"/>
              <portSpacing port="sink_document 1" spacing="0"/>
              <portSpacing port="sink_document 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="text:extract_information" compatibility="8.1.000" expanded="true" height="68" name="Extract Information" width="90" x="313" y="34">
            <parameter key="query_type" value="XPath"/>
            <list key="string_machting_queries"/>
            <parameter key="attribute_type" value="Nominal"/>
            <list key="regular_expression_queries"/>
            <list key="regular_region_queries"/>
            <list key="xpath_queries">
              <parameter key="text" value="//h:p/text"/>
            </list>
            <list key="namespaces"/>
            <parameter key="ignore_CDATA" value="true"/>
            <parameter key="assume_html" value="true"/>
            <list key="index_queries"/>
            <list key="jsonpath_queries"/>
          </operator>
          <connect from_port="document" to_op="Cut Document" to_port="document"/>
          <connect from_op="Cut Document" from_port="documents" to_op="Extract Information" to_port="document"/>
          <connect from_op="Extract Information" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Process Documents from Web" from_port="example set" to_port="output 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_output 1" spacing="0"/>
      <portSpacing port="sink_output 2" spacing="0"/>
    </process>
  </operator>
</process>
<?xml version="1.0" encoding="UTF-8"?><process version="8.1.003">
  <operator activated="true" class="append" compatibility="8.1.003" expanded="true" height="82" name="Append" width="90" x="179" y="34">
    <parameter key="datamanagement" value="double_array"/>
    <parameter key="data_management" value="auto"/>
    <parameter key="merge_type" value="all"/>
  </operator>
</process>

sgenzer

hello @17900713r - so in RapidMiner your XML is your process "code".

 

Scott