"Web mining operator"

17900713r17900713r Member Posts: 9 Contributor I
edited June 2019 in Help

Hello ,

I am new to  rapidminer and I am trying to use the cut document in the web mining operator.I would like to know how to construct a regular expression that will one select online reviews.You help will be much appreciated.I have been trying to follow the tutorials rapidminer suggested but they arent helping much.I am trying to scrap reviews from Amazon .

Answers

  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hello @17900713r welcome to the community. I don't believe we actually have a go-to resource for how to create RegEx expressions, but there are many others online. I personally use https://regexr.com/.

     

    Scott

     

  • 17900713r17900713r Member Posts: 9 Contributor I

    Thank you for your quick reply.May I please know where to locate my whole process code.I would like to paste it here.

  • 17900713r17900713r Member Posts: 9 Contributor I

    Below is my XML process.

    <?xml version="1.0" encoding="UTF-8"?><process version="8.1.003">
    <operator activated="true" class="concurrency:loop" compatibility="8.1.003" expanded="true" height="82" name="Loop" width="90" x="45" y="34">
    <parameter key="number_of_iterations" value="5"/>
    <parameter key="iteration_macro" value="iteration"/>
    <parameter key="reuse_results" value="false"/>
    <parameter key="enable_parallel_execution" value="true"/>
    <process expanded="true">
    <operator activated="true" class="web:process_web_modern" compatibility="7.3.000" expanded="true" height="68" name="Process Documents from Web" width="90" x="112" y="85">
    <parameter key="url" value="https://www.rottentomatoes.com/m/the_dark_knight/reviews/?page=%{iteration}"/>
    <list key="crawling_rules"/>
    <parameter key="max_crawl_depth" value="1"/>
    <parameter key="retrieve_as_html" value="false"/>
    <parameter key="enable_basic_auth" value="false"/>
    <parameter key="add_content_as_attribute" value="false"/>
    <parameter key="max_pages" value="4"/>
    <parameter key="max_page_size" value="1000"/>
    <parameter key="delay" value="200"/>
    <parameter key="max_concurrent_connections" value="100"/>
    <parameter key="max_connections_per_host" value="100"/>
    <parameter key="user_agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"/>
    <parameter key="ignore_robot_exclusion" value="true"/>
    <process expanded="true">
    <operator activated="true" class="text:cut_document" compatibility="8.1.000" expanded="true" height="68" name="Cut Document" width="90" x="179" y="34">
    <parameter key="query_type" value="XPath"/>
    <list key="string_machting_queries"/>
    <parameter key="attribute_type" value="Nominal"/>
    <list key="regular_expression_queries"/>
    <list key="regular_region_queries"/>
    <list key="xpath_queries">
    <parameter key="seg" value="//h:table[@class='table table-striped']/h:tr"/>
    </list>
    <list key="namespaces"/>
    <parameter key="ignore_CDATA" value="true"/>
    <parameter key="assume_html" value="true"/>
    <list key="index_queries"/>
    <list key="jsonpath_queries"/>
    <process expanded="true">
    <connect from_port="segment" to_port="document 1"/>
    <portSpacing port="source_segment" spacing="0"/>
    <portSpacing port="sink_document 1" spacing="0"/>
    <portSpacing port="sink_document 2" spacing="0"/>
    </process>
    </operator>
    <operator activated="true" class="text:extract_information" compatibility="8.1.000" expanded="true" height="68" name="Extract Information" width="90" x="313" y="34">
    <parameter key="query_type" value="XPath"/>
    <list key="string_machting_queries"/>
    <parameter key="attribute_type" value="Nominal"/>
    <list key="regular_expression_queries"/>
    <list key="regular_region_queries"/>
    <list key="xpath_queries">
    <parameter key="text" value="//h:p/text"/>
    </list>
    <list key="namespaces"/>
    <parameter key="ignore_CDATA" value="true"/>
    <parameter key="assume_html" value="true"/>
    <list key="index_queries"/>
    <list key="jsonpath_queries"/>
    </operator>
    <connect from_port="document" to_op="Cut Document" to_port="document"/>
    <connect from_op="Cut Document" from_port="documents" to_op="Extract Information" to_port="document"/>
    <connect from_op="Extract Information" from_port="document" to_port="document 1"/>
    <portSpacing port="source_document" spacing="0"/>
    <portSpacing port="sink_document 1" spacing="0"/>
    <portSpacing port="sink_document 2" spacing="0"/>
    </process>
    </operator>
    <connect from_op="Process Documents from Web" from_port="example set" to_port="output 1"/>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_output 1" spacing="0"/>
    <portSpacing port="sink_output 2" spacing="0"/>
    </process>
    </operator>
    </process>
    <?xml version="1.0" encoding="UTF-8"?><process version="8.1.003">
    <operator activated="true" class="append" compatibility="8.1.003" expanded="true" height="82" name="Append" width="90" x="179" y="34">
    <parameter key="datamanagement" value="double_array"/>
    <parameter key="data_management" value="auto"/>
    <parameter key="merge_type" value="all"/>
    </operator>
    </process>
  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager

    hello @17900713r - so in RapidMiner your XML is your process "code".

     

    Scott

     

Sign In or Register to comment.