Parsing out plain text from the Reuters RCV1 corpus - XPath, XML

Answers

• Options

  andk Member Posts: 21 Contributor II

  September 2011

  -push- sorry this is against all forum ethics, i know, but i can't believe that nobody here can help me. i am still stuck with the problem. i can read in the file but still i have problems with the /newsitem/text xpath because it always also gives out the tags or when i use /newsitem/text/p/node() only the first paragraph although my xml helptools like xmlspy and aquapath show that this would be the right adress to read out just the plain text.
  
  two add-on questions:
  
  1) i am sure i am not the only one here working with the reuters corpus volume 1 & 2: do you know of a way to read in the whole corpus more efficiently e.g. in a database?
  
  2) although it is just text if also 800k files, rapidminer has enormous memory problems to parse in the files on a 4gm ram machine. is that normal? due to that i had to manually split up the files into six parts with a little more than 600m each. parsing in one of this splits takes around 70 minutes. i suspect that the memory runs full and the system slows down. isn't there away to tell rapidminer to serialize the whole process .... so to read in one file extract the information i need by xpath and write the first result into a csv e.g. other software does this and seems to be much more efficient although unable to achieve what i can do with rapidminer.
  
  every help would be very very much appreciated!
  
  best regards,
  
  amdk

  0
• Options

  JEdward RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 578 Unicorn

  September 2011

  I can't help with your other 2 questions, but this seems to work for me on your test XML.
  Is it what you're after?
  
  JEdward.

  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  <process version="5.1.011">
    <context>
      <input/>
      <output/>
      <macros/>
    </context>
    <operator activated="true" class="process" compatibility="5.1.011" expanded="true" name="Process">
      <process expanded="true" height="280" width="212">
        <operator activated="true" class="read_xml" compatibility="5.1.011" expanded="true" height="60" name="Read XML" width="90" x="45" y="165">
          <parameter key="file" value="test.xml"/>
          <parameter key="xpath_for_examples" value="/newsitem/text"/>
          <enumeration key="xpaths_for_attributes">
            <parameter key="xpath_for_attribute" value="/newsitem/text"/>
          </enumeration>
          <parameter key="use_namespaces" value="false"/>
          <list key="namespaces"/>
          <parameter key="use_default_namespace" value="false"/>
          <list key="annotations"/>
          <list key="data_set_meta_data_information">
            <parameter key="0" value="text().true.text.attribute"/>
          </list>
        </operator>
        <connect from_op="Read XML" from_port="output" to_port="result 1"/>
        <portSpacing port="source_input 1" spacing="0"/>
        <portSpacing port="sink_result 1" spacing="0"/>
        <portSpacing port="sink_result 2" spacing="0"/>
      </process>
    </operator>
  </process>
  

  0
• Options

  andk Member Posts: 21 Contributor II

  September 2011

  @JEdward
  
  Thank you for your post. But it is really strange because the "Read XML" Operator seems not to exist anymore after I updated Rapidminer today in the morning. Your operator just appears as a dummy. Is this unique to my machine, my set up? Nevertheless, I tried the Read XML operator before and it doesn't work for me as I want to read in 800k xml files instead of just one. therefore i used the process documents from files operator. then the "generate extract" operator to read out the paths i need into a csv file (with the write csv operator). as i said all parts but one work like a charme. the problem is the text part as it is split up by paragraphs. and it seems as this is a problem for rapidminers xpath parser.

  0
• Options

  JEdward RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 578 Unicorn

  September 2011

  Are you able to post the XML of that part of your process? 
  I'm using what sounds like the same method on some other XML documents. 
  If I manage to get a few moments today I'll try to have a quick look at what differences there are between my XML documents & process and yours.   
  
  Thanks,
  JEdward.

  0