Cannot retrieve rmhdf5table data

Hyperrick

Hello everyone,

for a university project we have downloaded data with the operator "Crawl Web" and now we want to store it in the repository. This works wonderfully. Unfortunately I can't retrieve this data in my subsequent RM process. The following error message always appears:

I have already tried the following:

- New installation of RM 9.7

- RM 9.8 BETA (same problem)

- Repository location changed

My system:

- Win 10 X64
- RM Studio 9.7 (latest)
- all admin privileges

Unfortunately nothing leads to success. I have attached my process to store the data here, maybe you can help me. As mentioned the code to store works. I can't retrieve the .rmhdf5table-file (also attached).

Kind regards,

Patrick

<?xml version="1.0" encoding="UTF-8"?><process version="9.7.002">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="9.7.002" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="read_csv" compatibility="9.7.002" expanded="true" height="68" name="Read CSV" width="90" x="45" y="85">
        <parameter key="csv_file" value="C:/Users/patri/Dropbox/Team Asia/asia.csv"/>
        <parameter key="column_separators" value=";"/>
        <parameter key="trim_lines" value="false"/>
        <parameter key="use_quotes" value="true"/>
        <parameter key="quotes_character" value="&quot;"/>
        <parameter key="escape_character" value="\"/>
        <parameter key="skip_comments" value="false"/>
        <parameter key="comment_characters" value="#"/>
        <parameter key="starting_row" value="1"/>
        <parameter key="parse_numbers" value="true"/>
        <parameter key="decimal_character" value="."/>
        <parameter key="grouped_digits" value="false"/>
        <parameter key="grouping_character" value=","/>
        <parameter key="infinity_representation" value=""/>
        <parameter key="date_format" value=""/>
        <parameter key="first_row_as_names" value="true"/>
        <list key="annotations"/>
        <parameter key="time_zone" value="SYSTEM"/>
        <parameter key="locale" value="English (United States)"/>
        <parameter key="encoding" value="UTF-8"/>
        <parameter key="read_all_values_as_polynominal" value="false"/>
        <list key="data_set_meta_data_information"/>
        <parameter key="read_not_matching_values_as_missings" value="true"/>
        <parameter key="datamanagement" value="double_array"/>
        <parameter key="data_management" value="auto"/>
      </operator>
      <operator activated="true" class="filter_example_range" compatibility="9.7.002" expanded="true" height="82" name="Filter Example Range" width="90" x="179" y="85">
        <parameter key="first_example" value="1"/>
        <parameter key="last_example" value="1"/>
        <parameter key="invert_filter" value="false"/>
      </operator>
      <operator activated="true" class="web:retrieve_webpages" compatibility="9.3.001" expanded="true" height="68" name="Get Pages" width="90" x="313" y="85">
        <parameter key="link_attribute" value="roughguide link"/>
        <parameter key="page_attribute" value="content"/>
        <parameter key="random_user_agent" value="false"/>
        <parameter key="connection_timeout" value="10000"/>
        <parameter key="read_timeout" value="10000"/>
        <parameter key="follow_redirects" value="false"/>
        <parameter key="accept_cookies" value="none"/>
        <parameter key="cookie_scope" value="global"/>
        <parameter key="request_method" value="GET"/>
        <parameter key="delay" value="none"/>
        <parameter key="delay_amount" value="1000"/>
        <parameter key="min_delay_amount" value="0"/>
        <parameter key="max_delay_amount" value="1000"/>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="9.7.002" expanded="true" height="82" name="Select Attributes" width="90" x="447" y="85">
        <parameter key="attribute_filter_type" value="subset"/>
        <parameter key="attribute" value="content"/>
        <parameter key="attributes" value="content|continent|place|country|roughguide link"/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="attribute_value"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="time"/>
        <parameter key="block_type" value="attribute_block"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_matrix_row_start"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
      </operator>
      <operator activated="true" class="set_role" compatibility="9.7.002" expanded="true" height="82" name="Set Role" width="90" x="581" y="85">
        <parameter key="attribute_name" value="roughguide link"/>
        <parameter key="target_role" value="id"/>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="store" compatibility="9.7.002" expanded="true" height="68" name="Store" width="90" x="715" y="85">
        <parameter key="repository_entry" value="../data/raw_extract_asia"/>
      </operator>
      <connect from_op="Read CSV" from_port="output" to_op="Filter Example Range" to_port="example set input"/>
      <connect from_op="Filter Example Range" from_port="example set output" to_op="Get Pages" to_port="Example Set"/>
      <connect from_op="Get Pages" from_port="Example Set" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Set Role" to_port="example set input"/>
      <connect from_op="Set Role" from_port="example set output" to_op="Store" to_port="input"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
    </process>
  </operator>
</process>

MartinLiebig

Hi @Hyperrick ,

thank you for reporting this. Our development team looked into this. hdf5table is our new file format. It is indeed the case that very long string objects like you have them cause some trouble.We are working on a fix for it.

Generally speaking we would recommend to use a database system or something for large files. As a workaround you should be able to use Data to Documents and store the object as a Document object. After retrieving it you can get it back to a data set using Documents to Data.

Best,

Martin

Hyperrick

Hello Martin,

thank you! That indeed fixed it. Thank you! I used your workaround-method.

Patrick