Rapidminer OSX app should have "-XX:-MaxFDLimit" by default

aruberutouaruberutou Member Posts: 23 Contributor II
Hello,

Thanks for producing such a wonderful tool. It has helped my work tremendously.

One minor niggle, however, frustrated me for a quite a bit and I wanted to share my work-around.

I have been using SentiWordNet on an OSX installation, and it tended to die once it hit a certain number of files. Now, I am not sure if its a coding issue (i.e., should it really be opening so many files at once), but it easily fixed by adding "-XX:-MaxFDLimit" to the RapidMiner-Studio-App.sh file, at the appropriate places, as shown below:

# launch RapidMiner Studio
if [ $# -gt 0 ]; then
eval \"$JAVA_HOME\"/bin/java $JVM_OPTIONS -XX:-MaxFDLimit -cp \"${RM_CLASSPATH}\" com.rapidminer.launcher.GUILauncher \"$@\"
else
eval \"$JAVA_HOME\"/bin/java $JVM_OPTIONS -XX:-MaxFDLimit -cp \"${RM_CLASSPATH}\" com.rapidminer.launcher.GUILauncher

Again, I am not sure if it is this way by design, but maybe the RM team should consider including this option by default in the OSX app. The Windows version doesn't seem to suffer from this limitation.

Thanks,

Answers

  • Marco_BoeckMarco_Boeck Administrator, Moderator, Employee, Member, University Professor Posts: 1,993 RM Engineering
    Hi,

    thanks for bringing this to our attention! This looks more like a bug in the SentiWordNet extension to me, though. I have created an internal ticket for it.
    We won't globally disable the JVM open file limit of a little more than 10.000 files because that could potentially do more harm than good.

    Regards,
    Marco
  • aruberutouaruberutou Member Posts: 23 Contributor II
    Hi,

    Thanks for the quick follow-up. I haven't noticed any issues since disabling the limit; however, I am sure it is probably there for a reason. :)

  • weeweyweewey Member Posts: 5 Contributor II
    I have this problem as well. RM always give me an error when doing sentiment analysis to more than 200 over tweets.

    I tried the solution that aruberutou mentioned but I'm still facing the problem. ://   ??? ???

    This is the error message:
    "Error reading /usr/local/WordNet- 3.0/dict:/usr/local/WordNet-3.0/dict/index.verb (Too many open files)"
  • aruberutouaruberutou Member Posts: 23 Contributor II
    Hi, weewey,

    Sorry for the late response. In order to resolve this, I also had to increase the respective limits in OSX as well.

    Please refer to the following websites for details:

    http://blog.mact.me/2014/10/22/yosemite-upgrade-changes-open-file-limit
    http://superuser.com/questions/827984/open-files-limit-does-not-work-as-before-in-osx-yosemite/828010#828010

  • weeweyweewey Member Posts: 5 Contributor II
    @aruberutou

    Awesome! Thanks alot!! I got it fixed already.

  • weeweyweewey Member Posts: 5 Contributor II
    @aruberutou

    I realise that rapidminer server result in the same problem too.

    Do you know how to resolve it?

    Appreciate your help
  • Marco_BoeckMarco_Boeck Administrator, Moderator, Employee, Member, University Professor Posts: 1,993 RM Engineering
    Hi,

    depending on your OS, it's either the standalone.conf (Linux/Mac) or the standalone.conf.bat file which you can find in the SERVER/bin folder. Look for a line starting with "JAVA_OPTS=" and add your parameters to the existing options.

    Regards,
    Marco
  • rachel_lomaskyrachel_lomasky Member Posts: 52 Guru

    The problem is that you have put the "Open WordNet Dictionary" operator inside of the "Process Documents" operator.  You should open it outside, and then use a remember with the type "WordNet Dictionary."

    <?xml version="1.0" encoding="UTF-8"?><process version="7.4.000">
    <operator activated="true" class="wordnet:open_wordnet_dictionary" compatibility="5.3.000" expanded="true" height="68" name="Open WordNet Dictionary" width="90" x="380" y="340">
    <parameter key="resource_type" value="directory"/>
    <parameter key="directory" value="%{wordnet_directory}"/>
    </operator>
    </process>
    <?xml version="1.0" encoding="UTF-8"?><process version="7.4.000">
    <operator activated="true" class="remember" compatibility="7.4.000" expanded="true" height="68" name="Remember" width="90" x="581" y="340">
    <parameter key="name" value="wordnet"/>
    <parameter key="io_object" value="WordnetDictionary"/>
    <parameter key="store_which" value="1"/>
    <parameter key="remove_from_process" value="true"/>
    </operator>
    </process>

     

    And then you can use a "Recall" inside the "Process Documents":

    <?xml version="1.0" encoding="UTF-8"?><process version="7.4.000">
    <operator activated="true" class="recall" compatibility="7.4.000" expanded="true" height="68" name="Recall" width="90" x="313" y="238">
    <parameter key="name" value="wordnet"/>
    <parameter key="io_object" value="WordnetDictionary"/>
    <parameter key="remove_from_store" value="false"/>
    </operator>
    </process>
    <?xml version="1.0" encoding="UTF-8"?><process version="7.4.000">
    <operator activated="true" class="wordnet:find_hypernym_wordnet" compatibility="5.3.000" expanded="true" height="82" name="Find Hypernyms (WordNet)" width="90" x="514" y="34">
    <parameter key="use_prefix" value="true"/>
    <parameter key="synset_word_prefix" value="hyper:"/>
    <parameter key="maximum_recursion_depth" value="1"/>
    <parameter key="multiple_meanings_per_word_policy" value="Take only first meaning"/>
    <parameter key="multiple_synsets_policy" value="Take all synsets per meaning"/>
    <parameter key="multiple_synset_words_policy" value="Take only first synset word"/>
    <parameter key="concatenation" value="Concatenate result per synset"/>
    <parameter key="keep_original_tokens" value="true"/>
    <parameter key="keep_unmatched_tokens" value="false"/>
    <parameter key="take_ID_instead_of_words" value="false"/>
    <parameter key="work_on_type_noun" value="true"/>
    <parameter key="work_on_type_verb" value="true"/>
    <parameter key="work_on_type_adjective" value="true"/>
    <parameter key="work_on_type_adverb" value="true"/>
    </operator>
    </process>

Sign In or Register to comment.