how to properly connect to local mongo DB?

kaymankayman Member Posts: 662 Unicorn
edited November 2018 in Help

Hi there, for testing purposes I am running following setup :

 

-> Rapidminer Studio

-> RapidMiner Server (on same device as Studio)

-> Mongo DB (localhost)

 

When using Studio I can access Mongo without issues using local repository and using the server repository (within Studio). However, when I want to run the process on the server directly it fails and I get a 'database misconfiguration error'

 

Any idea what I might be doing wrong, is there any specific thing I need to do to be able to call Mongo when using the server ?

Tagged:

Best Answer

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist
    Solution Accepted

    Hi all,

     

    the latest RapidMiner update v7.6 also provides an updated version of the NoSQL Extension.

    After updating the extension on your local machine and on RapidMiner Server the connectivity issue no longer occurs.

     

    Best,

    Edin

Answers

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,503 RM Data Scientist

    Dear Kayman,

    you need to setup the connection on the server side. This connection is then passed to studio and you can use it there (also locally). There is an example for solr on the docs page: http://docs.rapidminer.com/server/administration/creating-connections/creating-other-conns.html - Mongo works pretty similar.

     

    ~Martin

    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • kaymankayman Member Posts: 662 Unicorn

    That's actually what I tried.

     

    The problem is that when I test the connection directly on the server (connect -> test connection parameters) I get a nice green tick, but when I use a process that uses the same conenction I get this :
     


    com.rapidminer.operator.UserError: The database configuration 'mongo_local_data' is invalid.

     

    I've also gave access right to the whole user group to ensure I didn't miss something there.

    So somewhere I am missing something. the error message is not really helping in understanding where I should do some troubleshooting. 

  • kaymankayman Member Posts: 662 Unicorn

    Anybody that was ever able to get local mongo working? I am not able to get it running.

     

    Testing shows ok, executing shows 'the database configuration is invalid'

    Running the same from desktop (while on server instance) works fine also, it only fails when executed directly on server.

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    Hi kayman,

     

    I have a local instance running which now works fine but I remember that I ran into similar issues. Can you please help me narrowing down the problem so I can try to reproduce the connection problem:

    1. Do you use authentication for the MongoDB connection? If yes - do you use the root user or did you create a special user?
    2. Did you name the connection you created from within Studio exactly the same as the one you created for Server?

    Best regards,

    Edin

     

  • kaymankayman Member Posts: 662 Unicorn

    Hi Edin,

     

    I use no authentication, and basically I keep all out of the box settings.

     

    So I'm using instance host = localhost, instance port is 27017.

     

    I'm having the same setup for my server repo, and as mentioned running the flow from withing desktop (but using the server instances) works fine, running it directly on the server fails. Testing the connection directly on the server gives no error.

     

    Hope you can use this to do some troubleshooting, it's kind of a holdback at the moment for me.

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    Hi kayman,

     

    after some investigation I hopefully found a solution to your issue.

    1. Login as admin on your RapidMiner Server webinterface - not within RapidMiner Studio
    2. Delete the existing connection
    3. Recreate the connection
    4. Grant permissions

    It seems that creating a MongoDB connection from withing Studio is a.t.m. not correctly populated to the server.

     

    Hope this works,

    Edin

  • kaymankayman Member Posts: 662 Unicorn

    Nope :-(
    Same problems.

     

    Steps taken : Create a complete new connection on server using the webinterface, as admin.

    I granted access, tested it and was confirmed ok. The test request was acknowledged by Mongo also so communication was indeed correct.

    Next I refreshed the connections in desktop and newly created one appears correctly. I created a small process that stores a few examples into Mongo.

    Running the process locally works fine, the data is added to Mongo as expected.

     

    Running the same process on server gives me 'com.rapidminer.operator.UserError : The database configuration 'mongo_test' is invalid.

     

    I've tried running the process both as admin and local user, but no difference. Both server and Desktop are 7.3

     

     

  • kaymankayman Member Posts: 662 Unicorn

    Quick update,

     

    I have the exact same behaviour on an additional test machine running Ubuntu. On this machine I have full admin rights and the same issue appears.

     

    Installing connection on server (through browser), direct connection is working but using the connection through a process fails giving exactly the same error message.

     

    Seems therefore using Mongo as localhost is not working (version 7.3)

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    Hi kayman,

     

    I did the exact same steps as you described and everything works fine on my machine running MongoDB on localhost.

    I am using RM version 7.3.001 and MongoDB version 3.4.1 (all on Windows).

     

    Sorry I could not help :(

     

    Best,

    Edin

  • kaymankayman Member Posts: 662 Unicorn

    Weird, it isn't really rocket science so this is giving me a serious headache as this is a potential showstopper for us.

     

    Did you try running a real process storing data, or did you just test the connection itself? 

    The connection works for me also, it just gives a problem when I actually want to use it inside of a process.

     

    I have it on 2 seperate machines, one running on windows and another running linux, only difference is that I am running mongo 3.2 on both instead of 3.4

    I'll try if I can upgrade to 3.4 and hope that makes a difference, otherwise I'm stuck indeed.

     

    This is my test script :

     

    <process version="7.3.001">
    <context>
    <input/>
    <output/>
    <macros/>
    </context>
    <operator activated="true" class="process" compatibility="7.3.001" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
    <operator activated="true" class="retrieve" compatibility="7.3.001" expanded="true" height="68" name="Retrieve sample" width="90" x="45" y="34">
    <parameter key="repository_entry" value="sampledata"/>
    </operator>
    <operator activated="true" class="text:data_to_json" compatibility="7.4.001" expanded="true" height="82" name="Data To JSON (2)" width="90" x="246" y="34">
    <parameter key="ignore_arrays" value="false"/>
    <parameter key="generate_array" value="false"/>
    <parameter key="include_missing_values" value="false"/>
    </operator>
    <operator activated="true" class="nosql:mongodb_document_writer" compatibility="7.2.000" expanded="true" height="82" name="Write MongoDB (2)" width="90" x="380" y="34">
    <parameter key="mongodb_instance" value="mng_Reviews_models"/>
    <parameter key="write_concern_" value="Acknowledged"/>
    <parameter key="write_concern" value="Acknowledged"/>
    <parameter key="collection" value="sample"/>
    <parameter key="skip_invalid_documents" value="true"/>
    </operator>
    <connect from_op="Retrieve amz_de_LG_tv" from_port="output" to_op="Data To JSON (2)" to_port="example set 1"/>
    <connect from_op="Data To JSON (2)" from_port="documents" to_op="Write MongoDB (2)" to_port="documents 1"/>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_result 1" spacing="0"/>
    </process>
    </operator>
    </process>

    Screenshot from 2017-02-15 14-36-12.pngScreenshot from 2017-02-15 14-39-12.png

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    I used your process (with the Iris data set as source), stored it in the admin home folder and also renamed my MongoDB connection.

    No error occurs - regardless of the way I execute the process.

    One thing that pops into my mind.

    What version of the NoSQL extension are you running on Server & Studio? Are those different?

     

    Best,

    Edin

  • kaymankayman Member Posts: 662 Unicorn

    Nope, both are 7.3.0 :-(

     

    Could also be something I did wrong with the Mongo installation, though as far as I recall that was also just a straight forward process and it seems to be working for any of my other applications using it (and with RM desktop)

     

    Could it be related to the Mongo dbpath or so? Not that I see anything wrong with that, for my windows setup that is located on C:\data\db si that should be accessible by the server also I assume?

  • hchahcha Member Posts: 1 Contributor I

    I have exactly same issue with Rapidminer server, desktop and NoSQL extenstion in version 7.4. Server runs in Ubuntu 14.04. Mongo is 3.4

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    The standard installation procedure, as well as the MongoDB path, should not be the issue - I used it to.

    Did you try to store and run the process not in a user home folder but instead in a generally available folder (e.g. groups)?

    Else, I am running out of ideas :(

     

    In the meantime I try to get some information from our development team.

     

    Best regards,

    Edin

  • kaymankayman Member Posts: 662 Unicorn

    I can give it a try in a different folder but don't expect much tbh.

     

    Could it be related to activation code? Mine is already a fairly old one, before noSQL was an option. Just trying to put additioanl ideas on the table...

     

    [UPDATE] Installing the process in another folder (like groups) did not make any difference. The problem remains the same.

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    The activation code is not the problem.

     

    @kayman and @hcha: Is it possible that you send me your server.log (located in InstallationDirectory/standalone/log/) of a day when the connection problem occurred (by PM)?

    It helps our developers to further narrow down the issue.

     

    Best regards,

    Edin

  • yunior_betancouyunior_betancou Member Posts: 1 Contributor I

    Hi @Edin_Klapic

     

    I have exactly the same issue listed in the thread.

    I am able to execute mongo reads from studio but it fails on the server. I have tried everythin listed in the thread with no avail.
    Below is the full server log for the process execution. One interesting part is the warning that reads: 

     

    10:25:16,583 WARNING [com.rapidminer.server.tools.config.ServerConfigurationManager] (scheduledprocess_1491747916577) User null is not allowed to view configurationEntry: Access denied for entry Localhost of mongodb_instance

    It seems the server configuration manager doesn't have a user thus it can't read the connection (my connections is called "Localhost").

     

    Is there source code for the noSQL connector open and if so where is it hosted to take a look at what is going on?

     

    Full log:

    10:25:15,695 INFO [de.rapidanalytics.web.tools.ProcessServiceDelegate] (http-/0.0.0.0:8080-7) Scheduled process execution /home/admin/processes/Retrieve Data with parameter ID 22. Will fire first at Sun Apr 09 10:25:15 EDT 2017.
    10:25:16,577 INFO [de.rapidanalytics.ejb.ProcessJobEJBImpl] (EJB default - 28) Job /home/admin/processes/Retrieve Data was assigned id 27
    10:25:16,577 INFO [de.rapidanalytics.execution.ProcessExecutionDispatcher] (scheduledprocess_1491747916577) Dispatching job 27 to queue DEFAULT. (SimpleProcessExecutor)
    10:25:16,577 INFO [de.rapidanalytics.execution.ProcessExecutionDispatcher] (scheduledprocess_1491747916577) Starting process /home/admin/processes/Retrieve Data in queue DEFAULT.
    10:25:16,580 INFO [com.rapidminer.Process.106] (scheduledprocess_1491747916577) Process context is: Macros: [_ra_user : admin, _ra_jobID : 27]; Input: []; Output: []
    10:25:16,580 INFO [com.rapidminer.Process.106] (scheduledprocess_1491747916577) No filename given for result file, using stdout for logging results!
    10:25:16,580 INFO [com.rapidminer.Process.106] (scheduledprocess_1491747916577) Process //_LOCAL/home/admin/processes/Retrieve Data starts
    10:25:16,583 WARNING [com.rapidminer.server.tools.config.ServerConfigurationManager] (scheduledprocess_1491747916577) User null is not allowed to view configurationEntry: Access denied for entry Localhost of mongodb_instance
    10:25:16,583 SEVERE [com.rapidminer.Process.106] (scheduledprocess_1491747916577) Process failed: com.rapidminer.operator.UserError: The database configuration 'Localhost' is invalid.: com.rapidminer.operator.UserError: The database configuration 'Localhost' is invalid.
    at com.rapidminer.extension.nosql.operator.mongodb.MongoDBConnector.doWork(MongoDBConnector.java:169)
    at com.rapidminer.operator.Operator.execute(Operator.java:1005) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:77) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:812) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:807) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_51]
    at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:807) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:428) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.operator.Operator.execute(Operator.java:1005) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.Process.run(Process.java:1205) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.Process.run(Process.java:1101) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.Process.run(Process.java:1054) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.Process.run(Process.java:1049) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at de.rapidanalytics.execution.AbstractProcessExecutor.runProcessNow(AbstractProcessExecutor.java:91) [rapidminer-server-ejb.jar:]
    at de.rapidanalytics.execution.SimpleProcessExecutor.execute(SimpleProcessExecutor.java:31) [rapidminer-server-ejb.jar:]
    at de.rapidanalytics.execution.ProcessExecutionDispatcher$ProcessRunnable.run(ProcessExecutionDispatcher.java:89) [rapidminer-server-ejb.jar:]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_51]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_51]
    Caused by: com.rapidminer.tools.config.ConfigurationException: No such configured object of name Localhost of mongodb_instance
    at com.rapidminer.server.tools.config.ServerConfigurationManager.logAccessViolationAndThrow(ServerConfigurationManager.java:102) [rapidminer-server-ejb.jar:]
    at com.rapidminer.server.tools.config.ServerConfigurationManager.checkAccess(ServerConfigurationManager.java:97) [rapidminer-server-ejb.jar:]
    at com.rapidminer.tools.config.ConfigurationManager.lookup(ConfigurationManager.java:340) [rapidminer-studio-core-7.4.0.jar:7.4.0]
    at com.rapidminer.extension.nosql.operator.mongodb.MongoDBConnector.doWork(MongoDBConnector.java:161)
    ... 18 more
    10:25:16,584 SEVERE [com.rapidminer.Process.106] (scheduledprocess_1491747916577) Here: Process[1] (Process)
    subprocess 'Main Process'
    ==> +- Read MongoDB[1] (Read MongoDB)
    +- JSON To Data[0] (JSON To Data)
    +- Select Attributes[0] (Select Attributes)
    +- Rename[0] (Rename)
    +- Generate Concatenation[0] (Generate Concatenation)
    10:25:16,584 INFO [de.rapidanalytics.execution.ProcessExecutionDispatcher] (scheduledprocess_1491747916577) Completed execution of process /home/admin/processes/Retrieve Data
    10:25:16,586 INFO [de.rapidanalytics.ejb.ProcessJobEJBImpl] (scheduledprocess_1491747916577) Marking process 27 as completed with exception com.rapidminer.operator.UserError: The database configuration 'Localhost' is invalid.
    10:25:16,595 INFO [de.rapidanalytics.execution.LogRecordCollector] (scheduledprocess_1491747916577) Saving records now
    10:25:16,617 INFO [de.rapidanalytics.execution.LogRecordCollector] (scheduledprocess_1491747916577) Completed saving records

     

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    Hi yunior_betancou,

     

    thank you for sharing the log file. Especially the User null part is interesting.

    We are investigating the problem and hope to be able to fix it soon.

     

    Regarding your other question:

    The noSQL extension is not publicly available.

     

    Best regards,

    Edin

     

     

     

  • sebassebas Member Posts: 7 Contributor I

    Same issue here. MongoDB localhost connection does not work on RapidMiner Server

     

    Symptoms:

    - Accessing the MongoDB directly from RapidMiner Studio works OK

    - Setting up the connection in RapidMiner Server and testing it directly from the web interface works OK

    - Running the process on RapidMiner Server generates exception : The database configuration is invalid

    - User null message found in log file just before exception trace

     

    Below the log dump

    12:12:37,452 WARNING [com.rapidminer.server.tools.config.ServerConfigurationManager] (scheduledprocess_1494331957423) User null is not allowed to view configurationEntry: Access denied for entry Mongo_Processed of mongodb_instance
    12:12:37,453 SEVERE [com.rapidminer.Process.21] (scheduledprocess_1494331957423) Process failed: com.rapidminer.operator.UserError: The database configuration 'Mongo_Processed' is invalid.: com.rapidminer.operator.UserError: The database configuration 'Mongo_Processed' is invalid.
    at com.rapidminer.extension.nosql.operator.mongodb.MongoDBConnector.doWork(MongoDBConnector.java:169)
    at com.rapidminer.operator.Operator.execute(Operator.java:1004) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:77) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:812) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.operator.ExecutionUnit$3.run(ExecutionUnit.java:807) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_121]
    at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:807) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:428) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.operator.Operator.execute(Operator.java:1004) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.Process.run(Process.java:1310) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.Process.run(Process.java:1173) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.Process.run(Process.java:1126) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.Process.run(Process.java:1121) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at de.rapidanalytics.execution.AbstractProcessExecutor.runProcessNow(AbstractProcessExecutor.java:91) [rapidminer-server-ejb.jar:]
    at de.rapidanalytics.execution.SimpleProcessExecutor.execute(SimpleProcessExecutor.java:31) [rapidminer-server-ejb.jar:]
    at de.rapidanalytics.execution.ProcessExecutionDispatcher$ProcessRunnable.run(ProcessExecutionDispatcher.java:89) [rapidminer-server-ejb.jar:]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]
    Caused by: com.rapidminer.tools.config.ConfigurationException: No such configured object of name Mongo_Processed of mongodb_instance
    at com.rapidminer.server.tools.config.ServerConfigurationManager.logAccessViolationAndThrow(ServerConfigurationManager.java:102) [rapidminer-server-ejb.jar:]
    at com.rapidminer.server.tools.config.ServerConfigurationManager.checkAccess(ServerConfigurationManager.java:97) [rapidminer-server-ejb.jar:]
    at com.rapidminer.tools.config.ConfigurationManager.lookup(ConfigurationManager.java:340) [rapidminer-studio-core-7.5.0.jar:7.5.0]
    at com.rapidminer.extension.nosql.operator.mongodb.MongoDBConnector.doWork(MongoDBConnector.java:161)
    ... 18 more
  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    Hi all,

     

    Thank you all for your valuable feedback incl. log files and stuff. I got the confirmation that we are working on this.

    I will keep you updated as soon as I have more details I can share.

     

    Best,

    Edin

  • kaymankayman Member Posts: 662 Unicorn

    Hi @Edin_Klapic, do you have some update on this? 

     

    Thanks in advance,

    K.

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 299 RM Data Scientist

    The dev team is pretty packed up right now.

    So no, unfortunately I have no update on this topic right now.

     

    Best,

    Edin

  • sebassebas Member Posts: 7 Contributor I

    I can confirm this works now

     

    Thanks !

Sign In or Register to comment.