Hive driver is not allowed to run

2131012221310122 Member Posts: 40  Guru
edited December 2018 in Help

I wanted to connect the impala db and the driver is forbidden.

Why?

 

This is my code, trying to get test a connection, but it got the error message:"no suitable driver for ***"。

I checked this code is other project before and it went well. 

3.png

 

 

So I debuged it and found the error line, it's in the RapidMiner code that a class named "DriverAdapter" cannot be found:

1.png2.jpg

 

Does anybody know how to fix it?

 

Tagged:

Best Answer

  • mborbelymborbely Posts: 14   RM Team Member
    Solution Accepted

    Hi,

     

    Sorry, I was a bit too hasty about my suggestion. Since this is a project dependency, you can only use it if you have it as a submodule in your project. But for that you'll need the source code, which is not possible, since this module is not open-source. You could provide this dep in a way which would only require this module as a Maven artifact, but unfortunately not even the artifact is public.

    You can however try some alternative solutions. One of things this extensionconfig does is that it creates a Plugin-Dependencies entry in your META-INF/MANIFEST.MF file, which tells Studio that this extension will require some other extension for functioning. Therefore you could try adding an entry like this to your manifest file: "Plugin-Dependencies: rmx_jdbc_connectors[8.1.0];" Note that this can only work because in this particular case you don't really need the source code of this module at compile-time, only at runtime. And I'm not really sure that this is even going to work, but it's worth a try.

     

    The other thing is that Studio only tries to load this driver adapter class for this particular jdbc prefix ("jdbc:hive2://") because you registered an entry for it in Manage Database Drivers. You could also try removing that entry, in which case org.apache.hive.jdbc.HiveDriver will be used directly instead of going to the DriverAdapter first. Problem is that your extension cannot provide the necessary driver classes (as a fat jar for example), because they are only loaded dynamically by the extension class loader, and are not on the classpath of the application, and DriverManager cannot handle this. https://stackoverflow.com/questions/14478870/dynamically-load-the-jdbc-driver This is basically the reason this DriverAdapter magic is in place. So you'll have to put the jdbc jar into the lib folder of your Studio home. Of course if you want to end up with something that's reusable by other people, this is not the best solution.

     

    Cheers,

    Máté

     

Answers

  • mborbelymborbely Employee, Member Posts: 14   RM Team Member

    Hi,

     

    First of all please note that this question does not belong to the Radoop Forum. In the future please use the Developer Forum for such questions.

     

    Second, it seems like your extension cannot load the DriverAdapter class, which is located in the rapidminer-extension-jdbc-connectors artifact. If the classloader of your plugin cannot access it, you're probably missing the following entry from your extension config.

     

    extensionConfig {
      dependencies {
        extension project(':rapidminer-extension-jdbc-connectors')
      }
    }

     

    Cheers,

    Máté

    sgenzer
  • sgenzersgenzer Administrator, Moderator, Employee, RapidMiner Certified Analyst, Community Manager, Member, University Professor Posts: 2,127  Community Manager

    hi @mborbely - placement was done by me. I saw "Hive" in the message and immediately moved it here to Radoop. It's a valid point whether or not it belongs here or Developer. :) I can move it.


    Scott

     

  • mborbelymborbely Employee, Member Posts: 14   RM Team Member

    Thanks for your reply. I can understand why one would think this belongs here, but you don't even need to have Radoop installed to bump into this problem. So even though we do similar things in Radoop, this doesn't really belong here. Also many people might be able to answer this question who would never read the Radoop thread. Therefore I think it's better placed there, but it's not a huge problem if it stays.

     

    Cheers,

    Máté

    sgenzer
  • 2131012221310122 Member Posts: 40  Guru

    First, thank you for your answer, but I can't get the dependency.

    Do you know the reason?

    1.png2.png

     

    @mborbely@sgenzer

  • 2131012221310122 Member Posts: 40  Guru

    Thank you a lot, the website (https://stackoverflow.com/questions/14478870/dynamically-load-the-jdbc-driver ) that you provided to me is very useful!

    :smileyhappy:

     

    But I got another problem with the lib folder.

    According to the website you mentioned(https://stackoverflow.com/questions/14478870/dynamically-load-the-jdbc-driver) ,I need to point the jar file using code like this:

    URL u = new URL("jar:file:/path/to/pgjdbc2.jar!/");
    String classname = "org.postgresql.Driver";
    URLClassLoader ucl = new URLClassLoader(new URL[] { u });
    Class.forName(classname, true, ucl);
    DriverManager.getConnection("jdbc:postgresql://host/db", "user", "pw");

     

    I tried two ways to use this code.

    (1)as you suggested, I added needed jar files to the lib folder of RM Studio.

    But I realize that I can't use the absolute path in the code to generate URL parameter. Otherwise once the installation address is modified , the required jar file cannot be found.

    So how can I get the relative path of RM Studio in my extension project?

     

    (2)I chosen to add lib folder in my extension project rather than RM Studio installation folder.

    lib.png

     

    This is my code to get the relative path and the jar file.

    jarloader.png

     

    Then I tested the getConnection() method and the result was ok, I got the data from hive database.

    shim.png

     

    But when I use it as operator in RM Studio and put this code in doWork() method, it went wrong.

    First, the lib folder in my extension project is empty after generates the jar file and the location is not in the resources folder.

    whereis.pngempty.png

    lib empty.png

     

    second, this code is no longer right anymore, it returns null.

    private static URL libUrl = ClassLoader.getSystemClassLoader().getResource("lib");

     

    And what make me more confused is this result:

    com.jpg

     

    So what's the result of classpath when I use RM studio with my extension installed?

     

    (3)Finally, I used the absolute path and it went ok in RM Studio.

     

     

     

    Here is my situation:

        If I use absulte path, it will be less flexible.

        But if I use relative path, it works fine in my local extension project but get error when running as a doWork() method in RM Studio.

     

    So If I want to use relative path, what value of URL should I set?

    Thank you a lot.

     

    @mborbely

     

    mborbely
  • mborbelymborbely Employee, Member Posts: 14   RM Team Member

    Hi,

     

    You slightly misunderstood what I was trying to say about classloaders. You don't want to create a new URLClassloader yourself. My point is that loading jdbc drivers dynamically instead of leveraging Java System-Classpath classloader makes your job harder, since DriverManager cannot handle these classes properly. However RapidMiner Studio creates a separate classloader for your extension and uses it to load the classes which are bundled into your extension fat jar. Therefore if you package your Driver classes into your extension, it will effectively be loaded like in the example of the Stackoverflow link, and you'll end up with the problems mentioned there.

    However if you put it into Studio lib folder, it will be loaded by the System-Classpath classloader, and you can bypass all these problems. But whoever wants to use your solution will also have to perform this step, which is acceptable if you only intend it to be used by a handful of people, but a bit clumsy if you want to share it with a larger audience.

Sign In or Register to comment.