High Availability Deployment - ActiveMQ
I set up a cluster of Rapidminer Servers for High Availability and configured an Elastic File Storage System (EFS), where persistent-home is shared across all instances and have multiple Write-Read access. All the servers are behind a Kubernetes sevice, which is connected to a Loadbalancer. This set up works perfectly with only one server instance.
However, when spinning up multiple RM Server instances, they are not are able to boot fully and I get 502 errors when trying to access the server service through the loadbalancer. In server.log, I see the following:
<div>INFO [stdout] (ServerService Thread Pool -- 56) 2020-04-04 08:42:19.453 INFO 200 --- [read Pool -- 56] o.a.activemq.store.SharedFileLocker : Database /rapidminer-home/data/broker/activemq/localhost/KahaDB/lock is locked by another server. This broker is now in slave mode waiting a lock to be acquired<br></div>
So I proceeded to set up an AmazonMQ broker (ActiveMQ PaaS), which I am able to connect via telnet from RM server (testing with only one instance for now). I modified "/persistent-rapidminer-home/configuration/execution.properties" with my ActiveMQ endpoint but RM Server is unable to connect (the embeddedBroker is disabled). I tried both, AMQP and OpenWirte protocols.
<div>2020-04-05 15:08:12,566 WARN [org.apache.activemq.transport.failover.FailoverTransport] (ActiveMQ Transport: tcp://b-zzzzzzz-1615-4d66-zzzzz-12be334477a4-1.mq.eu-west-1.amazonaws.com/192.168.86.70:5671@50426) Transport (tcp://b-zzzzzz-1615-4d66-zzzzzz-12be334477a4-1.mq.eu-west-1.amazonaws.com:5671) failed , attempting to automatically reconnect: java.io.EOFException</div>
Now, the questions:
1) Is the first log related to ActiveMQ conflicting when there are multiple servers running together? Would the solution be setting up an external broker that interconnects all servers and job agents?
2) I tried both, AMQP and OpenWire protocols. But as you can see, RM Server is unable to connect, although I can telnet into the endpoint. Any hints?
3) Is there anything else I need to change in Rapidminer Server?
Thanks in advance,