Configure Hazelcast
Hazelcast is used in ApiOmat to let the YAMBAS nodes communicate with each other in case a new node starting or going down or an application gets deployed. In detail, hazelcast takes care of the following tasks:
-
Notify other nodes on module deploy
-
Handle execution on special timer tasks (Push Messages, Cron, ...) on one node only
-
Failover after node crash
-
Caching the application object in a multinode environment (ApiOmat >= 2.4.7)
Hazelcast gets configured with its own file hazelcast.xml, which can be found at the configuration directory of the ApiOmat installation. If YAMBAS was installed manually, the config file can be provided using the following JVM argument:
-Dhazelcast.config=/path/to/hazelcast.xml
This configuration file is primary used to set up the method how cluster nodes may find and join each other. Additionally, it contains other parts like a special map behaviour:
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-basic.xsd" xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><group> <name>dev</name> <password>dev-pass17</password></group><network> <join> <multicast enabled="false"> <multicast-group>224.2.2.3</multicast-group> <multicast-port>54327</multicast-port> </multicast> <tcp-ip enabled="true"> <interface>IP address of app server 1</interface> <interface>IP address of app server 2</interface>... </tcp-ip> </join></network> <!-- configure near cache for app map; see http://docs.hazelcast.org/docs/3.5/manual/html/map-nearcache.html for more information --> <map name="AOMCache-com.apiomat.backend.model.core.Application"> </map> <map name="AOMCache-com.apiomat.backend.model.core.Customer"> </map></hazelcast>
Always use <multicast></multicast> and not <multicast/>, even with no specific configuration.
This howto will only take account of the <network> part of the configuration. To get a deeper knowledge of the other parts, please visit the original Hazelcast documentation.
Cluster
Please be aware, that in a cluster environment each node has to have the same hazelcast.xml content!
The following examples will show a proper configuration for the most common network setups. If you're using a one node setup, you can keep the hazelcast config file.
A successfull join can be checked during YAMBAS startup - the list of joined cluster nodes is printed in the logs right during startup:
Members [2] { Member [10.0.0.2]:31744 - bbe23fd9-a5ef-424b-8fdb-48b7446da79a Member [10.0.0.3]:31765 - 69713e18-22d1-4bca-bf83-019d1411a66c}
These 'Members' logs are only shown at INFO level, you have to set hazelcast logger level in your log4j.xml to INFO.
As of ApiOmat 3.3.2 the default log4j.xml already contains a logger setting for the com.hazelcast.internal.cluster.ClusterService which is already set to INFO.
Join methods
Multicast join
By default, Hazelcast uses multicast to discover other members that can form a cluster. If all cluster nodes are able to find each user using multicast, the network config should look like:
...<network> <join> <multicast enabled="true"> <multicast-group>224.2.2.3</multicast-group> <multicast-port>54327</multicast-port> </multicast> <tcp-ip enabled="false"/> </join></network>...Optionally, the multicast port and group IP can be changed.
TCP-IP Join
Under some circumstances (e.g. firewall, etc.) multicast is not possible. Then, in order to let all nodes find each other, the hazelcast configuration file has to be set up properly with all application server IPs added in the <tcp-ip> tag.
...<network> <join> <multicast enabled="false"/> <tcp-ip enabled="true"> <interface>IP address of app server 1</interface> <interface>IP address of app server 2</interface>... </tcp-ip> </join></network>...Each node of the setup must be listed in the <interface> tags.
Discovery join using Consul
In dynamic environments like docker, neither static IPs nor multicast will work to make the nodes find each other. In this case, we use a discovery service, where all nodes register themselfes and query other nodes. A consul (Kubernetes|Swarm) setup is required preliminary; the consul configuration must then be set up in the hazelcast config.
...<network> <port auto-increment="true">5701</port> <join> <multicast enabled="false"/> <tcp-ip enabled="false" /> <discovery-strategies> <discovery-strategy enabled="true" class="org.bitsofinfo.hazelcast.discovery.consul.ConsulDiscoveryStrategy"> <properties> <property name="consul-host">localhost</property> <property name="consul-port">8500</property> <property name="consul-service-name">hz-discovery-test-cluster</property> <property name="consul-healthy-only">true</property> <property name="consul-service-tags">hazelcast, test1</property> <property name="consul-discovery-delay-ms">10000</property> <property name="consul-acl-token"></property> <property name="consul-ssl-enabled">false</property> <property name="consul-ssl-server-cert-file-path"></property> <property name="consul-ssl-server-cert-base64"></property> <property name="consul-ssl-server-hostname-verify">true</property> <property name="consul-registrator">org.bitsofinfo.hazelcast.discovery.consul.LocalDiscoveryNodeRegistrator</property> <property name="consul-registrator-config"><![CDATA[ { "preferPublicAddress":false, "healthCheckProvider":"org.bitsofinfo.hazelcast.discovery.consul.ScriptHealthCheckBuilder", "healthCheckScript":"nc -z #MYIP #MYPORT", "healthCheckScriptIntervalSeconds":30, "healthCheckHttp":"http://#MYIP:80", "healthCheckHttpIntervalSeconds":30, "healthCheckTcp":"#MYIP:#MYPORT", "healthCheckTcpIntervalSeconds":30 } ]]></property> </properties> </discovery-strategy> </discovery-strategies> </join></network>...
After startup, you will see each YAMBAS cluster node registering itself in consul.
Hazelcast Configuration Tuning
In case you run into problems with the communication between the hazelcast nodes it's best practices to adjust the hazelcast.xml configuration.
Your first entry point is the offical reference manual of your hazelcast component here: https://hazelcast.org/documentation/ . Simply choose your hazelcast version and use either the navigation bar to begin your research or type specifc terms into the search bar.
E.g.: sometimes its helpful to increase the connection timeout between the nodes within the hazelcast network:
Increase Connection Timeout
The search bar of the official hazelcast documentation finally leads you to this artcle: http://docs.hazelcast.org/docs/3.8/manual/html-single/index.html#setting-connection-timeout .
The Connection timeout specifies the timeout value in milliseconds for members to accept connection requests within a hazelcast network. In the following configuration snippet the connection timeout is set to 5000 milliseconds:hazelcast.xml
...<network> ... <connection-timeout>5000</connection-timeout> ...</network>...
If your configuration still doesn't fit your system requirements you may increase the connection attempt limit following this article http://docs.hazelcast.org/docs/3.8/manual/html-single/index.html#setting-connection-attempt-limit .