. . .

Lurker (Monitoring) Installation

Introduction

The Lurker service gathers machine and program data from various nodes in the network, providing a single REST interface to access the data and query via timestamps.

images/download/attachments/61479909/monitoring_01.png images/download/attachments/61479909/monitoring_02.png

Lurker fetches measurs from the following sources:

  • calling local JAVA native methods like loadAverage or availableProcessors

  • querying local JMX data like java.lang:type=Threading

  • utilizing Jolokia, which is a JMX-HTTP bridge to access JMX values on remote hosts

Lurker can store the measurements either in memory or using MongoDB. For the latter use, please install MongoDB first.

Prerequisites

Open the following port in the firewall to other ApiOmat nodes:

  • 8089 (HTTP)

Installation of Consul is required!

Installation of AdoptOpenJDK is required when not using Docker!

System requirements

The following instance configurations are recommended.

Size

CPU

Memory

Disk

Typical Cloud Instance Types

All

2 core

4 GB RAM

300 MB

AWS: m5.large, m5.xlarge

 

 

 

 

Azure: Standard_A4_v2, Standard_A8_v2

Installation

USERNAME, PASSWORD, and TOKEN values are provided per customer from ApiOmat.

Debian/Ubuntu
# Install the repo key
wget -O - https://repo-int.apiomat.com/yambas/rest/web/Key/LIVE/apiomat-archive-keyring.asc | sudo apt-key add -
# Add the repository
echo "deb https://<USERNAME>:<PASSWORD>@repo.apiomat.com/yambas/rest/web/Repo/LIVE/deb ./" | sudo tee /etc/apt/sources.list.d/apiomat.list
 
# Update APT and install
apt-get update
apt-get install -y aom-lurker
service lurker start
Centos/Oracle/RedHat
# Add the repository
echo "[apiomat]
name=ApiOmat
baseurl=https://<USERNAME>:<PASSWORD>@repo.apiomat.com/yambas/rest/web/Repo/LIVE/rpm
gpgcheck=0
enabled=1" > /etc/yum.repos.d/apiomat.repo
 
# Install
yum -y --nogpgcheck install aom-lurker
service lurker start
Suse
# Add the repository
zypper addrepo --no-gpgcheck https://repo.apiomat.com/yambas/rest/web/Repo/LIVE/suse?token=<TOKEN> "ApiOmat"
 
 
# Refresh zypper and install
zypper refresh
zypper --no-gpg-checks --non-interactive install aom-lurker
service lurker start
Docker Compose
Docker Compose
# Docker login
docker login -u <USERNAME> -p <PASSWORD>
 
# Store the following file and execute:
docker-compose -f lurker.yml up

lurker.yml:

Docker Compose
version: '3'
services:
lurker:
image: apiomat/lurker:<VERSION>
command: --spring.cloud.consul.enabled=false
ports:
- "8090:8090"
Kubernetes
# Docker login
kubectl create secret docker-registry regcred \
 --docker-server=https://index.docker.io/v1/ \
--docker-username="<USERNAME>" \
 --docker-password="<PASSWORD>" \
--docker-email="<MAIL>"
 
# Store the following file and execute
kubectl apply -f lurker.yml
lurker.yml
---
# Source: apiomat-lurker/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
name: release-name-apiomat-lurker
labels:
app: apiomat-lurker
chart: apiomat-lurker-1.1.0
release: "release-name"
heritage: "Tiller"
spec:
ports:
- name: http
port: 8090
protocol: TCP
type: NodePort
selector:
app: apiomat-lurker
release: release-name
---
# Source: apiomat-lurker/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: apiomat-lurker
chart: apiomat-lurker-1.1.0
heritage: Tiller
release: release-name
name: release-name-apiomat-lurker
spec:
replicas: 1
selector:
matchLabels:
app: apiomat-lurker
chart: apiomat-lurker-1.1.0
release: release-name
template:
metadata:
labels:
app: apiomat-lurker
chart: apiomat-lurker-1.1.0
release: release-name
spec:
imagePullSecrets:
- name: <regcred>
containers:
- name: apiomat-lurker
image: docker.io/apiomat/lurker:1.1.0
command:
- --spring.cloud.consul.enabled=false
ports:
- containerPort: 8090
name: ms-port
resources:
limits:
memory: 4Gi
requests:
memory: 4Gi
 
livenessProbe:
httpGet:
path: /lurker
port: 8090
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 6
readinessProbe:
httpGet:
path: /lurker
port: 8090
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 6
imagePullPolicy: Always
securityContext:
capabilities: { }
privileged: false
 
---
# Source: apiomat-lurker/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: release-name-apiomat-lurker
labels:
app: apiomat-lurker
chart: apiomat-lurker-1.1.0
release: release-name
heritage: Tiller
spec:
rules:
- host: "lurker.example.domain"
http:
paths:
- path: /
backend:
serviceName: release-name-apiomat-lurker
servicePort: ms-port
 
---
# Source: apiomat-lurker/templates/requirements.yaml
Helm
# Docker login
kubectl create secret docker-registry regcred \
 --docker-server=https://index.docker.io/v1/ \
--docker-username="<USERNAME>" \
 --docker-password="<PASSWORD>" \
--docker-email="<MAIL>"
 
# We are using helm to deploy the yambas replicaset
helm init
helm repo update
helm repo add apiomat https://apinautengmbh.github.io/helm/
helm fetch apiomat/apiomat-lurker
# extract the archive (.tgz) and find the values.yml. Adjust the parameters as required.
helm install apiomat/apiomat-lurker --name apiomat-lurker --values values.yaml
Windows
  1. Download file: https://repo.apiomat.com/yambas/rest/web/Repo/LIVE/aom-lurker-<VERSION>.zip?token=<TOKEN> and unzip to installation folder

  2. Ensure that the path to your installed jdk\bin is set to the system-environment variable 'Path' (java.exe needs to be accessible for windows' system user!)

  3. Open administrative consule and execute

    Lurker-<VERSION>.exe install
    Lurker-<VERSION>.exe start

    to install as service and start afterwards.

    For a removal of the service, execute

    Lurker-<VERSION>.exe uninstall

After service start, the REST API will provide the following endpoints:

curl "http://localhost:8090/lurker/types" -u user:secret

curl "http://localhost:8090/lurker/q/ProcessCpuLoad?to=9999999999999&from=1460356971000&scaleMilliSeconds=100000" -u user:secret

curl "http://localhost:8090/lurker/qa/ProcessCpuLoad?to=9999999999999&from=1460356971000&scaleMilliSeconds=100000" -u user:secret

Configuration

Service configuration

The following configuration keys can be used in the main configuration file (%CONFIG_PATH/application.yml ) or as Docker arguments ( e.g. --storage.useMongo=false).

General service configuration possibilities are described at the configurations main page.

Key

Default

Explanation

security.user.password

secret

Password for the default "user" for authentication against Lurkers REST API

storage.useMongo

false

Possibility to switch between in-memory and MongoDB storage. If MongoDB is configured, measures are written into database lurker

storage.mem.maxElements

500000

Maximum number of measured values to store in memory; oldest elements are overwritten (FIFO). The number of data depends on the activated mbeans; these are stored every 10 seconds.

storage.mongo.serverNames

localhost

A comma separated list of servers of the MongoDB installation. Each name can provide a port. Examples:

10.0.0.15
db.myhost.tld
10.0.0.15:27017,10.0.0.15:27018, 10.0.0.15:27019
10.0.0.15, 10.0.0.16, 10.0.0.17:28019

This value is only needed when storage.useMongo was set to true

storage.mongo.userName

 

Username for MongoDB connection; requires authentication to be set up in MongoDB

storage.mongo.password

 

Password for MongoDB connection; requires authentication to be set up in MongoDB

storage.mongo.authDB

admin

Name of the authentication database in MongoDB

storage.mongo.useSSL

false

Set to TRUE to enable SSL connection to MongoDB

requestTimeout

50000

Request timeout (ms) when collecting data from other monitoring nodes

tomcat.url

 

URL to a Tomcat container where the jolokia.war is installed; This will parse JMX values provided by this tomcat container and provide them over a REST interface across the network. This way, also remote host values will be accessible in Lurker.

Jolokia

Jolokia must be installed as jolokia.war - remove any version number from the war file before copying it to webapps folder!

jmx.host

jmx.port

localhost

13991

Host and port of local JMX access. JMX is a Java technology that supplies tools for managing and monitoring applications , system objects, devices and service-oriented networks. Those resources are represented by objects called MBeans (for Managed Bean ).

mbean.names

- java.lang:type=OperatingSystem

- java.lang:type=Runtime,java.lang:type=Threading

Provides a yaml list of MBeans which can be queried and stored in Lurker. Since the available quantity of JMX values differs between JAVA applications, the default value lists only two common MBean names. Available MBean names can be found using the /types rest method

Dashboard configuration

To show monitoring values in the Dashboard, the /etc/apiomat/apiomat.yaml file has to be updated with the username/password settings of Lurker:

Key

Default

Explanation

monitoring.user

user

Default username of Lurker

monitoring.password

secret

Default password of Lurker

monitoring.discoveryServiceURL

localhost:8500

URL to Consul service discovery

When the monitoring settings are configured, the option "Monitoring" is automatically added to the dashboard menu in the admin panel.

Cluster setup

  1. Install and start Consul on each node

  2. Install and start Lurker on each node

  3. Check that all Lurker instances are running via Consul UI (http://<SERVERNAME>:8500)

  4. Log in to Dashboard as SuperAdmin and access the monitoring page. Measurements for each node will be selectable.

images/download/attachments/61479909/monitoring_09.png

Logging

Linux

Logging is done to a separate logfile:

Linux
tail -f /var/log/apiomat/lurker.log
Docker

Please replace <LURKER_CONTAINER_ID> with your running container ID. You can list your running container using 'docker container list' command.

docker logs -f <LURKER_CONTAINER_ID>
Kubernetes

Please replace <LURKER_CONTAINER_ID> with your running container ID. You can list your running container using 'kubectl get pods' command.

kubectl logs -f <LURKER_CONTAINER_ID>
Windows

All logs are written to Event manager.

Next steps

Install further microservices.