Install HA Services

HA Cluster setup

Summary

Deploy Mamori so the failure of a single app node does not take down the system: multiple application servers, a shared PostgreSQL database, a load balancer, and shared Mosquitto (required for multi-node). Optional InfluxDB and Grafana provide cluster monitoring.

Automation lives in the mamori-server-scripts GitHub project (ha/ directory). Clone that repository to get the scripts, then follow the runbook below. Detailed flags for each script are in ha/README.md and the step order in ha/HA-README.md.

Setting up a fully redundant Postgres cluster (replication) is beyond this document. Managed Postgres from a cloud provider is fine. The same applies to commercial load balancers (for example AWS/Azure ALB); this document assumes nginx + HAProxy on a gateway host.

Servers and services

  Clients (browsers / DB clients)
              |
              v
     +--------------------+
     |  LB / gateway      |
     |  nginx (HTTPS)     |
     |  HAProxy (proxies) |
     +---------+----------+
               |
        +------+------+
        |             |
        v             v
  +-----------+  +-----------+
  | App node1 |  | App nodeN |
  | mamori    |  | mamori    |
  +-----+-----+  +-----+-----+
        |              |
        +------+-------+
               |
     +---------+---------+
     |                   |
     v                   v
 +-----------+   +------------------------+
 | Postgres  |   | Shared-services box    |
 | PostgreSQL|   | Mosquitto (:1883)      |
 | 18        |   | InfluxDB (:8086)       |
 | mamorisys |   | Grafana (:3000)        |
 | audit,xcs |   +------------------------+
 +-----------+

Server	Services
LB / gateway	nginx (HTTPS → app `:80`), HAProxy (DB/SSH and other proxies → app nodes)
App nodes	Mamori container only (`mamori-var`, `mamori-nginx-conf`)
Postgres box	PostgreSQL 18 with databases `mamorisys`, `audit`, `xcs` (remote MD5 auth)
Shared-services box	Mosquitto, InfluxDB, Grafana (not on the LB or Postgres host)

Assumptions

App nodes are on a private subnet (or firewalled from end clients).
The load balancer can reach all app nodes; clients reach only the load balancer.
App nodes can reach Postgres (:5432) and Mosquitto (:1883) on the private network.
Clients must not reach Postgres, Mosquitto, or Influx directly.

Get the scripts

git clone https://github.com/mamori-io/mamori-server-scripts.git
cd mamori-server-scripts/ha

Each script supports -h / --help.

Procedure

1. Postgres box — shared database

Install Docker on the Postgres host, then:

bash install-ha-postgres.sh --password 'choose-a-strong-password'

This pulls official postgres:18, enables remote MD5 auth, and creates mamorisys, audit, and xcs.

Verify from another host:

PGPASSWORD='choose-a-strong-password' psql --host <pg-host> --port 5432 -U postgres -d mamorisys -c 'select version()'

2. First app node — join and prime the database

On node1:

cat >/tmp/cluster-details.env <<'EOF'
PG_HOST=<pg-host>
PG_PORT=5432
PG_USER=postgres
PG_PASSWORD=choose-a-strong-password
EOF

bash validate-new-node.sh
bash get-ha-media.sh --dir /tmp
bash install-ha-node.sh --media /tmp/mamori_cluster_docker.tgz
bash join-ha-node.sh --env-file /tmp/cluster-details.env
docker start mamori

First boot creates schema and objects in the shared databases (usually under a minute):

docker exec -it mamori tail -F /opt/mamori/var/log/mamori_fqod.log

Verify the node (before the load balancer)

Confirm login works on the node itself before relying on nginx/HAProxy. Use the HTTP UI test helpers from mamori-server-scripts (ha/). Default hub login is root / Mamori2022 unless changed.

Option A — curl (no nginx change):

rm -f /tmp/cj
curl -c /tmp/cj -b /tmp/cj -sS -o /dev/null http://127.0.0.1/
curl -c /tmp/cj -b /tmp/cj -sS -X POST http://127.0.0.1/sessions/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"root","password":"YOUR_PASSWORD"}'

Option B — browser (temporary nginx change; restore when done):

bash enable-http-ui-test.sh
# open http://<node-ip>/#/login and sign in
bash restore-http-ui-test.sh

Clear browser cookies (or use a private window) after restore. Always run restore-http-ui-test.sh before putting the node behind the HTTPS load balancer.

3. Shared-services box — Mosquitto (required for multi-node)

On the shared-services host:

mkdir -p /opt/mamori/mosquitto/{data,log}
cat >/opt/mamori/mosquitto/mosquitto.conf <<'EOF'
persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
bind_address 0.0.0.0
allow_anonymous true
EOF

docker create --name mosquitto --restart always --network host \
  --log-opt max-size=10m --log-opt max-file=5 \
  -v /opt/mamori/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf \
  -v /opt/mamori/mosquitto/data:/mosquitto/data \
  -v /opt/mamori/mosquitto/log:/mosquitto/log \
  eclipse-mosquitto
docker start mosquitto

On node1:

docker exec -it mamori msql "call SET_SERVER_PROPERTY('mqtt_server', 'tcp://<shared-services-host>:1883')"
docker exec -it mamori msql "sv restart mamori_fqod"

Offline image mirror (optional): https://mamori-io.sgp1.digitaloceanspaces.com/docker-images/eclipse-mosquitto.tgz

4. Load balancer — nginx and HAProxy

Configure nginx for HTTPS termination to the app upstream, and HAProxy for database and other proxies to the app nodes. Point backends at node1 first.

NOTE: the load balancer must set the X-Real-IP header to the client address; otherwise the hub reports the load balancer IP.

If using HAProxy with PROXY protocol, on node1:

docker exec -it mamori msql "call SET_SERVER_PROPERTY('haproxy', 'true')"
docker exec -it mamori msql "sv restart mamori_fqod"

Verify HTTPS login through the load balancer.

Example nginx/HAProxy configuration patterns and health-check notes are maintained with operations; use dump-lb-config.sh on a gateway that already has Mamori LB configs.

5. Additional app nodes

On an existing app node:

bash extract-cluster-details.sh -o /tmp/cluster-details.env

Copy the env file to the new node, then:

bash validate-new-node.sh
bash get-ha-media.sh --dir /tmp
bash install-ha-node.sh --media /tmp/mamori_cluster_docker.tgz
bash join-ha-node.sh --env-file /tmp/cluster-details.env
docker start mamori

Verify the node (before registering on the LB)

Do not register the node until login succeeds on the node host.

Option A — curl:

rm -f /tmp/cj
curl -c /tmp/cj -b /tmp/cj -sS -o /dev/null http://127.0.0.1/
curl -c /tmp/cj -b /tmp/cj -sS -X POST http://127.0.0.1/sessions/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"root","password":"YOUR_PASSWORD"}'

Option B — browser:

bash enable-http-ui-test.sh
# open http://<node-ip>/#/login and sign in
bash restore-http-ui-test.sh

Then on the load balancer:

bash dump-lb-config.sh
bash manage-lb-node.sh --register --name <hostname> --ip <internal-ip>
bash dump-lb-config.sh

6. Shared-services — InfluxDB and Grafana (optional)

Install InfluxDB and Grafana on the same shared-services host. Then on an app node:

docker exec -it mamori msql "call SET_SERVER_PROPERTY('influxdb_write_url', 'http://<shared-services-host>:8086/write?db=mamori')"

Open Grafana at http://<shared-services-host>:3000/monitor (default admin credentials are typically admin / admin on first login — change immediately). You may later proxy /monitor through the load balancer.

Manage nodes on the load balancer

bash manage-lb-node.sh --disable --name <hostname>
bash manage-lb-node.sh --enable --name <hostname>
bash manage-lb-node.sh --unregister --name <hostname>

WireGuard support (optional)

On hosts that need IPVS for WireGuard VIP distribution:

apt install ipvsadm

#!/bin/bash
VIP=10.240.0.36
PORT=51871
NODE1=10.240.0.11

ipvsadm -C
ipvsadm -A -u $VIP:$PORT -s sh
ipvsadm -a -u $VIP:$PORT -r $NODE1 -m

Add further app-node backends with additional ipvsadm -a lines as required.