HA Cluster setup

Summary

Deploy Mamori so the failure of a single app node does not take down the system: multiple application servers, a shared PostgreSQL database, a load balancer, and shared Mosquitto (required for multi-node). Optional InfluxDB and Grafana provide cluster monitoring.

Automation lives in the mamori-server-scripts GitHub project (ha/ directory). Clone that repository to get the scripts, then follow the runbook below. Detailed flags for each script are in ha/README.md and the step order in ha/HA-README.md.

Setting up a fully redundant Postgres cluster (replication) is beyond this document. Managed Postgres from a cloud provider is fine. The same applies to commercial load balancers (for example AWS/Azure ALB); this document assumes nginx + HAProxy on a gateway host.

Servers and services

  Clients (browsers / database clients)
                |
                v
   +-------------------------+
   | Load balancer / gateway |
   |   nginx (HTTPS)         |
   |   HAProxy (proxies)     |
   +-----------+-------------+
               |
        +------+------+
        |             |
        v             v
  +-----------+  +-----------+
  | App node1 |  | App nodeN |
  | mamori    |  | mamori    |
  +-----+-----+  +-----+-----+
        |              |
        +------+-------+
               |
        +------+------------+
        |                   |
        v                   v
 +--------------+   +------------------------+
 | Postgres box |   | Shared-services box    |
 | PostgreSQL18 |   | Mosquitto (:1883)      |
 | mamorisys,   |   | InfluxDB (:8086)       |
 | audit, xcs   |   | Grafana (:3000)        |
 +--------------+   +------------------------+

Assumptions

1. App nodes are on a private subnet (or firewalled from end clients).
2. The load balancer can reach all app nodes; clients reach only the load balancer.
3. App nodes can reach Postgres (:5432) and Mosquitto (:1883) on the private network.
4. Clients must not reach Postgres, Mosquitto, or Influx directly.

Get the scripts

git clone https://github.com/mamori-io/mamori-server-scripts.git
cd mamori-server-scripts/ha

Each script supports -h / --help.

Procedure

1. Postgres box — shared database

Option A — Docker Postgres (provided script)

Install Docker on the Postgres host, then:

bash install-ha-postgres.sh --password 'choose-a-strong-password'

This pulls official postgres:18, enables remote SCRAM-SHA-256 auth, then runs init-ha-postgres.sh and check-ha-postgres.sh to create and verify mamorisys, audit, and xcs.

Option B — your own Postgres (native or managed)

Use PostgreSQL 18 (or compatible). Enable remote SCRAM-SHA-256 auth and allow app nodes on port 5432. Then create and verify the empty databases:

bash init-ha-postgres.sh --host <pg-host> --port 5432 --user postgres --password 'choose-a-strong-password'
bash check-ha-postgres.sh --host <pg-host> --port 5432 --user postgres --password 'choose-a-strong-password'

Schema objects are created later on first app-node boot. check-ha-postgres.sh only confirms the instance is reachable and the three databases exist.

Verify from an app-node host:

PGPASSWORD='choose-a-strong-password' psql --host <pg-host> --port 5432 -U postgres -d mamorisys -c 'select version()'

2. First app node — join and prime the database

On node1 (no hand-written env file). Without --env-file, install prompts for Postgres and the portal root password, checks that mamorisys is unprimed, and writes /tmp/cluster-details.env for join:

bash validate-new-node.sh
bash get-ha-media.sh --dir /tmp
bash install-ha-node.sh --media /tmp/mamori_cluster_docker.tgz
# prompts PG_* + portal root; writes /tmp/cluster-details.env
bash join-ha-node.sh --env-file /tmp/cluster-details.env
bash start-ha-node.sh

Only the first app node sets the portal root password. First boot stores it encrypted; start-ha-node.sh then removes that env from the container. Additional nodes pass --env-file and do not prompt.

First boot creates schema and objects in the shared databases (usually under a minute):

docker exec -it mamori tail -F /opt/mamori/var/log/mamori_fqod.log

Verify the node (before the load balancer)

Confirm login works on the node itself before relying on nginx/HAProxy. Use the HTTP UI test helpers from mamori-server-scripts (ha/). Log in as root with the portal password chosen at install.

Option A — curl (no nginx change):

rm -f /tmp/cj
curl -c /tmp/cj -b /tmp/cj -sS -o /dev/null http://127.0.0.1/
curl -c /tmp/cj -b /tmp/cj -sS -X POST http://127.0.0.1/sessions/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"root","password":"YOUR_PASSWORD"}'

Option B — browser (temporary nginx change; restore when done):

bash enable-http-ui-test.sh
# open http://<node-ip>/#/login and sign in
bash restore-http-ui-test.sh

Clear browser cookies (or use a private window) after restore. Always run restore-http-ui-test.sh before putting the node behind the HTTPS load balancer.

3. Shared-services box — Mosquitto (required for multi-node)

On the shared-services host:

mkdir -p /opt/mamori/mosquitto/{data,log}
cat >/opt/mamori/mosquitto/mosquitto.conf <<'EOF'
persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
bind_address 0.0.0.0
allow_anonymous true
EOF

docker create --name mosquitto --restart always --network host \
  --log-opt max-size=10m --log-opt max-file=5 \
  -v /opt/mamori/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf \
  -v /opt/mamori/mosquitto/data:/mosquitto/data \
  -v /opt/mamori/mosquitto/log:/mosquitto/log \
  eclipse-mosquitto
docker start mosquitto

On node1:

docker exec -it mamori msql "call SET_SERVER_PROPERTY('mqtt_server', 'tcp://<shared-services-host>:1883')"
docker exec -it mamori sv restart mamori_fqod

Offline image mirror (optional): https://mamori-io.sgp1.digitaloceanspaces.com/docker-images/eclipse-mosquitto.tgz

4. Load balancer — nginx and HAProxy

Configure nginx for HTTPS termination to the app upstream; and configure HAProxy for database and other proxies to the app nodes. Point backends at node1 first.

NOTE: the load balancer must set the X-Real-IP header to the client address; otherwise the hub reports the load balancer IP.

If using HAProxy with PROXY protocol, on node1:

docker exec -it mamori msql "call SET_SERVER_PROPERTY('haproxy', 'true')"
docker exec -it mamori sv restart mamori_fqod

Verify HTTPS login through the load balancer.

Example nginx/HAProxy configuration patterns and health-check notes are maintained with operations; use dump-lb-config.sh on a gateway that already has Mamori load balancer configs.

5. Additional app nodes

On an existing app node:

bash extract-cluster-details.sh -o /tmp/cluster-details.env

Copy the env file to the new node, then:

bash validate-new-node.sh --env-file /tmp/cluster-details.env
bash get-ha-media.sh --dir /tmp
bash install-ha-node.sh --env-file /tmp/cluster-details.env --media /tmp/mamori_cluster_docker.tgz
bash join-ha-node.sh --env-file /tmp/cluster-details.env
bash start-ha-node.sh

Additional nodes pass --env-file and do not prompt. extract-cluster-details.sh includes encrypted DERBY_USER_ROOT; join-ha-node.sh applies it to the new node’s mamori-var. Log in with the same root password as the first node.

Verify the node (before registering on the load balancer)

Do not register the node until login succeeds on the node host.

Option A — curl:

rm -f /tmp/cj
curl -c /tmp/cj -b /tmp/cj -sS -o /dev/null http://127.0.0.1/
curl -c /tmp/cj -b /tmp/cj -sS -X POST http://127.0.0.1/sessions/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"root","password":"YOUR_PASSWORD"}'

Option B — browser:

bash enable-http-ui-test.sh
# open http://<node-ip>/#/login and sign in
bash restore-http-ui-test.sh

Then on the load balancer:

bash dump-lb-config.sh
bash manage-lb-node.sh --register --name <hostname> --ip <internal-ip>
bash dump-lb-config.sh

6. Shared-services — InfluxDB and Grafana (optional)

Install InfluxDB and Grafana on the same shared-services host. Then on an app node:

docker exec -it mamori msql "call SET_SERVER_PROPERTY('influxdb_write_url', 'http://<shared-services-host>:8086/write?db=mamori')"

Open Grafana at http://<shared-services-host>:3000/monitor (default admin credentials are typically admin / admin on first login — change immediately). You may later proxy /monitor through the load balancer.

Manage nodes on the load balancer

bash manage-lb-node.sh --disable --name <hostname>
bash manage-lb-node.sh --enable --name <hostname>
bash manage-lb-node.sh --unregister --name <hostname>

WireGuard support (optional)

On hosts that need IPVS for WireGuard VIP distribution:

apt install ipvsadm

#!/bin/bash
VIP=10.240.0.36
PORT=51871
NODE1=10.240.0.11

ipvsadm -C
ipvsadm -A -u $VIP:$PORT -s sh
ipvsadm -a -u $VIP:$PORT -r $NODE1 -m

Add further app-node backends with additional ipvsadm -a lines as required.