Troubleshooting

Symptoms and checks

Symptom	Check
Node gets no IP	journalctl -u dnsmasq -f, cable and PoE, ip -br addr show <RACK_IF>
Node gets an IP but does not connect to MQTT	docker compose ps, sudo nft list table inet celestian_gw (drop counters), node serial log ([mqtt] broker …)
LCD AUTH!	the broker rejected the credentials or client_id. Mainly in direct mode: check mqtt_user and mqtt_pass, and that the device is connected to the channel with the publish type. In gateway mode the local broker is anonymous, so check the gateway’s Mosquitto configuration
Node online on the LAN but not appearing in Celestian	docker compose logs mosquitto (bridge rejected = wrong ID/key or clientid not prefixed with the gateway ID); docker compose logs history; no storage rule on the channel
Data does not reach Celestian	the gateway has no publish connection to the channel (history: “ditolak Celestian (0x87)”); no storage rule
Duplicate telemetry and events in Celestian	bridge.conf still contains +/telemetry / +/event; remove both lines
Node clock wrong, ts empty	chronyc clients, chronyc tracking on the gateway
“Estimated time” in Celestian	the gateway has no NTP upstream (Reference ID 7F7F0101); set NTP_UPSTREAM or make sure there is internet/tunnel to NTP
history stops, “belum diisi”	fill GATEWAY_ID, GATEWAY_KEY, WORKSPACE, CHANNEL in .env
config / cmd from the server does not arrive	the gateway has no subscribe connection to the channel; downlink lines in bridge.conf
Rules, sources, or groups not visible in Celestian	the rules, sources, profiles, tgroups lines (and */reported, */current) are missing from bridge.conf
Firmware rollout fails with HTTP 503	CELESTIAN_WEB_URL / CELESTIAN_WEB_HOST not set (OTA on the gateway)

See also the FAQ for node-side problems.

Optional: WireGuard to the Celestian server

With WireGuard, the MQTT bridge runs inside an encrypted tunnel and the broker port need not be opened to the internet. The MQTT login (gateway ID + device key) is still used. This feature is prepared but not yet used in production.

1. Install and set up the tunnel (template host/wg0.conf.example):

   sudo apt-get install -y wireguard

   Copy the template to /etc/wireguard/wg0.conf, fill in the keys and addresses, then:

   sudo systemctl enable --now wg-quick@wg0

2. In mosquitto/conf.d/bridge.conf, point address to the server’s VPN address and the broker’s internal MQTT 5 port (not the public port), e.g. address 10.99.0.1:1884. Change CELESTIAN_HOST and the port in .env to match.
3. docker compose restart mosquitto, then check sudo wg show (there is a “latest handshake”) and the bridge log.

Notes:

• AllowedIPs on the gateway contains only the server’s VPN address, so the gateway’s ordinary internet route does not change.
• The Mosquitto container reaches the tunnel through the host; no extra Docker setup is needed.
• Nodes on the rack LAN still cannot reach the tunnel: the firewall rejects all forwarding from the rack LAN.
• The same tunnel can be used for SSH to the gateway from the server.