History storage

The history component (Python, SQLite in the history-data volume) sits between the local Mosquitto and Celestian. It stores telemetry and events on disk, merges them into SenFlat v2 messages, and resends them after the connection to Celestian recovers.

Matter	Behavior
Input	subscribes to sentinel/+/telemetry and sentinel/+/event (QoS 1, persistent session, so messages are not lost while the component is down). QoS 1 retransmissions from the node to Mosquitto are dropped by the node’s (id, q)
Merging	telemetry is merged per 2-second window (30 seconds while Celestian is down) or 256 KiB; events are sent immediately. One message holds many sources; point times created by the node are never altered. Max. 512 KiB per message, split (never truncated)
q	the gateway’s counter in the database, never goes backward. A retransmission uses the same q; Celestian drops duplicates
h and tx	h=0 for new data; h=1 if a send was already attempted or it is older than 30 seconds. tx = gateway send time (ms)
Celestian down	data waits on disk. Once connected, the queue is sent in order (16 messages at a time) and deleted only after PUBACK
Permission denied (0x87)	the data stays on disk and is retried every 30 seconds. Permanently rejected (broken or too-large payload) = dropped
Full	the oldest data is dropped and counted in the log

Storage limits

Set in .env:

Variable	Default	Meaning
HISTORY_MAX_BYTES	4 GiB	maximum size of the on-disk queue
HISTORY_MAX_AGE_S	259200 (3 days)	maximum age of data
HISTORY_MAX_MESSAGES	0	maximum number of messages; 0 = unlimited

Estimated load: one G1 node ±3.5 MB per day; 240 nodes ±0.8 GB per day. Installations with many Modbus metrics are much larger per node; raise HISTORY_MAX_BYTES (and make sure the SSD is large enough) or lengthen telemetry_interval_ms on the node if the size limit is reached before 3 days.

Clock checking

A node marks ts:1 if its NTP succeeded, but the node cannot know whether the gateway’s own chrony has a real NTP upstream. history checks chronyc tracking every 30 seconds. If the NTP upstream is lost for longer than GATEWAY_CLOCK_STALE_S (default 3 days), the sources it forwards are marked ts:0 (Celestian stores them with an estimated time and flags it).

Variable	Meaning
GATEWAY_CLOCK_STALE_S	how long the NTP upstream may be lost; 0 = strict (ts:0 as soon as the upstream is lost)
GATEWAY_CLOCK_CHECK=off	not checked (e.g. a MacBook as gateway)

The gateway’s NTP upstream is set in host/gateway.env (NTP_UPSTREAM).

Log

docker compose logs -f history

Shows a queue summary every minute. If the log says “belum diisi” (not filled in), fill GATEWAY_ID, GATEWAY_KEY, WORKSPACE, and CHANNEL in .env. If “ditolak Celestian (0x87)” (rejected by Celestian), the gateway device does not yet have a publish connection to that channel; the data is safe on disk and sent as soon as permission exists.