Health

Mistgate watches the fleet from two sides. The panel connects to every profile on every node the way a client app does (client-eye checks), and the agent on every node checks its own host (the node doctor). Alerts are derived from both and from the state of the nodes. Everything is on the Health page, in the tabs Alerts, Checks and Doctor; the node page has its own Doctor tab and shows the checks of that node on its Overview tab.

Client-eye checks#

How a check works#

For every profile on a node the panel opens a tunnel with a built-in client and, through that tunnel, requests two pages:

The host names are resolved by the node, not by the panel, so a broken resolver on the node shows up as a failed check.

A result is one of:

• OK: both pages answered.
• Degraded: the tunnel works but one of the two pages did not answer. The latency is shown in the warning colour. A degraded result never opens an alert by itself.
• Failed: the tunnel did not come up, or neither page answered.

The client depends on the protocol:

• Hysteria2. The official Hysteria2 client, running inside the panel. It dials the node's address and the profile's own port (not the port-hopping range, so a blocked hop range is not seen). It uses the profile's obfuscation (Salamander or Gecko) and the certificate rules of a real client: a self-signed certificate is pinned to the fingerprint the node reported, any other is checked against the profile's Domain (SNI).
• AmneziaWG. amneziawg-go inside the panel process on a user-space network stack: no network interface, no root, nothing added to the panel server's routes. It uses the same configuration a real device gets. Every round is a fresh device, so every round also tests the handshake. A node does not answer an unknown peer or a packet with the wrong obfuscation, so a missing handshake and a blocked UDP port look the same: both are reported as a timeout.

A protocol without a test client is shown as not checked.

The check uses a hidden system account on each profile of each node: it belongs to no user and no device, and it never appears in user lists, subscriptions, quotas or the MCP tools. On a Hysteria2 profile the node limits it to 2 Mbit/s. On an AmneziaWG profile it takes one address of the profile's client network.

Schedule#

Reading the matrix#

Health → Checks is a table of nodes (rows) by profiles (columns). A cell shows the latency in milliseconds or a state:

Only "fails" counts as a failure. The other states are skipped rounds: they are not stored and do not touch the failure streak.

Click a cell to see the last 24 hours in half-hour bars (red where a round failed), the exit IP and Failed in a row. An alert opens after two failed rounds in a row.

When a round fails, the cell says why:

What green means#

A green cell means that, within the last few minutes, a client from the panel's network completed the handshake with this profile on this node and reached two well-known sites through the node's exit. It does not prove that the port-hopping range works, that your users' networks let the traffic through, or that every app handles the profile.

Node doctor#

Every node agent runs 16 checks of its own host. They only read: files under /proc and /sys, a few configuration files and read-only commands (systemctl list-*, nft list, timedatectl show, dmesg, journalctl -k). Changing anything is a separate, confirmed step (see "Fixes" below).

When it runs#

• About 30 seconds after the agent connects, then every 10 minutes (with up to a minute of jitter).
• Run again on Health → Doctor or on the node's Doctor tab asks the node now and waits up to 30 seconds for the report.
• Every check has 10 seconds, the whole run 30 seconds. A check that hangs, or that cannot run on this host (a container, no systemd, not root), is Skipped with the reason. It is never reported as OK.
• A report older than 25 minutes is stale: the admin greys it out and says how old it is. An offline node shows its last report, and fixes are not available until it is back.
• An agent that predates the doctor is shown as "too old for the doctor". Update it on the Updates page.

Each item has one of four states: OK, Attention (a warning), Problem (a failure) and Skipped.

The checks#

A node without IPv6 that runs WARP is fine: WARP uses its IPv4 endpoint, and the admin shows the IPv6 item as OK.

A warning or problem opens an alert (see "Kinds of alerts" below). The certificate check is the exception: it opens a "Certificate is about to expire" alert instead of a doctor alert.

Fixes#

There are exactly four fixes. They are compiled into the agent: the panel sends only the fix id, and the agent checks the parameters (a profile must be one the node runs). None of them installs packages or deletes data.

A fix always takes two steps:

1. Click the fix button on the doctor item or on its alert. The panel asks the node what it would do ("Asking the node what it would do…"); the node answers with a dry run that changes nothing.
2. The dialog shows the plan: for example, the journal size before and after, or the profiles to restart and how many connections drop. Apply performs exactly that plan.

The plan is valid for 10 minutes, works once and is bound to the node, the fix and its parameters. One fix runs at a time on a node. Both steps go to the audit log. After a successful fix the node runs the affected checks again, and alerts that the fix cleared close as "fixed". Applying a fix needs the owner.

When a fix fails, the dialog says why:

For checks without a fix, the item opens a Manual block with the commands to run on the node: turning on time sync, finding what holds a port, reading the firewall rules, or a note to reboot from the hoster's panel.

"This is normal for this node"#

Some warnings are facts you accept: Docker's own nftables tables, a host without working IPv6. This is normal for this node on a doctor item or its alert stops it from raising an alert and from counting in the badges. The item moves to Accepted as normal and shows who accepted it and when.

• Only a warning (Attention) can be accepted. A problem cannot, and a certificate warning never can.
• The acceptance is for the exact fact the node reported (its detail code). It ends by itself when the node reports a different fact for that check or the check turns into a problem. An agent too old to send detail codes cannot have its warnings accepted.
• Count it again takes the acceptance back.
• The alert of an accepted warning closes as "accepted as normal".

Alerts#

How alerts open and close#

Alerts are a function of the current state. Every 10 seconds, and after every check round and doctor report, the panel works out which conditions hold now: a condition without an alert opens one, an alert whose condition is gone is resolved. There is one active alert per kind, node and subject (a profile, a doctor check). Because nothing depends on remembered events, alerts survive a panel restart.

When the panel cannot judge a condition, the alert stays as it is instead of closing: an offline node keeps its other alerts (only "Node is unreachable" is decided for it), a stale doctor report keeps the doctor alerts, and a check result from before the node went away keeps the check alerts.

An alert that fires again within an hour of closing reopens with its original id and start time. During the first 90 seconds after the panel starts, no "Node is unreachable" opens: the agents are still reconnecting.

Kinds of alerts#

The reason under a traffic alert names the likely cause:

Two more kinds exist in the API, but the panel does not raise them as alerts: a user who reached the quota (the user's status shows it) and a subscription that looks shared (written as an event, "a subscription link is used from N networks a day").

Severity and badges#

Alerts are Critical, Warning or Info. The badge in the header and on the Overview counts active alerts that are not muted and not Info. The active list shows Critical first, then the newest.

How an alert ended#

The History · 7 days list says how each alert closed:

Closed alerts are kept for 90 days. The API returns up to 30 days of history, at most 200 alerts.

Buttons on an alert#

Depending on what the alert says, its card offers:

• the doctor's fix (only while the doctor still offers it and the node is connected);
• Restart profile or Restart the profiles, with the number of connections that drop;
• Open the node's profiles, where a port or a domain is changed;
• the node's WARP card;
• Open node;
• This is normal for this node, for a doctor warning;
• Mute.

Muting#

Mute offers: For 1 hour, Until morning (08:00), For a day, For 7 days. The longest mute is 7 days; Unmute ends it.

• A muted alert stays on the list with "Muted until …". It is still evaluated and still closes by itself.
• It is not counted in the badges.
• If its severity rises, the mute ends and the alert counts again.
• Muting is written to the audit log.

Events on the node page#

The node's Events tab and the Overview's event feed keep a timeline that answers "what happened last night":

Events of severity info are kept for 90 days, warnings and errors for 400 days. Raw check rounds are kept for 25 hours after the day is summarised; the daily summaries for 90 days.

Who can do what#

An API token with the operator profile can mute alerts, start checks and run the doctor. A doctor fix through a token is possible only via MCP, as a plan the owner approves (see MCP).