System Health Monitor
This page was checked against the development-candidate source code on 2026-10-04. Field behavior is governed by the deployed firmware; the new per-pool endpoints, TLS protection, and resource observation still require centralized verification and deployment, so this page must not be taken as proof that a device has been updated. For the current candidate state, see Acceptance Status.
For identity and configuration checks before dealing with broken configuration, boot failure, or maintenance, see Fault Recovery and Configuration Preservation; configuration migration, network fallback, and OTA rollback each apply separately and cannot replace one another.
Monitoring Scope and Frequency
An enabled HealthMonitor, as it is advanced from the main loop, checks internal DRAM, the largest contiguous block, the low watermark, filesystem mounts, and Wi-Fi status every 5 seconds, and drives memory protection and recovery. This period is not a deadline guarantee from an independent real-time task.
| Data | Actual Scope | Notes |
|---|---|---|
| DRAM free / largest block / lowest watermark | Internal 8-bit heap, excluding PSRAM | Protects critical allocations such as networking; free PSRAM cannot offset internal pressure |
| Task stack watermark log | New candidates query only the caller itself every 60 seconds, with the scope marked current-task | ESP-IDF watermark unit is bytes; the caller name is not assumed, and handles of short-lived tasks that may have been destroyed are no longer queried |
| Runtime metrics summary | [METRICS] printed every 60 seconds, including heap/min/dram8/blk8/frag/mqtt_q/sse/poll/up | Queue, connection, and polling values are updated by their respective business paths; they do not prove the physical link is healthy |
| DRAM trend | Saves the hourly lowest watermark; at least 6 samples are required before a trend is judged | A suspected leak is flagged when the average watermark of the second half is more than 8192 bytes lower than that of the first half; this is neither leak localization nor a long-term-stability pass |
| CPU usage | cpuUsage in the current health structure is a placeholder 0 | Must not be interpreted as measured CPU idleness |
Unified Memory Accounting
The development candidate provides statistics for three allocator classes in data.memory.capabilityPools of the full GET /api/system/info?full=1 and GET /api/system/diag-bundle responses. A bare /api/system/info is compact by default; an explicit full request may still be degraded by memory protection, in which case the block stays absent and the values remain unknown:
| Pool | Allocation Capability | Fields |
|---|---|---|
internal8 | Internal, byte-accessible | observed/free/minFree/largestBlock |
internalDma | Internal, DMA, byte-accessible | Same as above |
psram8 | PSRAM, byte-accessible | Same as above |
Values are in bytes; unobserved values are null. The internal DMA pool overlaps the internal 8-bit pool, and the two must not be added together. atomicSnapshot=false means the three reads were not taken at the same atomic instant; minimumScope=sum_of_heap_low_watermarks means the sum of the low watermarks of each memory region — these lows did not necessarily occur at the same time, and they are not the exclusive peak of any single feature. sampledAtUptimeMs is device uptime, not UTC.
These endpoints do not reset watermarks, create sampling tasks, or trigger resource recovery. Compact, probe, and health responses do not add this sampling, and older firmware may also lack the fields. A missing field must be distinguished from a measured 0.
The pre-existing data.memory.heapTotal/heapFree/heapUsed/heapMinFree/heapMaxAlloc come from the ESP heap API; total/free/used/usagePercent are page-level summaries that may include PSRAM. When comparing the page with the API, use values from the same response; values from different requests or different heap scopes must not be subtracted directly.
Developer One-shot Task Stack Snapshot
Candidates after 2026-10-04 add POST /api/system/task-stack-snapshot, which requires login and developer mode; r20 centralized diagnostics r2 has passed 26 Host regression items, but the full firmware candidate and real hardware have not been accepted or deployed. Pages and health checks never call it automatically. This debugging operation may temporarily suspend scheduling and must not be used as a routine liveness or polling endpoint.
A request must pass the HealthMonitor checks of at least 32768 bytes of free internal 8-bit memory and a largest contiguous block of at least 8192 bytes; scans are not started at SEVERE or above, while a capture is busy, during the 30-second cooldown, or when there are more than 48 tasks. PSRAM cannot substitute for the internal budget. An unknown result never returns a stale list, nor is an SDK return of 0 or over-capacity interpreted as "no tasks".
Each entry contains only the task number, priority, state, and highWaterBytes copied by the SDK; the task name is null, and potentially destroyed tasks or name pointers are no longer accessed. The number is not a cross-reboot identity, and 0 bytes is a valid low watermark. completeAtCapture=true only means the SDK list was complete at capture time; it does not mean the task still exists when the response is read, nor does it prove that the worst-case business path has been covered.
The response keeps the task counts before and after, plus captureDurationUs. The latter is the measured duration of the SDK call and cannot limit or interrupt that call; hardDeadlineEnforced=false and atomicSnapshot=false. Developers must still validate scan overhead, internal heap, HTTP latency, and the WDT on the specific board, with a stable candidate, under controlled load. A single snapshot is not equivalent to memory gains across all features, task peaks in all scenarios, or long-term-stability acceptance.
Memory Protection and Recovery
The NORMAL/WARN/SEVERE/CRITICAL level is determined from free internal DRAM, contiguous blocks, fragmentation, and chip tier, with boot grace and recovery hysteresis. The full level cannot be inferred from a single free value. An absolute DRAM emergency is not masked by boot grace.
As pressure rises, the system may pause logging, Modbus, rule execution, and optional peripherals while keeping the management entry available and attempting recovery. When resource release is busy or fails, a pause request does not equal completed release; also check the instance/retirement and Worker states. The current development batch changes MQTTS protection to a runtime pause that preserves scheme/port/enabled and forbids automatic plaintext fallback; this is still pending unified verification. memoryDegradation.mqttsPaused indicates the pause, and mqttsProtection=pause_without_plaintext_fallback states the policy; the legacy compatibility field mqttsDowngraded is fixed at false. The metrics counterparts are memguard.mqtts_paused/mqtts_protection, whose legacy mqtts_downgraded is likewise fixed at false.
Downgrade records already written to flash by old firmware are not erased. Only with complete and valid evidence of the original TLS scheme/port does a new candidate load with that secure configuration; when metadata is missing, startup is refused — the port must not be guessed and plaintext must not continue. Configurations persistently disabled by the old version with enabled=false stay disabled until the user explicitly chooses to save. Runtime pause, historical disablement, and queued recovery are different states; a queued rebuild does not mean connected.
GET /api/health and GET /api/system/status may return data.memoryDegradation. Its level, active, modbusStopped, periphExecPaused, optionalPeripheralsSuspended, and actions fields describe the degradation state; these fields are not exclusive per-feature memory measurements. Temporary degradation, user-saved switches, and whether actual objects are preserved should be checked separately against Feature Configuration.
Under extremely low memory or prolonged failure to recover, the code may schedule a restart, with boot-loop protection. Timing conditions differ across paths and cannot be generalized as "restart on one low watermark" or "a uniform 90-second restart". Management pages, recovery, and restart can still be affected by OOM or low-level faults; accessibility under every fault is not guaranteed.
Endpoint Rejection and Backoff
Heavy endpoints may return 503 due to the protection level or insufficient memory. When the response carries Retry-After, back off accordingly; without that header, still back off a bounded amount and avoid concurrent retries. Authentication errors 401/403 must be handled as login/permission issues first, not treated as memory shortage.
OTA additionally checks for an available update partition and at least 20 KiB of internal DRAM; this batch's fix excludes PSRAM from that admission accounting. The contiguous-block threshold for new SSE connections and broadcasts is 8192 bytes, likewise unified to the internal 8-bit basis in this batch; the original thresholds and auth/capacity protection are kept, pending centralized regression. When SSE slots are full, new connections are usually rejected; connection registration also retains a fallback that selects the oldest slot by connection time — it is not an LRU service guarantee ordered by recent activity.
Diagnostic Endpoints
| Endpoint | Response Scope | Wrapping and Limits |
|---|---|---|
GET /api/health, GET /api/system/health | status/timestamp/freeHeap/memory plus available degradation state | Same lightweight handler, standard success/data wrapping; status=healthy alone does not prove the system is not degraded |
GET /api/system/info?full=1 | Full system info, page memory, and available per-pool snapshot | Login required; a bare URL is compact by default, an explicit full request may still be degraded by memory protection, and compact/probe paths have fewer fields |
GET /api/system/metrics | Runtime metrics such as heap/mqtt/sse/poll/memguard/stability | Login required; returns the metrics JSON directly without success/data wrapping; returns 503 when HealthMonitor is unavailable |
GET /api/system/diag-bundle | data.device/restart/memory/filesystem/network/runtime/logs | Login required, subject to heavy-request and low-memory protection; network and some content are returned according to actual object availability |
Metrics Fields and Units
heap.free/min/largest_block/dram_free/dram_largest are in bytes; the historical compatibility fields heap.psram_total/psram_free are in KiB. uptime is in seconds, and boot_time_ms and poll.duration_ms are in milliseconds. Do not mix these units when reading.
memguard.level_name gives the level, and stability.dram_history_count/leak_suspect/uptime_hours give the trend-observation state. heap.tls_supported is only a legacy PSRAM condition flag; it does not prove that the TLS handshake, certificate, or platform authentication passed. The current endpoints do not provide a taskStackWatermarks all-task array, nor any so-called hourly report that automatically locates leaking objects.
Restart and Runtime Snapshot
The diagnostic bundle's restart.resetReason/resetReasonCode/abnormalRestart describe the current boot. Usable lastRestartReason/lastRestartContext/lastUptimeMs/preRestart exist only when snapshotValid=true. lastUptimeMs is the uptime timestamp of the last snapshot, not UTC.
preRestart may contain pre-restart heap, contiguous block, fragmentation, guard, Wi-Fi, and queue values; it is a single retained snapshot, not a restartHistory list or a complete power-loss record. New candidates keep the legacy named-task stack fields as null and report taskWatermarkStatus=unknown, because the original name-based query cannot guarantee the task is still alive at later reads; a stack overflow is never inferred from unknown 0 bytes. The legacy RTC structure and evidence are retained, and no all-task scan has been added to the restart flow. Without a valid snapshot, reasons, counts, or times must not be fabricated.
runtime.taskCount is the task count, and sseClients/sseMaxClients are provided when the corresponding Web objects exist; they do not represent per-task stacks, actual throughput, or physical link quality. filesystem describes LittleFS statistics; the SD card has its own diagnostics and is not treated as the same filesystem.
Collection and Acceptance
The fragmentation percentage is computed as 100 × (1 − largest/free) over the same internal DRAM range, with boundary protection. Low free memory or a small contiguous block may come from load peaks, long-term fragmentation, or a leak, and a single sample cannot distinguish among them; keep the sampling time, firmware, board type, configuration, business load, and raw failure records.
Actual feature gains must be compared across six states — disabled, enabled with no instance, single instance, multiple instances, disable-takes-effect, and re-enabled — recording internal DRAM, contiguous block, DMA, PSRAM, task stacks, and peaks. Estimated budgets, object existence, host tests, and compile results cannot substitute for these measurements.
24-hour/7-day acceptance is based on the real, valid sample span of the same stable candidate with continuous business coverage. Device uptime, script process existence, login waits, screenshots, or export times cannot make up the measurement period; after recovery, previous disconnections, bad responses, or restarts must not be erased. These field items are still pending authorization and execution; see the Stability and Release Checklist.
When troubleshooting, first reduce duplicate open pages, save the lightweight state and existing logs, then export one diagnostic bundle when memory permits. Modifying features or migrating configuration must preserve user intent and backups; "automatically trimming large configurations" must not be a default deployment behavior. Anything involving outputs, RTC, wiring, or flashing must be confirmed against the specific device procedure.