Why Do Devices Keep Going Offline? A Troubleshooting Note on a Mass Disconnection Incident

One Monday at 3 a.m., the on-call alert group exploded: the number of online devices dropped from 5,200 to 2,100, then slowly climbed back within half an hour. No device reboots, no platform release — so where was the problem? This is a record of the complete troubleshooting process.

Step 1: Look at the Time Pattern

Plotting the offline events on a timeline, we found the disconnections were concentrated between 03:00 and 03:05, and almost all offline devices were in the coverage area of the same batch of base stations. This ruled out platform-side issues — if the Broker or backend services were abnormal, the disconnections would have been unrelated to device geography.

Step 2: Understand the Keep-alive Mechanism of MQTT Persistent Connections

The MQTT client and Broker maintain a persistent connection via keepAlive: the client promises to send at least one packet within that interval (sending a PINGREQ when there is no business message), and the Broker declares the connection dead if it receives nothing for 1.5 times the keepAlive interval.

But there is a well-known trap: carrier NAT session timeouts are often shorter than keepAlive. On 4G networks, idle NAT sessions are typically reclaimed after 5–15 minutes. If keepAlive is set to 20 minutes, the NAT session has long been torn down while the device remains unaware — it only discovers the connection is dead the next time it tries to send data, which shows up as "fake online".

Step 3: Find the Real Culprit

After contacting the carrier, we confirmed: at 03:00 that night, that batch of base stations performed a batch cleanup of core network sessions (a routine maintenance operation), forcibly reclaiming all NAT sessions that had been idle beyond the threshold. Devices with a reasonable keepAlive setting (60 seconds) detected the disconnection within 90 seconds and reconnected automatically; but one batch of our old firmware had keepAlive set to 600 seconds, and had to wait 15 minutes before discovering it was "dead" — those were the 3,000 devices.

Fixes and Prevention

1. Firmware OTA: unify the keepAlive of old devices down to 60–120 seconds;
2. Reconnection strategy: exponential backoff + random jitter, to avoid thousands of devices reconnecting in the same second and crashing the Broker (a reconnection storm);
3. Last Will and Testament (LWT): on abnormal disconnection, the Broker publishes the offline status on the device's behalf, giving the platform second-level offline awareness;
4. Platform monitoring: set an alert threshold on "number of devices going offline per unit time" instead of only watching the total online count — this incident's alert was triggered by the total count change, but locating the cause took two hours.

Summary

With persistent connection problems, it is the norm that the device, the network, and the platform all "look normal". The troubleshooting mantra: first cluster by dimension (region / batch / firmware version), then follow the protocol state machine to find who is lying.

Related Documentation

• MQTT Access: MQTT connection parameters and data sending/receiving between devices and the platform.
• Device Management: view device online/offline status and device logs — the entry point for troubleshooting disconnection issues.
• Alert Center: configure alerts on metrics such as "number of devices going offline per unit time" to achieve second-level offline awareness.