NGINX Load Balancing for FMQ Cluster

1. Features and Advantages

As a reverse proxy server located in front of the MQTT servers, it initiates connection requests to the FMQ cluster on behalf of clients and processes and returns responses, hiding cluster details and improving system maintainability and scalability.
It can terminate SSL-encrypted MQTT connections, offloading encryption and decryption work from the FMQ cluster, which improves performance, simplifies certificate management, and enhances security.
It provides flexible load-balancing policies, such as sticky load balancing, distributing requests to FMQ nodes according to different policies to share traffic and requests, improving performance and reliability.

2. Installing and Using NGINX

(1) Prerequisites

Create a cluster consisting of 3 FMQ nodes (fmq1-cluster.fmq.io, fmq2-cluster.fmq.io, fmq3-cluster.fmq.io, MQTT TCP port 1883, MQTT WebSocket port 8083).

(2) Installing NGINX

Environment requirements
: Install the GNU C and C++ compilers, the PCRE library, the zlib compression library, and the OpenSSL library
bash
```
sudo apt-get update
sudo apt-get install build-essential libpcre3-dev zlib1g-dev libssl-dev
```
1
2

Download the source code

bash

wget https://nginx.org/download/nginx-1.24.0.tar.gz

Extract the source package

bash

tar -zxvf nginx-1.24.0.tar.gz
cd nginx-1.24.0

Start compiling:

plaintext

make

Install

bash

sudo make install
sudo ln -s /usr/local/nginx/sbin/nginx /usr/local/bin/nginx

Start using
- Check the configuration file:
  bash
```
sudo nginx -t
```
  1
- Start NGINX:
  bash
```
sudo nginx
```
  1
- Reload the configuration:
  bash
```
sudo nginx -s reload
```
  1
- Stop NGINX:
  bash
```
sudo nginx stop
```
  1

4. Configuring NGINX Reverse Proxy and Load Balancing

(1) Configuring MQTT Reverse Proxy

bash

stream {
  upstream mqtt_servers {
    server fmq1-cluster.fmq.io:1883 max_fails=2 fail_timeout=10s;
    server fmq2-cluster.fmq.io:1883 down;
    server fmq3-cluster.fmq.io:1883 backup;
  }

  server {
    listen 1883;
    proxy_pass mqtt_servers;
    proxy_protocol on;
    proxy_connect_timeout 10s;   
    proxy_timeout 1800s;
    proxy_buffer_size 3M;
    tcp_nodelay on;       
  }
}

(2) Configuring MQTT SSL Reverse Proxy

bash

stream {
  upstream mqtt_servers {
    server fmq1-cluster.fmq.io:1883;
    server fmq2-cluster.fmq.io:1883;
  }

  server {
    listen 8883 ssl;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_certificate /usr/local/NGINX/certs/fmq.pem;
    ssl_certificate_key /usr/local/NGINX/certs/fmq.key;
    ssl_verify_depth 2;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!MD5;
    proxy_pass mqtt_servers;
    proxy_protocol on;
    proxy_connect_timeout 10s;   
    proxy_timeout 1800s;
    proxy_buffer_size 3M;
    tcp_nodelay on;
  }
}

(3) Configuring MQTT WebSocket Reverse Proxy

bash

http {
  upstream mqtt_websocket_servers {
    server fmq1-cluster.fmq.io:8083;
    server fmq2-cluster.fmq.io:8083;
  }

  server {
    listen 80;
    server_name mqtt.example.com;

    location /mqtt {
      proxy_pass http://mqtt_websocket_servers;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
      proxy_buffering off;
      proxy_connect_timeout 10s;        
      proxy_send_timeout 3600s;            
      proxy_read_timeout 3600s;            
      proxy_set_header Host $host;            
      proxy_set_header X-Real-IP $remote_addr;            
      proxy_set_header REMOTE-HOST $remote_addr;            
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }
}

(4) Configuring MQTT WebSocket SSL Reverse Proxy

bash

http {
  upstream mqtt_websocket_servers {
    server fmq1-cluster.fmq.io:8083;
    server fmq2-cluster.fmq.io:8083;
  }

  server {
    listen 443 ssl;
    server_name mqtt.example.com;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_certificate /usr/local/NGINX/certs/fmq.pem;
    ssl_certificate_key /usr/local/NGINX/certs/fmq.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!MD5;
    location /mqtt {
      proxy_pass http://mqtt_websocket_servers;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
      proxy_set_header Host $host;            
      proxy_set_header X-Real-IP $remote_addr;            
      proxy_set_header REMOTE-HOST $remote_addr;            
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;      
      proxy_buffering off;
    }
  }
}

(5) Configuring Load-Balancing Policies

Round Robin

: The default policy; distributes requests to backend servers in turn.

bash

upstream backend_servers {
  server fmq1-cluster.fmq.io:1883;
  server fmq2-cluster.fmq.io:1883;
  server fmq3-cluster.fmq.io:1883;
}

Weighted Round Robin

: Assigns different weights to nodes to influence the request distribution ratio.

bash

upstream backend_servers {
  server fmq1-cluster.fmq.io:1883 weight=3;
  server fmq2-cluster.fmq.io:1883 weight=2;
  server fmq3-cluster.fmq.io:1883 weight=1;
}

IP Hash

: Computes a hash from the client IP address and assigns requests to a fixed backend server.

bash

upstream backend_servers {
  ip_hash;
  server fmq1-cluster.fmq.io:1883;
  server fmq2-cluster.fmq.io:1883;
  server fmq3-cluster.fmq.io:1883;
}

Least Connections

: Distributes requests to the server with the fewest current connections.

bash

upstream backend_servers {
  least_conn;
  server fmq1-cluster.fmq.io:1883;
  server fmq2-cluster.fmq.io:1883;
  server fmq3-cluster.fmq.io:1883;
}

5. Using NGINX Plus to Optimize FMQ Deployment

(1) Configuring MQTT Sticky Session Load Balancing

bash

stream {
  mqtt_preread on;

  upstream backend_servers {
      hash $mqtt_preread_clientid consistent;
      server fmq1-cluster.fmq.io:1883;
      server fmq2-cluster.fmq.io:1883;
      server fmq3-cluster.fmq.io:1883;
  }

  server {
      listen 1883;
      proxy_pass backend_servers;
      proxy_connect_timeout 1s;
  }
}

(2) Configuring the Client ID Replacement Feature

bash

stream {
    mqtt on;

    server {
        listen 1883 ssl;
        ssl_certificate /etc/NGINX/certs/fmq.pem;
        ssl_certificate_key /etc/NGINX/certs/fmq.key;
        ssl_client_certificate /etc/NGINX/certs/ca.crt;      
        ssl_session_cache shared:SSL:10m;
        ssl_verify_client on;
        proxy_pass 10.0.0.113:1883;
        proxy_connect_timeout 1s;  

        mqtt_set_connect clientid $ssl_client_serial;
    }
}

6. Optimizing NGINX Performance and Enabling Monitoring

(1) Basic NGINX Configuration Adjustments

bash

worker_processes auto;

events {
  worker_connections 20480;
}

(2) NGINX Multi-NIC Reverse Proxy for Massive Connections

bash

stream {
  split_clients "$remote_addr$remote_port" $multi_ip {        
    20% 10.211.55.5;        
    20% 10.211.55.20;        
    20% 10.211.55.21;        
    20% 10.211.55.22;        
    * 10.211.55.23;    
  }

  upstream mqtt_servers {
    server fmq1-cluster.fmq.io:1883;
    server fmq2-cluster.fmq.io:1883;
  }

  server {
    listen 1883;

    proxy_pass mqtt_servers;
    proxy_bind $multi_ip;
  }
}

(3) NGINX Status Monitoring

bash

http {
  server {
    listen 8888;
    
    location /status {            
      stub_status on;            
      access_log off;            
    }
  }
}

7. Appendix: Main Parameter Descriptions

Parameter	Description
proxy_protocol	Enables the PROXY protocol so that FMQ can obtain the real client IP
proxy_pass	Defines the backend MQTT server address; client requests are forwarded here
proxy_connect_timeout	Timeout for establishing a connection with the backend; the connection attempt is aborted on timeout
proxy_timeout	Backend server timeout; the connection is closed if there is no response
proxy_buffer_size	Buffer size for storing backend data; handles large data streams
tcp_nodelay	Enables TCP_NODELAY, disabling Nagle's algorithm to reduce latency
ssl_session_cache	Configures the SSL session cache to speed up the handshake process; e.g., shared:SSL:10m is a 10 MB cache
ssl_session_timeout	Sets the SSL session timeout to 10 minutes; cleared if not reused
ssl_certificate	Specifies the SSL certificate file path that proves the server's identity
ssl_certificate_key	Specifies the certificate private key file path
ssl_protocols	Specifies the allowed SSL/TLS protocol versions
ssl_ciphers	Configures the allowed encryption algorithms (cipher suites); e.g., HIGH:!aNULL:!MD5 excludes insecure options
ssl_client_certificate	Specifies the CA certificate path used to verify client certificates
ssl_verify_client	Enables client certificate verification, requiring clients to present a valid certificate
ssl_verify_depth	Sets the client certificate verification depth; e.g., 1 means verifying only one level

NGINX Load Balancing for FMQ Cluster ​

1. Features and Advantages ​

2. Installing and Using NGINX ​

(1) Prerequisites ​

(2) Installing NGINX ​

4. Configuring NGINX Reverse Proxy and Load Balancing ​

(1) Configuring MQTT Reverse Proxy ​

(2) Configuring MQTT SSL Reverse Proxy ​

(3) Configuring MQTT WebSocket Reverse Proxy ​

(4) Configuring MQTT WebSocket SSL Reverse Proxy ​

(5) Configuring Load-Balancing Policies ​

5. Using NGINX Plus to Optimize FMQ Deployment ​

(1) Configuring MQTT Sticky Session Load Balancing ​

(2) Configuring the Client ID Replacement Feature ​

6. Optimizing NGINX Performance and Enabling Monitoring ​

(1) Basic NGINX Configuration Adjustments ​

(2) NGINX Multi-NIC Reverse Proxy for Massive Connections ​

(3) NGINX Status Monitoring ​

7. Appendix: Main Parameter Descriptions ​

NGINX Load Balancing for FMQ Cluster

1. Features and Advantages

2. Installing and Using NGINX

(1) Prerequisites

(2) Installing NGINX

4. Configuring NGINX Reverse Proxy and Load Balancing

(1) Configuring MQTT Reverse Proxy

(2) Configuring MQTT SSL Reverse Proxy

(3) Configuring MQTT WebSocket Reverse Proxy

(4) Configuring MQTT WebSocket SSL Reverse Proxy

(5) Configuring Load-Balancing Policies

5. Using NGINX Plus to Optimize FMQ Deployment

(1) Configuring MQTT Sticky Session Load Balancing

(2) Configuring the Client ID Replacement Feature

6. Optimizing NGINX Performance and Enabling Monitoring

(1) Basic NGINX Configuration Adjustments

(2) NGINX Multi-NIC Reverse Proxy for Massive Connections

(3) NGINX Status Monitoring

7. Appendix: Main Parameter Descriptions