High CPU usage after implemented State Recovery

[yuribraskulki-anotaai]

I'm facing a problem in a production project with CPU usage after changes on the server to use the State recovery

The only changes was the use of the State Recovery, and change of the redis adapter to redis streams adapter

The server instance:

const io = new socketIO.Server({
            cors: { origin: "*", methods: ["*"] },
            pingInterval: 15000,
            pingTimeout: 30000,
            allowEIO3: true,
            transports: ["websocket"],
            connectionStateRecovery: {
                maxDisconnectionDuration: 15 * 60 * 1000, // 15 minutes as miliseconds
            },
        })

Before:

Node 14
"socket.io": "4.0.1"
"socket.io-redis": "6.1.1"

With state Recovery:

Node 20
"socket.io": "^4.8.1",
"socket.io-redis-streams-adapter": "^0.1.1"

So with the changes the CPU usage of the websocket pod's was reaching its max, so we scaled them to support the needs, but what is strange is the difference between them, and the peak of the CPU usage always come with a peak of disconnections, that made me think that the problem is with something with the state recovery itself

Before the state recovery the peak of CPU usage was 0.24 core/pod
Now it is reaching 1.3 core/pod with 4 more pod's

Today we made a staging test to validate if the reconnection was the problem, and with 100 rooms, and no reconnects the usage was stable, after with reconnects the CPU usage got 3 times higher

I would like to know if someone faced the same problem or something like it, and if this high usage of the CPU is expected with the recovery

The tests that we ran today was with this script

const io = require("socket.io-client")

const jwt = require("jsonwebtoken")
const { randomUUID } = require("crypto")

function testSocket(roomQuantity, dummyEventsQuantity, secondsUntilDisconnect, cycles, isStg = false) {
    const localUrl = "http://localhost:3000/"
    const stgUrl = "OurStagingWebsocketUrl"

    const roomsList = []
    for (let i = 0; i < roomQuantity; i++) {
        roomsList.push(randomUUID())
    }

    let totalEvents = 0

    roomsList.forEach((roomId) => {
        const token = jwt.sign({ room: roomId }, jwtKey)
        let currentCycle = 0

        const url = isStg ? stgUrl : localUrl

        const socket = io(url, {
            query: { token },
            transports: ["websocket"],
        })

        socket.on("connect", () => {
            console.log("Connected to the server")

            console.log("recovered: ", socket.recovered)

            for (i = 0; i < dummyEventsQuantity; i++) {
                socket.emit("subscribe", { dummyNumber: i + 1, roomId })
                totalEvents++
                console.log("Summary of events: ", { roomEventNumber: i + 1, totalEvents, roomId })
            }

            if (secondsUntilDisconnect) {
                setTimeout(() => {
                    if (socket.io.engine) {
                        socket.io.engine.close()
                    }
                }, secondsUntilDisconnect * 1000)
            }
        })

        socket.on("connect_error", (error) => {
            console.error("Connection Error:", error)
        })

        socket.on("disconnect", () => {
            console.log("Disconnected from the server: ", currentCycle)
            if (currentCycle === cycles) {
                console.log("Ending Process")
                process.exit(1)
            }

            currentCycle++
        })
    })

    console.log("Rooms used ", roomsList)
}

// roomQuantity, dummyEventsQuantity, secondsUntilDisconnect, cycles, isStg
testSocket(10, 0, 120, 5, false)