Doubts about onConnect/onDisconnect logic

[popovitsj]

I have some doubts about the implementation of the onConnect/onDisconnect logic.

The flow, as I understand it, is as follows:

1. When a user connects, a message is sent to SQS which triggers a lambda which queries all currently connected users and sends a status update for the connected user to all users.
2. When a user disconnects, the same happens.
3. When a user is in the connections table, but has lost its connection (410 http code), the connection gets deleted. In this case, this disconnect is not broadcast, so the user will appear as online to other users.

Aside from the issue in point 3, the main issue I'm seeing here is when we have a lot of users, so the lambda for broadcasting is a heavy operation, and we have a user with a laggy connection who is repeatedly connecting and disconnecting. Aside from the cost of constantly broadcasting this, it seems to me we also can't guarantee that the lambda for broadcasting the status change will be started in order, thus causing a wrong state on the client.

My question is basically, how can we improve this logic to make it more robust?

[popovitsj]

In an attempt to answer my own question, I guess we could make the statusQueue a fifo queue and set concurrency to 1?

[santatamas]

Love it, very valid points you made. Regarding point 3), you could send an OFFLINE status message to users when you detect a stale connection after you deleted the item from DynamoDB. And you're right, a FIFO queue could be used to fix the potential ordering issue.

I'll be very clear here - this solution does not scale well. Manually maintaining the connections table (and running scan operations on every broadcast) is going to be very expensive with large amount of connected users. It'll also be slow, because you have to send an API request per user per message. Maybe I should add a section to the readme to make this more visible - this sample is meant to demonstrate the usage of the websockets API. But the service design itself is opinionated as it decouples the bi-directional nature of websockets from the request/response based backend implementation.

For large-scale chat applications, you should either use AWS AppSync, or IoT Core (MQTT over websockets).

[popovitsj]

What would be considered large scale here? Are we still fine with say, a 1000 connections?

Is there also an example application for the AppSync approach?

[santatamas]

This is a tricky question - it all depends. If you check the service limits you can see that the service itself can scale to those numbers. If let's say you use the chat functionality in a mostly 1-1 manner (e.g. private chats), then this solution will scale to much larger numbers easily. If you mainly use it for broadcasting (e.g. people are in the same room that counts as broadcasting!) and thousands of people are actively messaging the room, then eventually your broadcast API calls will get throttled.

You can find an AppSync-based chat example here.

Let me know if this helps!