Skip to content

Latest commit

 

History

History
107 lines (96 loc) · 6.51 KB

Message_Queue.md

File metadata and controls

107 lines (96 loc) · 6.51 KB
Raw

Message Queue (MQ)

Pros & Cons

Pros

  • Asynchronous
    • A producer doesn’t wait for the message to be processed.
  • Decoupling
    • Producers are completely unaware of consumers (Don't have to know the IPs and ports of receivers).
    • Producers or consumers can be modified and deployed independently.
  • Buffering/Rate limiting
    • It can act as a buffer if the consumer is unavailable or overloaded.
  • Fan-out
    • The producer simply drops the message on the queue, and the multiple consumers process the message based on their needs at their own pace.

Cons

  • Single point of failure
    • The message queue can be a potential single point of failure.
  • Increase complexity
    • Increase system complexity and operational complexity.

Selection

Factors

  • Supported programming languages

  • Message ordering

    • The order of messages can be preserved by the queue or not.

  • Message retention/expiration

    • The limited period of time the message will be retained in the queue before being discarded.

  • Delivery guarantees

    At-least-once Exactly-once At-most-once
    Concept Every message from the source system will reach its destination, but sometimes retries will cause duplicates Every message from the source system will reach the destination with no possibility for loss or duplication Every message from the source system may reach the destination with no possibility for duplication
    Delivey times >= 1 = 1 <= 1
    Use case
    • Data duplication is not a big issue
    • Deduplication is possible
    • Financial-related use cases (payment, trading, accounting, etc.)
    • Duplication is not acceptable
    • The downstream service or third party doesn’t support idempotency.
    • A small amount of data loss is acceptable (Monitoring metrics)
    Diagram Screenshot 2023-10-04 at 4 56 20 PM Screenshot 2023-10-04 at 4 56 34 PM Screenshot 2023-10-04 at 4 56 08 PM

  • Availability

  • Scalability

  • Latency

Comparison

RabbitMQ Kafka
Protocol AMQP Binary protocol over TCP
Message routing Support multiple routing rules Single
Message consumption push pull
Message ordering Ordered at queue level Ordered at partition level (Not topic level)
Message deleteion Delete on ACK Delete on retention period expires
Message deleyed delivery Support Not support
Message delivery gurantees
  • At-most-once
  • At-least-once
    		•
  • At-most-once
  • At-least-once
  • Exactly-once (by idempotent producers, transactional consumers)
    		•
    Message persistence Less Strong (persisting messages to disk)
    Message pileup pressure Cannot handle very well
    Client API
  • Java
  • Ruby
  • JavaScript
  • Go
  • C
  • Swift
  • Spring
  • Elixir
  • PHP
  • .NET
    		•
  • Java
  • Ruby
  • Python
  • Node.js
    		•
    Federated queues Yes No
    Complexity Less More
    Throughput 20K / sec Millions / sec
    Scaling strategies Mostly vertical scaling Horizontal scaling
    Learning curve Gentle Steep
    Ecosystem Smaller Larger

    How to choose

    • Choose RabbitMQ if
      • Multiple routing mechanisms
      • Low complexity and flat learning curve
      • Lightweight
      • For medium or smaller companies
    • Choose Kafka if
      • Higher throughput
      • Persistence of messages
      • For large companies

    Common problems and solutions

    How to guarantee message order

    If a message queue cannot preserve the order of messages on its own, how to keep messages delivered in order (FIFO).

    • Solution 1: Add sequencing information into each message.

    How to handle duplicated message

    If there are duplicate messages in a message queue, how the receiver handles duplicate messages (Achieve exactly-once delivery guarantee from at-least-once delivery guarantee).

    • Solution 1: Make the receiver idempotent.
      • The receiver can process the same message multiple times without no additional effect.
    • Solution 2: Track each message which has been processed into a data store and discard duplicates.
      • Add a unique identifier into each message.
      • Create a deduplication table on the consumer side to record which message has been processed.
      • Add an unique index on the message identifier column in the deduplication table (the table will reject to insert a record whose message identifier value already exists in the table).

    Use cases

    References