Unexpected matching with sequence number

[allenhula]

Recently I found my device will lose some messages. After enabling verbose log and digging into, I found it is due to unexpected matching with sequence number.
Here are my analysis:

1. When the issue happening, I only found a PUBREL packet sent to device from Protocol Gateway. Normally, it should send PUBLISH packet at first.

Publish Cloud to device	Message Id received is 1ed186d2-27cd-4ca0-9e7c-fa1b24fcc1d1
Publish Cloud to device	Message Id received
Publish Cloud to device	Publish message received at 09-28-2017 03:04:56.843 for Cloud to Device transmission.
PubRelPacket[Type=PUBREL, QualityOfService=AtLeastOnce, Duplicate=False, Retain=False]	PacketPUBREL sent at 09-28-2017 03:04:56.859.

2. I checked the code, found below condition:

async Task PublishToClientQos2Async(IChannelHandlerContext context, MessageWithFeedback messageWithFeedback, PublishPacket packet, string[] userContext)
        {
            int packetId = packet.PacketId;
            IQos2MessageDeliveryState messageInfo = await this.qos2StateProvider.GetMessageAsync(this.identity, packetId);
 
            if (messageInfo != null && messageWithFeedback.Message.SequenceNumber != messageInfo.SequenceNumber)
            {
                await this.qos2StateProvider.DeleteMessageAsync(this.identity, packetId, messageInfo);
                messageInfo = null;
            }
 
            if (messageInfo == null)
            {
                await this.publishPubRecProcessor.SendRequestAsync(context, packet, new AckPendingMessageState(messageWithFeedback, packet));
                CommonEventSource.Log.Info($"Packet {packet.PacketType} sent.", packet.ToString(), userContext);
            }
            else
            {
                await this.PublishReleaseToClientAsync(context, packetId, messageWithFeedback.FeedbackChannel, messageInfo, PreciseTimeSpan.FromStart);
            }
        }

3. So it should goes with this condition that

else
{
      await this.PublishReleaseToClientAsync(context, packetId, messageWithFeedback.FeedbackChannel, messageInfo, PreciseTimeSpan.FromStart);
}

Which means messageInfo != null && messageWithFeedback.Message.SequenceNumber != messageInfo.SequenceNumber.
4. Then I checked my state provider which is storage table, I founded below records with device id as filtering condition

PartitionKey	RowKey	Timestamp	LastModified	MessageNumber
7	DEVICE001_33002	2017-05-03T10:58:07.765Z	2017-05-03T10:58:09.450Z	233
7	DEVICE001_33944	2017-06-19T05:38:47.198Z	2017-06-19T05:38:46.826Z	1175
1	DEVICE001_32786	2017-08-17T00:42:48.145Z	2017-08-17T00:42:49.222Z	17
7	DEVICE001_32795	2017-08-17T01:11:41.114Z	2017-08-17T01:11:41.286Z	26

5. There were obsolete records, and defenitely not the right state for messages I am sending now because current date is 2017-09-28. And here message number is the value of sequence number. So seems sequence number recycled to reach a duplicate one with value in table.