instances not up to date status list

[djay]

We are using load-balancing across instances with 2 threads each, 4 on each server. I'm getting some strange errors with status updates not appearing. Either directly after entering a status or if refreshing the microblog page several times as it round-robins through the various zope instances. Eventually the instances all get up to date but it takes a few minutes.

[djay]

It looks like the issue was related to the two servers have clocks out by 3 min. So one server was refusing to show future updates. Perhaps the code could be made more robust by when showing the latest updates, to include future updates to handle clock problems?

[gyst]

On 18/06/14 13:26, Dylan Jay wrote:

It looks like the issue was related to the two servers have clocks out
by 3 min. So one server was refusing to show future updates. Perhaps the
code could be made more robust by when showing the latest updates, to
include future updates to handle clock problems?

Thanks for the bug report, Dylan.

I checked the code but there's no filter that excludes future status
updates.

More probably this has to do with the in-memory batch queing mechanism
and is possibly related to #1.

It's difficult to find and fix these kind of Heisenbugs without a
reproducible test case.

Anything you can add to diagnose this would be helpful. If you see this
recurring it would be informative if you could disable the batch queing
temporarily to see if that removes the issue or not. You can do that by
setting plonesocial.microblog.statuscontainer.MAX_QUEUE_AGE=0.

Please don't do that if your traffic exceeds 20 statusupdate insertions
per second or you'll get commit conflicts.

:*CU#

 Guido Stevens  |  +31.43.3618933  |  http://cosent.nl

 s o c i a l   k n o w l e d g e   t e c h n o l o g y

[djay]

Syncing the time of the servers solved the problem. I don't see how it
could be a write queue problem since the writes were visible on some of the
instances. The code for getting the keys show inc max and min does involve
time.time so I suspect that is involved.