Allow for using and working with offset locks #10
Conversation
harlowja
force-pushed
the
offset-locks
branch
2 times, most recently
from
a1c4f13
to
498bdca
Compare
|
We would find this feature very useful since we are looking to create many locks (i.e. a lock per id). |
|
So currently I was looking at waiting for #26 (comment) (see comment here). |
|
Is there any reason why this PR has not been merged? I visited this issue a year ago and I ended up using thousands of lock files in my project. I really need to address this problem now but the PR is still not merged. |
|
Only reason was I was waiting for someone to review :) |
harlowja
force-pushed
the
offset-locks
branch
5 times, most recently
from
a6cbbd4
to
5cfeee4
Compare
Offset locks allow for lock inside a single file using byte offsets, this avoids having to create hundreds of lock files for each use case, and instead allows for an application to be smarter, and shard locks from a single file for its use-cases (assuming the file can pick a large enough file with enough offsets to satisfy its needs). This makes it much easier to cleanup lock files, track them and know when to delete them (because deleting per-lock lock files is hard to do when an application is always online, because knowing when to delete a lock file is a non-trivial problem when an application has no interface to tell which locks are alive or dead and which are safe to delete).
There was a problem hiding this comment.
Hello,
Some feedback from my side.
I guess you need to adapt the interprocess_locked function too to allow user to use offset through the decorator approach too.
I mean this decorator use the InterProcessLock which represents _FcntLock and _WindowsLock under the hood.
Only _FcntLock really handle offsets but _WindowsLock inherit from _InterProcessLock so the_trylock and _unlock methods both accept the use_offset params.
In the nt case (Windows) offset features will be ignored due to the SUPPORTS_OFFSETS = False but with posix environments it can be useful to activate this kind of features through the decorator approach.
Maybe in the windows side it can be useful to log some info/debug messages to the logger to inform user that even if the offset is not None it will be ignored, to avoid misleading.
Thoughts?
I started to play with these changes few days ago through a direct usage and also through the oslo.concurrency scope so I will surely send more comments soon.
|
Hello, I designed some pocs/tools to test these changes and to help other to review/test it. Please find them here (https://github.com/4383/oslo.concurrency.lab). For the moment this poc test these changes outside of the oslo scope (direct use of fasteners).
You can a look to them for more details and infos. Also for more infos on how to use this POC you can read the related doc: Except my previous comment these changes look fine for the moment. Now I will spend time to test this through the oslo scope and implement some related pocs. |
|
Any feedbacks related to my previous messages? |
Offset locks allow for lock inside a single file
using byte offsets, this avoids having to create hundreds
of lock files for each use case, and instead allows for
an application to be smarter, and shard locks from a single
file for its use-cases (assuming the file can pick a large
enough file with enough offsets to satisfy its needs).
This makes it much easier to cleanup lock files, track them
and know when to delete them (because deleting per-lock lock
files is hard to do when an application is always online, because
knowing when to delete a lock file is a non-trivial problem when
an application has no interface to tell which locks are alive or
dead and which are safe to delete).