InterProcessLock does not lock

[jumpojoy]

It seems that flock and lockf works differently, and lockf does lock file as expected.

There is test code I'm using

import fasteners
import sys
import time
import threading
import fcntl
from oslo_concurrency import lockutils

def test_concurrency(i):
  with lockutils.lock(name='lock-c.file',
            lock_path='/var/lib/openstack/locks/',
            external=True):
    print(time.time())
    print(i)
    time.sleep(10)

def test_concurrency_ext(i):
  with lockutils.external_lock(name='lock-c.file',
                 lock_file_prefix='ext-lock',
                 lock_path='/var/lib/openstack/locks/'):
    print(time.time())
    print(i)
    time.sleep(10)


def test_fcnl_flock(i):

    file_path = "example.txt"
    with open(file_path, "a") as file:
        fcntl.flock(file.fileno(), fcntl.LOCK_EX)

        print(time.time())
        print(i)
        time.sleep(10)

        fcntl.flock(file.fileno(), fcntl.LOCK_UN)

def test_fcnl_lockf(i):

    file_path = "example.txt"
    with open(file_path, "a") as file:
        fcntl.lockf(file.fileno(), fcntl.LOCK_EX)

        print(time.time())
        print(i)
        time.sleep(10)

        fcntl.flock(file.fileno(), fcntl.LOCK_UN)

for tf in [test_concurrency, test_concurrency_ext, test_fcnl_flock, test_fcnl_lockf]:
  print(tf)
  t1 = threading.Thread(target=tf, args=[1])
  t2 = threading.Thread(target=tf, args=[2])
  t1.start()
  t2.start()
  t1.join()
  t2.join()

And it produce following output on Ubuntu 22.04 and on macOS

python3 1.py 
<function test_concurrency at 0x7f4c509fc1f0>
1721200767.7912707
1
1721200777.8025727
2
<function test_concurrency_ext at 0x7f4c4feebaf0>
1721200787.8146389
2
1721200787.8147857
1
<function test_fcnl_flock at 0x7f4c4feebb80>
1721200797.8260574
1

1721200807.832955
2
<function test_fcnl_lockf at 0x7f4c4feebc10>
1721200817.8442693
1
1721200817.8444033
2

Which basically shows that lockutils.external_lock and test_fcnl_lockf does not lock on file as expected. Please help to fix this issue.

[jumpojoy]

https://stackoverflow.com/questions/28470246/python-lockf-and-flock-behaviour

[psarka]

Is this related to fasteners?

The title of the issue mentions InterProcessLock, but your snippet does not use InterProcessLock or any other fasteners functions. Can you please make a small example which illustrates deficiency of fasteners library?

Also, one thing of note, some interprocess locks do not work with threads. If you are looking for a process lock, test it with multiple processes, not multiple threads.

[jumpojoy]

@psarka thanks for quick reply!

I'm sorry that I didn't provide enough information, external_lock from lockutils uses InterProcessLock [0] from fasteners. And it seems InterProcessLock is not thread safe which I discovered later [1]. Do you think adding docstrings to both fasteners and oslo.concurrency that will highlight this info will be useful?

[0] https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L208
[1] https://stackoverflow.com/questions/28470246/python-lockf-and-flock-behaviour