Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lh fix individual read deduplication and limiting #195

Open
wants to merge 95 commits into
base: master
Choose a base branch
from
Open

lh fix individual read deduplication and limiting #195

wants to merge 95 commits into from

Commits on Nov 29, 2024

  1. [fix][build] Fix error "Element encoding is not allowed here" in pom.… 

    …xml (apache#23655)
    @lhotari
    lhotari authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    66a8f73 View commit details
    Browse the repository at this point in the history

  2. [improve][client] Replace NameUtil#generateRandomName with RandomStri… 

    …ngUtils#randomAlphanumeric (apache#23645)
    @geniusjoe
    geniusjoe authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    32b3ccf View commit details
    Browse the repository at this point in the history

  3. [improve] [pip] PIP-373: Add a topic's system prop that indicates whe… 

    …ther users have published TXN messages in before. (apache#23210)

[improve] [pip] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before. (apache#23210)
    @poorbarcode
    poorbarcode authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    3a50255 View commit details
    Browse the repository at this point in the history

  4. [fix][broker] Revert "[improve][client] Add log when can't add messag… 

    …e to the container (apache#23657)
    @Technoboy-
    Technoboy- authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    d1753ee View commit details
    Browse the repository at this point in the history

  5. [fix][broker]: support missing broker level fine-granted permissions (a… 

    …pache#23637)
    @mattisonchao
    mattisonchao authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    eb60d0a View commit details
    Browse the repository at this point in the history

  6. [improve] Upgrade OpenTelemetry library to 1.44.1 version (apache#23656)

    @lhotari
    lhotari authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    9ed44dd View commit details
    Browse the repository at this point in the history

  7. [improve][client] Reduce unshaded dependencies and shading warnings i… 

    …n shaded Java client modules (apache#23647)
    @lhotari
    lhotari authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    429f7de View commit details
    Browse the repository at this point in the history

  8. [fix][broker] support missing tenant level fine-granted permissions (a… 

    …pache#23660)
    @mattisonchao
    mattisonchao authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    280997e View commit details
    Browse the repository at this point in the history

  9. [fix][client] Fix race-condition causing doReconsumeLater to hang whe… 

    …n creating retryLetterProducer has failed (apache#23560)
    @hanmz
    hanmz authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    bf1f677 View commit details
    Browse the repository at this point in the history

  10. [improve][client] Enhance error handling for non-exist subscription i… 

    …n consumer creation (apache#23254)
    @Shawyeok
    Shawyeok authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    7fc88d6 View commit details
    Browse the repository at this point in the history

  11. [feat][broker] Implement allowBrokerOperationAsync in PulsarAuthoriza… 

    …tionProvider to avoid exception thrown (apache#23663)
    @mattisonchao
    mattisonchao authored Nov 29, 2024
    Configuration menu
    Copy the full SHA
    4603722 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2024

  1. Add solution to PulsarMockBookKeeper for intercepting reads

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    a61b426 View commit details
    Browse the repository at this point in the history

  2. Add validation to getManagedLedgerMaxReadsInFlightSizeInMB

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    a2de406 View commit details
    Browse the repository at this point in the history

  3. Don't divide by 0 if it can be avoided

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    29c8952 View commit details
    Browse the repository at this point in the history

  4. Estimate entry size for InflightReadsLimiter by keeping stats

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    cd2427d View commit details
    Browse the repository at this point in the history

  5. Limit replay messages by bytes size

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    6b63b32 View commit details
    Browse the repository at this point in the history

  6. Test limiting by bytes size

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    25f1c31 View commit details
    Browse the repository at this point in the history

  7. Test

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    ddc9fa9 View commit details
    Browse the repository at this point in the history

  8. Avoid exception which commonly happens

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    7df7a94 View commit details
    Browse the repository at this point in the history

  9. Add debug logging to InflightReadsLimiter

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    8f3f09c View commit details
    Browse the repository at this point in the history

  10. Clear the cache in the test

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    dd4bb66 View commit details
    Browse the repository at this point in the history

  11. Add logging

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    f774f43 View commit details
    Browse the repository at this point in the history

  12. Fix read limits for individual reads

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    9a78b8e View commit details
    Browse the repository at this point in the history

  13. Revert "Test" 

    This reverts commit 3ae9991.
    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    9280cc9 View commit details
    Browse the repository at this point in the history

  14. Reduce timeout

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    f452ba8 View commit details
    Browse the repository at this point in the history

  15. Apply cap for delayed messages too

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    bf54324 View commit details
    Browse the repository at this point in the history

  16. Improve asyncReplayEntries

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    cc3f530 View commit details
    Browse the repository at this point in the history

  17. Refactor to have a single replay method

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    124bf6e View commit details
    Browse the repository at this point in the history

  18. Optimize filtering deleted messages

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    6a1a431 View commit details
    Browse the repository at this point in the history

  19. deprecate synchronous replayEntries method

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    e7aade5 View commit details
    Browse the repository at this point in the history

  20. move asyncReplayEntries without sortEntries as a default method

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    c4836d0 View commit details
    Browse the repository at this point in the history

  21. Perform replay reads in ranges

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    5884e7b View commit details
    Browse the repository at this point in the history

  22. Make test to fail

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    83013df View commit details
    Browse the repository at this point in the history

  23. Fix disabled timeout in InflightReadsLimiter

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    041928f View commit details
    Browse the repository at this point in the history

  24. Add queuing to InflightReadsLimiter when the limit is reached

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    63629fb View commit details
    Browse the repository at this point in the history

  25. revisit

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    e14514c View commit details
    Browse the repository at this point in the history

  26. Ensure that cached entry readerIndex is not tied to the original one

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    89adfb6 View commit details
    Browse the repository at this point in the history

  27. Adjust InflightReadsLimiterTest

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    b4e4b2e View commit details
    Browse the repository at this point in the history

  28. Add timeout executor to InflightReadsLimiter

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    f9919cc View commit details
    Browse the repository at this point in the history

  29. Polish

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    dae77e5 View commit details
    Browse the repository at this point in the history

  30. Fix test

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    8466578 View commit details
    Browse the repository at this point in the history

  31. Instantiate ArrayList directly

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    235dc41 View commit details
    Browse the repository at this point in the history

  32. Support caching replayed entries

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    c719306 View commit details
    Browse the repository at this point in the history

  33. Start addressing the removal issue with a removal queue

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    d2d9356 View commit details
    Browse the repository at this point in the history

  34. Move EntryWrapper to upper level

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    4b99aee View commit details
    Browse the repository at this point in the history

  35. Move RangeCache to cache package

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    807f0c9 View commit details
    Browse the repository at this point in the history

  36. Start adding removal queue

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    dce251f View commit details
    Browse the repository at this point in the history

  37. Move towards adding removal queue changes

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    6e75ca9 View commit details
    Browse the repository at this point in the history

  38. More moves towards removal queue solution

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    39d0258 View commit details
    Browse the repository at this point in the history

  39. Handle removing by size in scheduled task so that blocking would be rare

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    cdc70b3 View commit details
    Browse the repository at this point in the history

  40. Remove unused code

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    cffa151 View commit details
    Browse the repository at this point in the history

  41. Update RangeCacheTest with removal queue

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    3c4af56 View commit details
    Browse the repository at this point in the history

  42. Fix imports

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    0098d3a View commit details
    Browse the repository at this point in the history

  43. Add license headers

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    55fe582 View commit details
    Browse the repository at this point in the history

  44. Handle all cache evictions by the same thread

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    a7aed9d View commit details
    Browse the repository at this point in the history

  45. Move RangeCacheRemovalCounters to top level

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    d1c28a7 View commit details
    Browse the repository at this point in the history

  46. Use removal queue in RangeCacheTest

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    308afe5 View commit details
    Browse the repository at this point in the history

  47. Handle eviction

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    3b5508a View commit details
    Browse the repository at this point in the history

  48. Handle adding atomically to removal queue and cache

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    4b53193 View commit details
    Browse the repository at this point in the history

  49. Improve javadoc

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    b329372 View commit details
    Browse the repository at this point in the history

  50. Use unbounded queue for removal queue since cache size is bounded by … 

    …size
    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    e0b2fc5 View commit details
    Browse the repository at this point in the history

  51. Refactor eviction when cache size exceeds evictionTriggerThreshold

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    4fb16e6 View commit details
    Browse the repository at this point in the history

  52. Start adding cacheEvictionByExpectedReadCount

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    35cafd2 View commit details
    Browse the repository at this point in the history

  53. Add CachedEntry

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    17a44af View commit details
    Browse the repository at this point in the history

  54. Activate cursor when consumers disconnect and connect

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    080b985 View commit details
    Browse the repository at this point in the history

  55. disable checkCursorsToCacheEntries when eviction by read count is ena… 

    …bled
    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    4f7426f View commit details
    Browse the repository at this point in the history

  56. Always keep cursors with connected consumers in "active" state when e… 

    …viction is based on expected read count
    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    ec7022c View commit details
    Browse the repository at this point in the history

  57. Add a way to find cursors before current cursor

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    3e02069 View commit details
    Browse the repository at this point in the history

  58. Pass predicate for deciding whether to cache or not

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    ebcec70 View commit details
    Browse the repository at this point in the history

  59. Reduce coupling to EntryImpl

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    d56b1d6 View commit details
    Browse the repository at this point in the history

  60. Extract abstract base class to be used for Entry implementations

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    e934e90 View commit details
    Browse the repository at this point in the history

  61. Introduce CachedEntry for eviction by read count implementation

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    80d9541 View commit details
    Browse the repository at this point in the history

  62. Remove generics type parameters from RangeCache since there's no need… 

    … to reuse the class with different types
    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    6500339 View commit details
    Browse the repository at this point in the history

  63. Add TODO about the solution for handling skipping of the entries

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    50bc822 View commit details
    Browse the repository at this point in the history

  64. Add note of replacing the use of peek

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    09464c1 View commit details
    Browse the repository at this point in the history

  65. update design

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    bb905eb View commit details
    Browse the repository at this point in the history

  66. Evaluate entry size when it gets added

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    bbbb7c4 View commit details
    Browse the repository at this point in the history

  67. Fix issue in getting the entry

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    197d038 View commit details
    Browse the repository at this point in the history

  68. Fix RangeCacheTest

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    eee7522 View commit details
    Browse the repository at this point in the history

  69. Remove generics

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    ff1add7 View commit details
    Browse the repository at this point in the history

  70. Add simple stash

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    4eae832 View commit details
    Browse the repository at this point in the history

  71. Add note

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    d3df21c View commit details
    Browse the repository at this point in the history

  72. Fix test that expects old defaults

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    b1d50ce View commit details
    Browse the repository at this point in the history

  73. Adapt test to new behavior

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    8e8cf68 View commit details
    Browse the repository at this point in the history

  74. Improve removal queue

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    731148c View commit details
    Browse the repository at this point in the history

  75. Fix EntryCacheManagerTest

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    4ca1b7f View commit details
    Browse the repository at this point in the history

  76. Support configuring ManagedLedgerConfig defaults for tests

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    952b2e1 View commit details
    Browse the repository at this point in the history

  77. Fix ManagedLedgerTests

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    336b472 View commit details
    Browse the repository at this point in the history

  78. Use waitForPendingCacheEvictions in EntryCacheManagerTest

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    d2bd347 View commit details
    Browse the repository at this point in the history

  79. Use Double.NaN instead of Double.POSITIVE_INFINITY

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    3470a0f View commit details
    Browse the repository at this point in the history

  80. Start adding expected read count solution

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    929942a View commit details
    Browse the repository at this point in the history

  81. Improve ManagedCursorContainerTest

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    20b262b View commit details
    Browse the repository at this point in the history

  82. Implement getNumberOfCursorsAtSamePositionOrBefore

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    1326dc9 View commit details
    Browse the repository at this point in the history

  83. Implement EntryReadCountHandler

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    c0750c3 View commit details
    Browse the repository at this point in the history

  84. Implement decreasing read count on release

    @lhotari
    lhotari committed Dec 2, 2024
    Configuration menu
    Copy the full SHA
    f58ad8c View commit details
    Browse the repository at this point in the history