Improvement/bb 580 circuit breaker alert #2516

benzekrimaha · 2024-06-19T14:10:40Z

Linked Issue :
https://scality.atlassian.net/browse/BB-580

This is what the alert looks like on the platform :

bert-e · 2024-06-19T14:10:43Z

Hello benzekrimaha,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2024-06-19T14:10:53Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

codecov · 2024-06-19T14:11:31Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.00%. Comparing base (c8f5475) to head (7788e47).

Additional details and impacted files

Files	Coverage Δ
lib/CircuitBreaker.js	`100.00% <100.00%> (ø)`

... and 5 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	`66.11% <ø> (ø)`
Core Library	`74.21% <100.00%> (-0.05%)`	⬇️
Ingestion	`69.24% <ø> (+0.26%)`	⬆️
Lifecycle	`74.54% <ø> (ø)`
Oplog Populator	`75.90% <ø> (ø)`
Replication	`57.22% <ø> (-0.04%)`	⬇️
Bucket Scanner	`85.76% <ø> (ø)`

@@                 Coverage Diff                 @@
##           development/8.6    #2516      +/-   ##
===================================================
- Coverage            69.00%   69.00%   -0.01%     
===================================================
  Files                  192      192              
  Lines                12678    12681       +3     
===================================================
+ Hits                  8748     8750       +2     
- Misses                3920     3921       +1     
  Partials                10       10

Flag	Coverage Δ
api:retry	`9.75% <0.00%> (-0.01%)`	⬇️
api:routes	`9.65% <0.00%> (-0.01%)`	⬇️
bucket-scanner	`85.76% <ø> (ø)`
ingestion	`12.64% <66.66%> (+0.04%)`	⬆️
lib	`7.62% <33.33%> (+<0.01%)`	⬆️
lifecycle	`19.56% <66.66%> (+0.01%)`	⬆️
notification	`0.89% <0.00%> (-0.01%)`	⬇️
replication	`19.20% <66.66%> (-0.03%)`	⬇️
unit	`42.12% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

lib/CircuitBreaker.js

monitoring/ingestion/ingestion-producer-alert.yaml

francoisferrand · 2024-06-24T06:55:17Z

lib/CircuitBreaker.js

@@ -46,14 +46,24 @@ const circuitBreakerGauge = ZenkoMetrics.createGauge({
    labelNames: ['type'],
 });

+const circuitBreakerCounter = ZenkoMetrics.createCounter({
+    name: 's3_circuit_breaker_err',


s3_circuit_breaker_errors_count

monitoring/ingestion/ingestion-producer-alert.yaml

monitoring/lifecycle/alerts.test.yaml

tests/unit/lifecycle/CircuitBreakerGroup.spec.js

monitoring/lifecycle/alerts.yaml

tests/unit/lib/util/circuitBreaker.spec.js

package.json

lib/CircuitBreaker.js

benzekrimaha · 2024-07-05T14:51:49Z

/create_integration_branches

bert-e · 2024-07-05T14:51:59Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/BB-580-circuit-breaker-alert into
development/8.6
w/8.7/improvement/BB-580-circuit-breaker-alert will be merged into development/8.7

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-07-05T14:52:00Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

benzekrimaha · 2024-07-05T14:52:18Z

@bert-e create_pull_requests

bert-e · 2024-07-05T14:52:29Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/BB-580-circuit-breaker-alert into
development/8.6
w/8.7/improvement/BB-580-circuit-breaker-alert will be merged into development/8.7 (pull request INTEGRATION [PR#2516 > development/8.7] Improvement/bb 580 circuit breaker alert #2518)

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.5

Follow integration pull requests if you would like to be notified of
build statuses by email.

The following options are set: create_pull_requests, create_integration_branches

bert-e · 2024-07-05T14:52:29Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_pull_requests, create_integration_branches

benzekrimaha · 2024-07-05T15:50:20Z

/approve

bert-e · 2024-07-05T15:50:33Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/8.6
✔️ development/8.7

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.5

Please check the status of the associated issue BB-580.

Goodbye benzekrimaha.

The following options are set: approve, create_pull_requests, create_integration_branches

benzekrimaha force-pushed the improvement/BB-580-circuit-breaker-alert branch 3 times, most recently from 602972a to 32c3941 Compare June 19, 2024 15:46

benzekrimaha marked this pull request as ready for review June 19, 2024 15:52

francoisferrand reviewed Jun 20, 2024

View reviewed changes

lib/CircuitBreaker.js Outdated Show resolved Hide resolved

francoisferrand reviewed Jun 20, 2024

View reviewed changes

lib/CircuitBreaker.js Outdated Show resolved Hide resolved

francoisferrand reviewed Jun 20, 2024

View reviewed changes

lib/CircuitBreaker.js Outdated Show resolved Hide resolved

francoisferrand reviewed Jun 20, 2024

View reviewed changes

monitoring/ingestion/ingestion-producer-alert.yaml Outdated Show resolved Hide resolved

benzekrimaha force-pushed the improvement/BB-580-circuit-breaker-alert branch from e9653b8 to 73b8545 Compare June 21, 2024 13:25

benzekrimaha requested a review from francoisferrand June 21, 2024 13:35

francoisferrand reviewed Jun 24, 2024

View reviewed changes

monitoring/ingestion/ingestion-producer-alert.yaml Outdated Show resolved Hide resolved

francoisferrand reviewed Jun 24, 2024

View reviewed changes

monitoring/lifecycle/alerts.test.yaml Outdated Show resolved Hide resolved

francoisferrand reviewed Jun 24, 2024

View reviewed changes

monitoring/lifecycle/alerts.test.yaml Show resolved Hide resolved

benzekrimaha commented Jun 24, 2024

View reviewed changes

tests/unit/lifecycle/CircuitBreakerGroup.spec.js Show resolved Hide resolved

benzekrimaha force-pushed the improvement/BB-580-circuit-breaker-alert branch 3 times, most recently from b8a6a56 to 3460fe8 Compare July 1, 2024 07:13

benzekrimaha requested review from francoisferrand and williamlardier July 1, 2024 07:14

francoisferrand reviewed Jul 3, 2024

View reviewed changes

monitoring/lifecycle/alerts.yaml Outdated Show resolved Hide resolved

francoisferrand reviewed Jul 3, 2024

View reviewed changes

tests/unit/lib/util/circuitBreaker.spec.js Outdated Show resolved Hide resolved

francoisferrand reviewed Jul 3, 2024

View reviewed changes

package.json Outdated Show resolved Hide resolved

benzekrimaha force-pushed the improvement/BB-580-circuit-breaker-alert branch from dd7afc5 to 9a8e512 Compare July 3, 2024 10:14

benzekrimaha requested a review from francoisferrand July 3, 2024 10:15

benzekrimaha force-pushed the improvement/BB-580-circuit-breaker-alert branch from 6e2de94 to 9a8e512 Compare July 3, 2024 15:42

Kerkesni approved these changes Jul 4, 2024

View reviewed changes

francoisferrand approved these changes Jul 5, 2024

View reviewed changes

lib/CircuitBreaker.js Outdated Show resolved Hide resolved

benzekrimaha added 5 commits July 5, 2024 16:03

BB-580:circuitBreakerCounter introduction

f0691eb

BB-580:alert setup

e8d9c95

BB-580:alert test

b385c4f

BB-580:circuitbreaker test update

33c239d

BB-580:lint warn fix

f71bab5

benzekrimaha force-pushed the improvement/BB-580-circuit-breaker-alert branch from d312b33 to 48937f9 Compare July 5, 2024 14:06

BB-580:breakbeat version bump

715d589

benzekrimaha force-pushed the improvement/BB-580-circuit-breaker-alert branch from 7bbfd1d to 715d589 Compare July 5, 2024 14:18

BB-580:project version bump

7788e47

bert-e mentioned this pull request Jul 5, 2024

INTEGRATION [PR#2516 > development/8.7] Improvement/bb 580 circuit breaker alert #2518

Merged

bert-e merged commit 7788e47 into development/8.6 Jul 5, 2024
9 checks passed

bert-e deleted the improvement/BB-580-circuit-breaker-alert branch July 5, 2024 15:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvement/bb 580 circuit breaker alert #2516

Improvement/bb 580 circuit breaker alert #2516

benzekrimaha commented Jun 19, 2024 •

edited

Loading

bert-e commented Jun 19, 2024

bert-e commented Jun 19, 2024

codecov bot commented Jun 19, 2024 •

edited

Loading

francoisferrand Jun 24, 2024

benzekrimaha commented Jul 5, 2024

bert-e commented Jul 5, 2024

bert-e commented Jul 5, 2024

benzekrimaha commented Jul 5, 2024

bert-e commented Jul 5, 2024

bert-e commented Jul 5, 2024

benzekrimaha commented Jul 5, 2024

bert-e commented Jul 5, 2024

Improvement/bb 580 circuit breaker alert #2516

Improvement/bb 580 circuit breaker alert #2516

Conversation

benzekrimaha commented Jun 19, 2024 • edited Loading

bert-e commented Jun 19, 2024

Hello benzekrimaha,

bert-e commented Jun 19, 2024

Request integration branches

codecov bot commented Jun 19, 2024 • edited Loading

Codecov Report

francoisferrand Jun 24, 2024

Choose a reason for hiding this comment

benzekrimaha commented Jul 5, 2024

bert-e commented Jul 5, 2024

Integration data created

bert-e commented Jul 5, 2024

Waiting for approval

benzekrimaha commented Jul 5, 2024

bert-e commented Jul 5, 2024

Integration data created

bert-e commented Jul 5, 2024

Waiting for approval

benzekrimaha commented Jul 5, 2024

bert-e commented Jul 5, 2024

benzekrimaha commented Jun 19, 2024 •

edited

Loading

codecov bot commented Jun 19, 2024 •

edited

Loading