Multiple policy-bot instances status update issue

[pravindahal]

We are trying to run two instances of policybot in our organization (running multiple instances has been mentioned in #705 (comment)). We have configured two different GitHub apps and are running two instances of PolicyBot with the following settings:

    POLICYBOT_PUBLIC_URL: https://api.domain.com/policy-bot
    POLICYBOT_OPTIONS_STATUS_CHECK_CONTEXT: policy-bot
    GITHUB_APP_INTEGRATION_ID: "123456"
    GITHUB_OAUTH_CLIENT_ID: xxxxxxxxxxxxxxxxxx

    POLICYBOT_PUBLIC_URL: https://api.anotherdomain.com/policy-bot
    POLICYBOT_OPTIONS_STATUS_CHECK_CONTEXT: policy-bot-new
    POLICYBOT_OPTIONS_POLICY_PATH: .policy.new.yml
    GITHUB_APP_INTEGRATION_ID: "654321"
    GITHUB_OAUTH_CLIENT_ID: yyyyyyyyyyyyyyyyyyy

All the permissions and events have been set up as per the latest documentation.

We have given access to all repos for the first GitHub app and a handful of repos to the second app.

In a repo that both apps have access to, we only have a .policy.new.yml and no .policy.yml. Also, we have only required policy-bot-new: main status check in this repo. Yet, the first policybot app keeps trying to update the status and it results in messages like this in logs:

Entity 'policy-bot-new-app[bot]' overwrote status check 'policy-bot-new: main' to state='failure' description=''policy-bot-app[bot]' overwrote status to 'failure'' targetURL=''

This keeps happening until it eventually fails with:

POST https://api.github.com/repos/orgname/repo-name/statuses/91025c56927a4215acb631274b30g6ce894dc1bf: 422 Validation Failed [{Resource:Status Field: Code:custom Message:This SHA and context has reached the maximum number of statuses.}]

It seems like one app tries to update status and that event triggers other app to try to do the same.

Are we doing something wrong when it comes to running multiple instances?

If it matters, at this time, we are running multiple versions (v1.32.0 and v1.36.3).

[bluekeyes]

This was a bug (#799) that was fixed in 1.36.0. In earlier versions, Policy Bot used a simple prefix match to determine if another status check overwrote its own, which could lead to this conflict.

There are three ways you could fix this:

1. Upgrade both of your instances to 1.36.0 or newer
2. Change the status check contexts so that one is not a prefix of the other (e.g. policy-bot and new-policy-bot)
3. Switch which version is using which status check context so that the 1.36.3 instance uses the prefix (policy-bot) and the 1.32.0 version uses the longer context (policy-bot-new)

[pravindahal]

@bluekeyes Thanks! We were already using the different context (sorry I forgot to mention).

Our issue has been resolved.

We updated both to 1.36.0 and I also had to double check that the check was connected to the correct app (in our case, we just selected "any source").