[BUG] code review found 17 items only commented 3 on PR

[joelsprunger]

Current Behavior:

I added LangSmith tracing to my github action.

    env:
      LANGCHAIN_TRACING_V2: "true"
      LANGCHAIN_ENDPOINT: "https://api.smith.langchain.com"
      LANGCHAIN_API_KEY: ${{ secrets.LANGCHAIN_API_KEY }}
      LANGCHAIN_PROJECT: "code_review_gpt"

In looking over the traces outputs for my large PR, I see 17 comments in 5 or 6 groups. However the Summary chain to generate emojis) only has 3 of these comments as input. These are the ones that go to the PR comments. The other 15 seem to be discarded.

Expected Behavior:

The PR comments should contain ALL the Risk items found in code review. Perhaps the emoji summarizer needs to randomly select the comments to add, or add a finite number of comments prioritized by risk level.

Steps To Reproduce:

Hmm. Tough. Big PR. Check the tracing.

Suggestions on the root cause

I am guessing the root cause is the finite context window for the summary. You can still limit the comments used in the summary, but add ALL comments to the PR.

[joelsprunger]

I suggest generating a github artifact code-review-gpt-details.md with all the PR comments.

If there are more than 3, the action could generate the artifact, and add a comment:

More comments are available in your github artifacts

Please follow these steps to view the artifact:

1. Navigate to the Actions tab of this repository.
2. Click on the workflow run that corresponds to your PR.
3. Find the Artifacts section at the bottom of the workflow run page.
4. Click on the code-review-gpt-details.md artifact

[mattzcarey]

Sounds like a great addition. When I built this I wanted to reduce the cognitive load on a human reviewer hence the whole ranking and limiting to 3 comments. Happy to generate an artifacts file that makes a lot of sense.