Add python skills and test. by LarryOsterman · Pull Request #375 · microsoft/skills

LarryOsterman · 2026-07-10T22:49:52Z

Adds Python skill-effectiveness experiments, expands skill reference documentation, and updates the Vally test harness.

Changes:

Adds Python Vally experiments and standardizes test models.
Adds capability/non-hero references and lifecycle guidance.
Updates Vally/Rust graders, harness output capture, and CI checks.
Reviewed Changes

…mprovements to skill-experiments script

…tion or class' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

Updated pnpm dependencies Removed agentic workflows for evaluation Renamed workflow 2 Renamed workflow Oops, wrong evaluation Try pull_request_target Added back evaluation workflow separate from agentic workflow Try defining a token for vally to use For ce update to prompt CI fixes Added lockfile explicitly Fixed lockfile enable copilot-requests permission Moved back to agentic workflow; update to vally 0.7.0 Moved back to agentic workflow; update to vally 0.7.0 Restored older ealuations Updated evaluations And set the environment variable Install copilot Just try out copilot cli before doing expensive stuff

…kills into larryo/python_skills

…a bit

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

…et available from crates.io

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Reducing confusion on vally scope notifications.

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

richardpark-msft

Still looking through it, but found some things worth addressing now.

richardpark-msft · 2026-07-15T00:43:41Z

        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4


You can use the latest setup-node action here instead of v4. v6 (currently v6.5.0) has been stable for months; I'd stick with v6 rather than v7, which only just shipped. Apply this to move up a major version:

Suggested change

- uses: actions/setup-node@v4

- uses: actions/setup-node@v6

One thing to watch: since v5, setup-node auto-caches when package.json has a packageManager field. This step just does a global npm install, so it should be a non-issue, but set package-manager-cache: false if you see odd caching.

richardpark-msft · 2026-07-15T00:43:41Z

            fi
-          elif [ "${{ github.event_name }}" = "pull_request" ]; then
-            collect_from_changed_dirs "${{ github.event.pull_request.base.sha }}" "${{ github.sha }}" | sort -u > "$specs_file"
+          elif [ "${{ github.event_name }}" = "pull_request" ] || [ "${{ github.event_name }}" = "pull_request_target" ]; then


You didn't end up using pull_request_target, so you can probably just revert this line (and any other ones related to it).

richardpark-msft · 2026-07-15T00:43:41Z

+          echo "::notice::Running Vally evaluations for same-repo PR from ${{ github.event.pull_request.head.repo.full_name }}."
+
+      - name: Vally PR scope notice (cross-repo)
+        if: ${{ steps.specs.outputs.has_specs == 'true' && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository }}


I was going back and forth on this with copilot - the checks (for PR vs non-PR) are a bit noisy but apparently you can break it apart a bit, which I think would help. So your overall purpose (everybody gets linted) happens easily enough and eval is a bit more special, but now it's confined to it's own job.

jobs: lint: runs-on: ubuntu-latest # runs for everyone, no token steps: - uses: actions/checkout@v6 # resolve specs + schema-only lint eval: needs: lint runs-on: ubuntu-latest if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository permissions: contents: read copilot-requests: write env: GH_TOKEN: ${{ github.token }} steps: - uses: actions/checkout@v6 # build grader plugin + run vally eval

richardpark-msft · 2026-07-15T00:43:41Z

+    [string]$SkillPattern = "*-py",
+
+    [Parameter(Mandatory = $false)]
+    [ValidateRange(1, 128)]


Please get me the 128 core machine...

richardpark-msft · 2026-07-15T00:43:41Z

@@ -0,0 +1 @@
+Error: Experiment config file not found: Q:\src\skills\tests\scenarios\agent-framework-azure-ai-py\skill_effectiveness_experiment.yaml


Is this file supposed to be checked in? Also, Q:\ is not universal....

richardpark-msft · 2026-07-15T00:43:41Z

@@ -0,0 +1,94 @@
+Running experiment 'azure-monitor-ingestion-py-skill-experiment'
+Output: Q:\src\skills\tests\scenarios\azure-monitor-ingestion-py\vally\vally-experiment-results\2026-06-30T23-35-17-490Z


Another one, not sure if this file should be checked in. Maybe do a scrub and then add in a .gitignore file?

richardpark-msft · 2026-07-15T00:43:41Z

+    $npmLoaderPath = $null
+    $loaderCandidates = @(
+        (Join-Path $copilotDir "node_modules\@github\copilot\npm-loader.js"),
+        "q:\.tools\.npm-global\node_modules\@github\copilot\npm-loader.js"


Might be worth doing a pass, making sure you don't have other paths hardcoded from your dev setup.

richardpark-msft · 2026-07-15T00:43:41Z

+        [bool]$EnableNodeLoaderFallback = $true
+    )
+
+    $copilotCommand = Get-Command copilot -ErrorAction SilentlyContinue


There's a lot of code in here trying to be accomodating to all sorts of scenarios around copilot, but it's probably okay to just say 'use the copilot in the path, and if it's not t here, then bail' and then you can delete a bunch of code.

LarryOsterman added 30 commits June 26, 2026 13:53

Ensure that python scenarios pass with current models

445df74

Updated python skills; added skills links

a71fe6c

Added initial evaluations for all python skills

b103ad9

Updated vally evaluations and verified functionality

c38e89a

Added vally experiments for all the python skills

4441d58

Updated skill generator to improve compactness of skills

234315e

Update rust evaluations to give tools more time to run; significant i…

2c5195e

…mprovements to skill-experiments script

Updated node version to current

95a59b4

Added rust toolchain install for rust evals

34655a4

CI pipeline and build fixes

adc50d0

Neuter vally evaluations for now

b053b67

CI pipeline possible fix

2800214

CI pipeline possible fix

d972fc7

Just try out copilot cli before doing expensive stuff

cb7adf9

Install copilot

c3c0bd6

And set the environment variable

6f35944

Updated evaluations

ff4da64

Restored older ealuations

5bdc963

Moved back to agentic workflow; update to vally 0.7.0

3335acd

Moved back to agentic workflow; update to vally 0.7.0

9fc75a9

enable copilot-requests permission

8e39594

enable copilot-requests permission again

1883683

Fixed lockfile

7286a99

Added lockfile explicitly

0cf1bb0

CI fixes

3ca00fb

For ce update to prompt

4772426

Try defining a token for vally to use

8c45247

Added back evaluation workflow separate from agentic workflow

a312f96

Try pull_request_target

dc4a46b

Oops, wrong evaluation

d79c98f

github-code-quality Bot found potential problems Jul 10, 2026

View reviewed changes

Comment thread tests/compare-experiment.mjs Fixed

github-actions Bot mentioned this pull request Jul 10, 2026

[aw] No-Op Runs #376

Closed

LarryOsterman and others added 12 commits July 10, 2026 16:09

Removed agentic workflows for evaluation

dc7d0a4

Updated pnpm dependencies

5e60981

Potential fix for pull request finding 'Unused variable, import, func…

fb0bd8a

…tion or class' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>

Updated vally version

fb96170

Merge branch 'larryo/python_skills' of https://github.com/microsoft/s…

eaf5cf2

…kills into larryo/python_skills

Use different model for graders and evals

0d3ec2d

Fixed all the grader models

daacded

Don't run pipeline when PR originates from a fork; cleaned up script …

30f7e2c

…a bit

Fixed grader tools for rust evaluations

6c256ea

Use custom grader to run tools instead of just cargo

d2fb8f7

Use custom grader to run tools instead of just cargo

d5ff87a

LarryOsterman marked this pull request as ready for review July 13, 2026 23:30

LarryOsterman requested a review from thegovind as a code owner July 13, 2026 23:30

Copilot AI review requested due to automatic review settings July 13, 2026 23:30

LarryOsterman requested a review from g2vinay as a code owner July 13, 2026 23:30

Copilot started reviewing on behalf of LarryOsterman July 13, 2026 23:31 View session

Copilot AI reviewed Jul 13, 2026

Removed azure-servicebus-rust skills because rust servicebus is not y…

f1c7d90

…et available from crates.io

Copilot AI review requested due to automatic review settings July 14, 2026 00:28

Copilot started reviewing on behalf of LarryOsterman July 14, 2026 00:29 View session

Copilot AI reviewed Jul 14, 2026

LarryOsterman requested a review from a team July 14, 2026 21:33

Clarify PR scope evaluation notice

41cf7de

Reducing confusion on vally scope notifications.

Copilot AI review requested due to automatic review settings July 14, 2026 22:00

Copilot started reviewing on behalf of LarryOsterman July 14, 2026 22:01 View session

Copilot AI reviewed Jul 14, 2026

richardpark-msft reviewed Jul 15, 2026

View reviewed changes

		@@ -0,0 +1 @@
		Error: Experiment config file not found: Q:\src\skills\tests\scenarios\agent-framework-azure-ai-py\skill_effectiveness_experiment.yaml

		@@ -0,0 +1,94 @@
		Running experiment 'azure-monitor-ingestion-py-skill-experiment'
		Output: Q:\src\skills\tests\scenarios\azure-monitor-ingestion-py\vally\vally-experiment-results\2026-06-30T23-35-17-490Z

Uh oh!

Conversation

LarryOsterman commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

richardpark-msft left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LarryOsterman commented Jul 10, 2026 •

edited

Loading