Minimal python package support by abstractedfox · Pull Request #1881 · cloudflare/agents

abstractedfox · 2026-07-06T16:43:15Z

This implements the beginning of package support for dynamic python workers. At present, it allows defining a flat list of pure Python packages as dependencies, which it installs in the dynamic worker without dependency resolution

This is the beginning of work on this feature, it's a little comment-heavy to that end and has some patterns that may not be permanent (these have generally been noted as such). As of this commit, there are tests that start to poke at this, namely the one that brings in fastapi, which currently fails due to an extension it doesn't like.

Proof of concept for package support in dynamic Python workers. As of this commit, it can retrieve an sdist with a flat layout from pypi and install it into the virtual filesystem. It doesn't do dependency resolution, so the current tests fail when attempting to import FastAPI's deps.

Used sdists before, now it uses wheels. Pre cleanup commit

changeset-bot · 2026-07-06T16:43:21Z

⚠️ No Changeset found

Latest commit: 4c56501

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

This test only needs to confirm that retrieval of a hard list of pure python packages, without dependency resolution, works.

This seems to be specific to JS packages

This dependendency caused issues when working in a container, but its removal was purely for my sake This reverts commit 4f8990d.

Also makes PyprojectToml interface less stringent

These got caught by the checks done by `pnpm run check`

pkg-pr-new · 2026-07-08T18:04:05Z

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@1881

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@1881

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@1881

create-think

npm i https://pkg.pr.new/create-think@1881

hono-agents

npm i https://pkg.pr.new/hono-agents@1881

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@1881

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@1881

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@1881

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@1881

commit: 4c56501

devin-ai-integration

Devin Review found 9 potential issues.

devin-ai-integration · 2026-07-08T18:36:17Z

+  if (packageJson) {
+    try {
+      const pkg = JSON.parse(packageJson);
+      const deps = pkg.dependencies ?? {};
+      return Object.keys(deps).length > 0;
+    } catch {
+      return false;
+    }


🔴 Python dependency check is skipped when a package.json file exists, even if it has no npm dependencies

The dependency-presence check always returns from the package.json branch (return Object.keys(deps).length > 0 at packages/worker-bundler/src/installer.ts:773) without ever reaching the pyproject.toml check, so Python dependencies listed in pyproject.toml are silently ignored whenever a package.json is also present.

Impact: Projects with both package.json and pyproject.toml will fail to install any Python dependencies.

Control-flow short-circuit in hasDependencies

When packageJson is truthy, the function enters the if (packageJson) block at packages/worker-bundler/src/installer.ts:769. Inside, it either returns true/false from the try block (line 773) or false from the catch block (line 775). In all cases, execution never reaches the if (pyprojectToml) block at line 779.

The fix should check both files and return true if either has dependencies, e.g. by not returning early from the package.json branch when it finds zero deps, or by combining the results with ||.

Suggested change

if (packageJson) {

try {

const pkg = JSON.parse(packageJson);

const deps = pkg.dependencies ?? {};

return Object.keys(deps).length > 0;

} catch {

return false;

}

if (packageJson) {

try {

const pkg = JSON.parse(packageJson);

const deps = pkg.dependencies ?? {};

if (Object.keys(deps).length > 0) return true;

} catch {

// Failed to parse, fall through to check pyproject.toml

}

}

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-07-08T18:36:18Z

+      return result;
+    }

-  // Collect dependencies to install
-  const depsToInstall: Record<string, string> = {
-    ...packageJson.dependencies,
-    ...(dev ? packageJson.devDependencies : {})
-  };
+    // Collect dependencies to install
+    const depsToInstall: Record<string, string> = {
+      ...packageJson.dependencies,
+      ...(dev ? packageJson.devDependencies : {})
+    };

-  if (Object.keys(depsToInstall).length === 0) {
-    return result; // No dependencies to install
-  }
+    if (Object.keys(depsToInstall).length === 0) {
+      return result; // No dependencies to install


🔴 Python dependency installation is skipped when package.json has no npm dependencies or fails to parse

The installer returns early (return result at packages/worker-bundler/src/installer.ts:129 and :139) when package.json is present but has no dependencies or is malformed, so the pyproject.toml processing block is never reached and Python packages are never fetched.

Impact: When both package.json and pyproject.toml exist, Python dependencies are silently not installed if the package.json has zero npm dependencies.

Early returns in installDependencies skip pyproject.toml

In installDependencies, the if (packageJsonContent) block at line 123 contains two early return result statements:

Line 129: returns when package.json fails to parse

Line 139: returns when package.json has zero dependencies

Both prevent execution from reaching the if (pyprojectTomlContent) block at line 163. The fix should remove these early returns or restructure the flow so both blocks are always evaluated.

Prompt for agents

In installDependencies (packages/worker-bundler/src/installer.ts), the packageJsonContent block has two early return statements (lines 129 and 139) that prevent the pyprojectTomlContent block from ever executing when package.json is present. The fix is to remove these early returns and instead let execution fall through to the pyprojectToml block. For the parse failure case (line 129), change return result to just a warning (which it already pushes) and remove the return. For the zero-deps case (line 139), remove the return so execution continues to the pyproject.toml check. The npm install logic should still be skipped when there are no npm deps, but the function should not return early.

Was this helpful? React with 👍 or 👎 to provide feedback.

This seems to be fine. Python Workers shouldn't specify package.json.

devin-ai-integration · 2026-07-08T18:36:19Z

+    expect(response.status).toBe(200);
+    expect(await response.text()).toBe("hi");
+  });
+}, 20000);


🟡 Test timeout is passed to describe() which ignores it, leaving tests with the default timeout

The extended timeout value is passed as a third argument to describe() (packages/worker-bundler/src/tests/e2e.test.ts:1019) which does not accept a timeout parameter, so the individual test cases inside run with the default timeout instead of the intended 20-second limit.

Impact: Python worker tests that need extra time may intermittently fail with timeout errors.

vitest describe() does not support timeout parameter

At lines 1019 and 1099, describe("...", () => { ... }, 20000) passes 20000 as a third argument. In vitest, only it()/test() accept a timeout as the third argument. The describe() function signature is describe(name, fn) or describe(name, options, fn) where options is an object — a bare number as the third argument is silently ignored.

The timeout should be moved to each individual it() call inside the describe block, e.g.:

it("executes a Python script...", async () => { ... }, 20000);

Prompt for agents

In packages/worker-bundler/src/tests/e2e.test.ts, the timeout value 20000 is passed as the third argument to describe() at lines 1019 and 1099. vitest's describe() does not accept a timeout parameter — only it()/test() do. Move the timeout to each individual it() call inside the describe blocks. For example, change it('executes a Python script...', async () => { ... }) to it('executes a Python script...', async () => { ... }, 20000). This needs to be done for both describe blocks (lines 989-1019 and 1021-1099).

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-07-08T18:36:21Z

@@ -1,5 +1,5 @@
 {
-  "compatibility_date": "2026-06-11",
+  "compatibility_date": "2026-05-15",


🟡 Test configuration compatibility date changed away from the repo-wide standard

The test wrangler config's compatibility date is changed to "2026-05-15" (packages/worker-bundler/src/tests/wrangler.jsonc:2), diverging from the repo-wide standard of "2026-06-11" mandated by the repository rules.

Impact: Tests may run against an older compatibility surface, potentially masking or causing subtle behavior differences.

AGENTS.md rule violation

AGENTS.md states under "Workers conventions":

All wrangler configs use compatibility_date: "2026-06-11"

Every other wrangler.jsonc in the repo uses "2026-06-11". This PR changes the worker-bundler test config from "2026-06-11" to "2026-05-15".

Suggested change

"compatibility_date": "2026-05-15",

"compatibility_date": "2026-06-11",

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-07-08T18:36:22Z

+    return {
+      mainModule: entryPoint,
+      modules,
+      wranglerConfig: {
+        compatibilityDate: wranglerConfig?.compatibilityDate ?? "2026-01-01",
+        compatibilityFlags: wranglerConfig?.compatibilityFlags ?? [
+          "python_workers"
+        ]
+      }
+    };


🔍 Install warnings are silently dropped for Python worker results

When the entry point is a .py file, createWorker returns early at packages/worker-bundler/src/index.ts:157-166 with a hardcoded result object. The installWarnings array collected at line 127 is never included in this return value, unlike the JS/TS paths which attach warnings at lines 195 and 226. Any warnings from dependency installation (e.g., failed packages, parse errors) will be silently discarded for Python workers.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-07-08T18:36:23Z

+function extractWheel(
+  data: Uint8Array,
+  result: InstallResult
+): Record<string, string> {
+  const unzipped = unzipSync(data);
+  const files: Record<string, string> = {};
+  const textDecoder = new TextDecoder();
+
+  for (const [path, content] of Object.entries(unzipped)) {
+    if (!isTextFile(path)) {
+      result.warnings.push(
+        `Could not install file ${path}, extension must match an approved text format type. This may corrupt this dependency.`
+      );
+      continue;
+    }
+    files[path] = textDecoder.decode(content);
+  }
+
+  return files;
+}


🔍 Python wheel extraction silently drops common Python file types

The extractWheel function at packages/worker-bundler/src/installer.ts:562-581 filters files through isTextFile(), which only allows a fixed set of extensions. Python wheels commonly contain .pyi (type stubs), .cfg, .ini, and other files that won't pass this filter. Each skipped file generates a warning saying "This may corrupt this dependency," which could produce a large volume of warnings for real-world packages. More importantly, .pyi files are sometimes needed at runtime by packages that use typing.get_type_hints(). Consider expanding the allowed extensions for Python wheels.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-07-08T18:36:24Z

+    for (const path of fileSystem.list()) {
+      if (path !== "pyproject.toml") {
+        const content = fileSystem.read(path);
+        if (content !== null) {
+          modules[path] = content;
+        }
+      }
+    }


🔍 Python modules output includes all filesystem files, potentially including node_modules

The Python worker path at packages/worker-bundler/src/index.ts:148-155 iterates over all files in the filesystem (except pyproject.toml) and includes them in the output modules. If a project has both package.json with npm dependencies and a Python entry point, the installed node_modules/ tree would be included in the modules output. This could result in a very large modules object being passed to the Worker Loader. Consider filtering out node_modules/ and python_modules/ metadata directories.

Was this helpful? React with 👍 or 👎 to provide feedback.

These are just files specified by the user in the createWorker call, right? So this AI review comment seems wrong.

At that point it should be those files + all the packages that were just added, so my assumption is anything that's in there would've been put there deliberately

devin-ai-integration · 2026-07-08T18:36:25Z

+      for (const [filePath, content] of Object.entries(packageFilesWheel)) {
+        fileSystem.write(`python_modules/${name}/${filePath}`, content);
+      }


🔍 Package name used as filesystem directory may not match Python importable name

At packages/worker-bundler/src/installer.ts:410, Python packages are written to python_modules/{name}/ where name is the raw dependency string from pyproject.toml. However, Python package names and importable module names often differ (e.g., the PyPI package Pillow is imported as PIL, python-dateutil is imported as dateutil). The PyPI JSON API lookup will work (it's case-insensitive), but the filesystem path won't match the import name. Additionally, stripWheelToPackage strips the first directory component from wheel contents, but the wheel's internal directory name (the actual importable package name) may differ from the pyproject.toml dependency name. This could cause import failures at runtime.

Was this helpful? React with 👍 or 👎 to provide feedback.

Yeah, that's a fair point. Does the metadata help here to determine where to install this?

devin-ai-integration · 2026-07-08T18:36:27Z

+      const packageFilesWheel = stripWheelToPackage(
+        extractWheel(new Uint8Array(buffer), result)
+      );
+
+      // Mark as installed before writing to prevent cycles
+      installedPackages.set(name, version);
+      result.installed.push(`${name}@${version}`);
+
+      // Add files to python_modules
+      for (const [filePath, content] of Object.entries(packageFilesWheel)) {
+        fileSystem.write(`python_modules/${name}/${filePath}`, content);
+      }


🟨 Wheel archive entries are not validated for path traversal before writing to filesystem

Files extracted from Python wheel archives via unzipSync (packages/worker-bundler/src/installer.ts:566) are written to the virtual filesystem without validating that the entry paths don't contain path traversal sequences like ../. After stripWheelToPackage processing, a malicious wheel entry such as pkg/../../escape.py would be written as python_modules/{name}/../escape.py, potentially placing files outside the intended python_modules/{name}/ directory in the virtual filesystem.

Was this helpful? React with 👍 or 👎 to provide feedback.

Not a big deal in this case

dom96

Overall approach looks good! Left a few comments with suggestions.

CC @hoodmane @ryanking13 @threepointone for reviews too

dom96 · 2026-07-10T10:39:11Z

+  if (entryPoint.endsWith(".py")) {
+    const modules: Modules = {};
+    for (const path of fileSystem.list()) {
+      if (path !== "pyproject.toml") {


Nit:

Suggested change

if (path !== "pyproject.toml") {

if (path === "pyproject.toml") { continue; }

dom96 · 2026-07-10T10:40:08Z

+    for (const path of fileSystem.list()) {
+      if (path !== "pyproject.toml") {
+        const content = fileSystem.read(path);
+        if (content !== null) {
+          modules[path] = content;
+        }
+      }
+    }


These are just files specified by the user in the createWorker call, right? So this AI review comment seems wrong.

dom96 · 2026-07-10T10:44:24Z

+      return result;
+    }

-  // Collect dependencies to install
-  const depsToInstall: Record<string, string> = {
-    ...packageJson.dependencies,
-    ...(dev ? packageJson.devDependencies : {})
-  };
+    // Collect dependencies to install
+    const depsToInstall: Record<string, string> = {
+      ...packageJson.dependencies,
+      ...(dev ? packageJson.devDependencies : {})
+    };

-  if (Object.keys(depsToInstall).length === 0) {
-    return result; // No dependencies to install
-  }
+    if (Object.keys(depsToInstall).length === 0) {
+      return result; // No dependencies to install


This seems to be fine. Python Workers shouldn't specify package.json.

dom96 · 2026-07-10T10:45:54Z

-    result.warnings.push("Failed to parse package.json");
-    return result;
-  }
+  if (packageJsonContent) {


Code style suggestion: instead of pushing this into its own if branch, create a new installDependenciesPython function and move the Python logic there, then call this function when pyprojectTomlContent is non-null.

This will make it easier to review this too.

dom96 · 2026-07-10T10:55:11Z

+      const match = trimmed.match(/^([A-Za-z0-9._-]+)(.*)$/);
+      if (!match) {
+        result.warnings.push(
+          `Skipping invalid pyproject.toml dependency: ${dep}`
+        );
+        continue;
+      }


I wouldn't bother parsing these. For this PR you should only support dependencies without version requirements being specified. In the future this is something that the package resolver would handle the parsing for anyway, so no need to worry about it here.

So remove this code and just accept the dep names as they are.

dom96 · 2026-07-10T11:04:47Z

+  const installPromise = (async () => {
+    try {
+      // The JS impl. does the metadata part in another function, consider moving it if this gets too long
+      // Fetch package metadata from PyPI JSON API


Yeah, might be worth splitting this up

dom96 · 2026-07-10T11:06:14Z

+      for (const [filePath, content] of Object.entries(packageFilesWheel)) {
+        fileSystem.write(`python_modules/${name}/${filePath}`, content);
+      }


Yeah, that's a fair point. Does the metadata help here to determine where to install this?

dom96 · 2026-07-10T11:06:45Z

+      const packageFilesWheel = stripWheelToPackage(
+        extractWheel(new Uint8Array(buffer), result)
+      );
+
+      // Mark as installed before writing to prevent cycles
+      installedPackages.set(name, version);
+      result.installed.push(`${name}@${version}`);
+
+      // Add files to python_modules
+      for (const [filePath, content] of Object.entries(packageFilesWheel)) {
+        fileSystem.write(`python_modules/${name}/${filePath}`, content);
+      }


Not a big deal in this case

dom96 · 2026-07-10T11:19:06Z

+      compatibilityDate:
+        dynamic_worker.wranglerConfig?.compatibilityDate ?? "2026-01-01",
+      compatibilityFlags: dynamic_worker.wranglerConfig?.compatibilityFlags ?? [
+        "python_workers"


Should we expect createWorker to set this and therefore assert that these are non-null?

That's probably a good idea

dom96 · 2026-07-10T11:19:50Z

+  it("works with a pure python package", async () => {
+    const id = "test-worker-" + testId++;
+    const createWorkerResult = await createWorker({
+      files: {
+        "index.py": [
+          "from workers import Response, WorkerEntrypoint",
+          "import typing_extensions",
+          "import typing_inspection",
+          "class Default(WorkerEntrypoint):",
+          "  async def fetch(self, request):",
+          "    return Response.json({",
+          '      "typing_extensions": typing_extensions.__name__,',
+          '      "typing_inspection": typing_inspection.__name__',
+          "    })"
+        ].join("\n"),
+        "pyproject.toml": [
+          "[project]",
+          'name = "dummy"',
+          'version = "0.0.0"',
+          // typing_extensions has zero dependencies. typing_inspection depends on typing_extensions
+          'dependencies = ["typing_extensions", "typing_inspection"]'
+        ].join("\n")
+      }
+    });


Nice! Exactly what I was looking for :)

These are already set implicitly by createWorker

abstractedfox added 7 commits June 28, 2026 00:15

remove dependency that was causing issues

4f8990d

Reconstitute work on this repo

3b8ea8f

Add index.py as default entry point

96153da

Clarifying comment

98ad2f7

Use wheels properly

e01a96e

Used sdists before, now it uses wheels. Pre cleanup commit

abstractedfox added 12 commits July 6, 2026 15:09

Wheel changeover cleanup

2fd4db5

Use simpler packages for minimal package support test

248a149

This test only needs to confirm that retrieval of a hard list of pure python packages, without dependency resolution, works.

Move python logic out of 'bundle' path

2ca7e73

This seems to be specific to JS packages

Slight cleanup, remove test

58e6874

More cleanup

1cf3db3

Re-add parameter that was mistakenly deleted

ceda163

Revert "remove dependency that was causing issues"

7244ec5

This dependendency caused issues when working in a container, but its removal was purely for my sake This reverts commit 4f8990d.

Order deps alphabetically

493cf55

Remove unused variable

4fa5fce

Make createWorker Python return more uniform with JS

bf8d3a9

Make createWorker registry pattern more consistent with JS

6b365dd

Also makes PyprojectToml interface less stringent

Small fixes

7a2a85f

These got caught by the checks done by `pnpm run check`

abstractedfox changed the title ~~Draft: Python package progress~~ Minimal python package support Jul 8, 2026

abstractedfox marked this pull request as ready for review July 8, 2026 18:32

devin-ai-integration Bot reviewed Jul 8, 2026

View reviewed changes

dom96 reviewed Jul 10, 2026

View reviewed changes

abstractedfox added 5 commits July 10, 2026 10:02

Style fixes

ecbd776

Remove parsing for version string

8016a35

Comment dep version parameter

766e58c

Compensate for distribution packages' names not matching their imports

40cbc98

Remove default values from test

f3289dc

These are already set implicitly by createWorker

abstractedfox added 3 commits July 10, 2026 13:33

Move Python package metadata collection into its own function

6fde2e0

Move Py deps installation into its own function

b711067

Account for type checker

4c56501

	"compatibility_date": "2026-05-15",
	"compatibility_date": "2026-06-11",

	if (path !== "pyproject.toml") {
	if (path === "pyproject.toml") { continue; }

Uh oh!

Conversation

abstractedfox commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

pkg-pr-new Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dom96 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

abstractedfox commented Jul 6, 2026 •

edited

Loading

changeset-bot Bot commented Jul 6, 2026 •

edited

Loading

pkg-pr-new Bot commented Jul 8, 2026 •

edited

Loading