[test] Add docker-compose to faciliate browser test execution

[luke-mino-altherr]

Summary

Improves Playwright browser test infrastructure by adding Docker support and fixing cross-platform compatibility issues.

Problem

The existing browser test setup was complex and had several issues:

• Platform-specific screenshot issues: Tests generated Linux-specific screenshots that failed on macOS due to font rendering differences
• Complex local setup: Required manually installing and configuring ComfyUI backend, managing dependencies, and setting up test environment
• User context errors: Backend threw "Unknown user: default" errors in multi-user mode when tests made API calls without proper user context
• Inconsistent test environment: Different developers had varying local setups leading to flaky tests

Solution

Docker Infrastructure

• Dockerfile: Multi-stage build for ComfyUI with built frontend, includes system dependencies for PyTorch
• docker-compose.test.yml: Orchestrates ComfyUI backend and Playwright test containers with proper networking and health checks
• Dockerfile.playwright: Dedicated container for running Playwright tests with all browsers pre-installed
• npm run test:browser:docker: Single command to run entire test suite in containers

Cross-Platform Compatibility

• Screenshot tolerance: Added 3% pixel difference tolerance for local testing to handle font rendering variations
• Gitignore patterns: Ignore platform-specific screenshot files (e.g., *-darwin.png) to prevent commit conflicts

User Context Fixes

• Default user creation: Global setup now creates a "default" user in users.json to fix errors logged in the backend API logs.

Benefits

• One-command testing: npm run test:browser:docker runs complete test suite
• Platform consistency: Tests run in identical Linux containers regardless of host OS
• Easier onboarding: New developers don't need complex local ComfyUI setup
• Reliable CI/CD: Consistent containerized environment eliminates "works on my machine" issues
• Faster debugging: HTML reports and screenshots automatically generated and accessible via browser

Usage

# Run all browser tests in Docker
npm run test:browser:docker

# View test results at http://localhost:9323 (if report server starts)

┆Issue is synchronized with this Notion page by Unito

[christian-byrne]

If the primary issue with local testing is screenshots due to things like FreeType vs. DirectWrite, isn't this change the only thing that's necessary?

[webfiltered]

I actually think this is not something we can merge. 😅

At only 1024x768, this ignores anything that changes less than 23,000 pixels. It would cripple the usefulness of playwright.

[luke-mino-altherr]

I am running on a mac. Since the only screenshots committed to the repository are linux based screenshots, i need to execute these tests in a linux based execution environment in order to properly validate there are no visual regressions.

Also as a new developer, i found the initial setup complex -- Docker cleanly abstracts that initial setup and server startup.

[luke-mino-altherr]

I actually think this is not something we can merge. 😅

At only 1024x768, this ignores anything that changes less than 23,000 pixels. It would cripple the usefulness of playwright.

Ah fair point. We could lower the tolerance? I was most often seeing differences at less than 1%. Also note, this is only in effect during local testing. Github Actions will still check for exact matches

[webfiltered]

I'm super curious about the discrepancies, and wonder if it's just a difference between the image GH actions uses and the docker image? As the display should be virtualised too, we may be able to get pixel-perfect.

I can take a look on mac/windows and see if anything stands out (likely within a day or two).

[webfiltered]

I actually think this is not something we can merge. 😅

At only 1024x768, this ignores anything that changes less than 23,000 pixels. It would cripple the usefulness of playwright.