shot-scraper video command

[simonw]

I'd really love the ability to use shot-scraper to record an automated video of interactions with a web app.

This is an ambitious challenge!

[simonw]

Got a VERY rough prototype working like this:

@cli.command()
@click.argument("url")
@click.argument("script")
@click.option(
    "-o",
    "--output",
    type=click.File("w"),
    help="Save output video to this file",
)
def video(
    url,
    script,
    output,
):
    """
    Record a video of the specified page

    The script should be a Python script using the following functions:

    \b
    sleep(0.5) - delay for half a second
    page.click('selector') - click the element identified by the selector
    page.scrollTo('selector', 300) - scroll that selector in view, take 300ms
    page.type('selector', 'Hello') - type 'Hello' into the specified input
    press('Enter') - press the Enter key
    """
    if not output:
        output = "/tmp/shot-scraper-video"
    url = url_or_file_path(url, _check_and_absolutize)
    with sync_playwright() as p:
        browser = p.chromium.launch()
        context = browser.new_context(
            record_video_dir=output,
            record_video_size={"width": 1280, "height": 720},
        )
        page = context.new_page()
        page.goto(url)
        exec(script, {"page": page, "sleep": time.sleep})
        context.close() # this saves the video
        browser.close()

Example usage:

shot-scraper video https://simonwillison.net/ '
sleep(1)
page.click("a:nth-of-type(3)")
sleep(3)
'

This produced the following video as /tmp/shot-scraper-video/95becf0c7ad1d278446018af7f44c49b.webm (541KB):

95becf0c7ad1d278446018af7f44c49b.webm

[simonw]

I'd like the mouse cursor to be visible. This looks helpful for that: https://www.npmjs.com/package/mouse-helper

Also maybe this comment:

• Is there a way to show mouse pointer in puppeteer? puppeteer/puppeteer#4378 (comment)

I think I'd need some extra utility functions for things like "smoothly scroll the page to this element" and "apply a yellow flash animated highlight to this element" and "smoothly move the mouse cursor to here".

It would also be nice if the video wasn't .webm - I think that's the only thing Playwright can output, but I could do a thing that attempts to use ffmpeg (if available) to convert to another specified format.

Needs a lot of thought about the best scripting language subset to expose, what other options, what browser viewport size etc.

[simonw]

Asked for tips on Pallets Discord here: https://discord.com/channels/531221516914917387/531571912619458591/1204292849256890388 - when I thought I wanted a design that looked more like this, but needed a way to preserve the relative order of Click options:

my-tool --delay 300 \
  --click 'a[href="/"]' \
  --delay 1000 \
  --fill 'input[name="q"]' 'my search term' \
  --delay 200 \
  --click 'button[id="search"]'

I think I like the Python DSL better though.