error.HeaderTooLarge

[7flash]

To reproduce, begin with example from README and replace URL to "https://x.com"

'use strict'

import puppeteer from 'puppeteer-core';

// use browserWSEndpoint to pass the Lightpanda's CDP server address.
const browser = await puppeteer.connect({
  browserWSEndpoint: "ws://127.0.0.1:9222",
});

// The rest of your script remains the same.
const context = await browser.createBrowserContext();
const page = await context.newPage();

// Dump all the links from the page.
await page.goto('https://x.com');

const links = await page.evaluate(() => {
  return Array.from(document.querySelectorAll('a')).map(row => {
    return row.getAttribute('href');
  });
});

console.log(links);

await page.close();
await context.close();
await browser.disconnect();

which results in failure on the client

 95 |         }
 96 |         this.#navigationResponseReceived?.resolve();
 97 |     }
 98 |     #onFrameDetached(frame) {
 99 |         if (this.#frame === frame) {
100 |             this.#terminationDeferred.resolve(new Error('Navigating frame was detached'));
                                                    ^
error: Navigating frame was detached

because of this error in running lightpanda instance

info(server): accepting new conn...
info(server): client connected
warning(http_client): HTTP error: error.HeaderTooLarge (GET https://x.com/ 0)

[karlseguin]

x.com sends a single header line that's 10043 bytes. We currently limit individual header lines to 4096 bytes. Thinking out loud, we could:

1 - increase the maximum allowed header line
2 - ignore any very large header (with a log message)
3 - ignore this specific header content-security-policy for all websites

Any of these behaviors could be under a CLI flag.... e..g ./lightpanda --max_header_len 100000 --ignore_response_header 'content-security-policy'