Archiving relay with Internet Archive

[bmann]

For self hosted PDS, you can choose one or more upstream relays, with the Bluesky being the default.

What if we ran an explicitly archiving relay:

• connection to Internet Archive
• content / posts / blobs (Media) all archived there
• Per user flag for availability of content for training or other use cases
• Run Wayback machine on links at the time of posting

I say relay because this would be an independent way to connect PDS.

One could make another connection for Bsky accounts or other appviews that opt in

[Shrinks99]

Run Wayback machine on links at the time of posting

Alternatively, archive it yourself! Seems like something that Scoop or Browsertrix Crawler would be well suited for if grabbing the browser-based view is important. Firing off API requests to Webrecorder's Browsertrix instance is also a (paid) option! All of these produce WACZ files which are dead easy to view with no server-side setup required unlike a traditional "wayback" style viewer.

That said, despite having worked extensively to develop Browsertrix, my personal opinion regarding the archiving of social media (and especiallly regarding a protocol-based approach like AtProtocol) is that the website HTML and other interface components are less important to archive for every post, and that the content of the posts themselves are the important thing to capture. For a system like the Wayback Machine which is entirely built upon serving web archives this absolutely makes sense — posts are linked to on the internet, and therefore clicking a link within their archive viewer will reveal the archived webpage of the social post — but if one wanted to build an "Archive of Bluesky" I'd be very interested in going down the relay path as you mention. Maybe the idea of what constitutes a "web archive" can change as (with protocols) archives can become ever more integrated into a given network!

Bryan also made the interesting suggestion of putting CAR records inside WARC records. 👀

[bmann]

Yeah we already have repos, and an archiving relay would keep a copy of the source records and back it up — that’s what I meant.

The links in content is what would also be interesting to archive and could in fact be privately accessible as part of a full text search.

there are a variety of ways to archive. Outsourcing that to Wayback is one way.