PATCH: Increase the OEmbed sizeLimit from 250 KB to 1 MB

SAPI IG Articles have massive `<head>` elements, with massive `<style>` blocks, which push the `<meta>` tags we are interested in beyond vanilla’s 250 KB `fetch()` sizeLimit, causing Preview Links to miss out on important data. A copy of the fetched html is stored in `db.rocketchat_oembed_cache#data.content.body`, so increasing the sizeLimit impacts our storage requirements. In practice, the impact isn't so great. Pages are cached only for a week. On Mongo/Production, we have under 1500 docs, only 600 of which reach the 250k limit. Assuming _all_ of those would hit the higher 1 MB limit, that's an increased storage of 450 MB. ``` ((1 * 1024 * 1024) - (250 * 1024)) * 600 / 1024 / 1024 //=> 453.5 MB ``` TODO: An improvement on this would be to extract out only the `<head>` element from the html, strip out `<style>` and `<script>` elements from it, and store the rest.
seekingalpha · Jun 17, 2024 · 393a5d2 · 393a5d2
1 parent fffaaaf
commit 393a5d2
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/apps/meteor/app/oembed/server/server.ts b/apps/meteor/app/oembed/server/server.ts
@@ -96,7 +96,7 @@ const getUrlContent = async (urlObj: URL, redirectCount = 5): Promise<OEmbedUrlC
 	});
 
 	const url = data.urlObj.toString();
-	const sizeLimit = 250000;
+	const sizeLimit = 1 * 1024 * 1024;  // 1 MB. Was originally 250 KB, but SAPI MP articles exceed that 2-fold.  // 250000;
 
 	log.debug(`Fetching ${url} following redirects ${redirectCount} times`);