HTML UI Lambda memory usage

[rimutaka]

Too many HTML UI Lambda requests fail because serde uses too much RAM deserializing into Value.

Examples:

https://stackmuncher.com/?Rust multiset | Jan 2, 2022
https://stackmuncher.com/?Java com.thoughtworks.xstream.io.xml.CompactWriter | Jan 2, 2022
https://stackmuncher.com/quyixiao | Jan 2, 2022
https://stackmuncher.com/?Java javax.management.modelmbean.RequiredModelMBean | Jan 2, 2022
https://stackmuncher.com/?JavaScript octokit | Jan 2, 2022
https://stackmuncher.com/?Java org.junit.jupiter.api.extension.Extension | Jan 2, 2022
https://stackmuncher.com/?Java com.fasterxml.jackson.databind.introspect.Annotated | Jan 2, 2022
https://stackmuncher.com/?JavaScript print | Jan 2, 2022
https://stackmuncher.com/?JavaScript combinators | Jan 2, 2022

A potential solution can be converting into a struct first and then convert into Value. I do not know if it will produce a smaller object.

512MB results in <10% failure rate, but given that lists are larger than dev profiles it would be search results pages that fail the most.

[rimutaka]

2022-02-01 update

It is kind of tolerable.

1. The number of errors is low
2. The cost is $8 a month
3. 812MB lambda size

There is a discussion in serde-rs/json#635 regarding this same problem and a potential solution in https://github.com/Diggsey/ijson crate. It is not known if it will work with Tera. @Diggsey thinks it might (Diggsey/ijson#6).

Alternative solutions:

1. Experiment with shrinking Report struct
2. Try iJson crate
3. Split the lambda in 2: a smaller one for requests up to a certain size and a separate much larger sub-lambda to handle oversized requests.

None of the above is a quick fix for the problem.

[rimutaka]

Looks like Tera adds a significant overhead as well. This example includes double conversion: ES -> String -> struct Report -> Value.

2022-01-31T10:24:32.141962Z  INFO stm_shared::elastic: ES query 265 response: 10812199 bytes
2022-01-31T10:24:32.142242Z  INFO stm_html_ui::handler: RAM total KB: 16638595, used 3551578/10150, tot swap: 0, used swap: 0/0 - matching_devs call_es_api_raw
2022-01-31T10:24:34.812766Z  INFO stm_html_ui::handler: RAM total KB: 16638595, used 3615574/63996, tot swap: 0, used swap: 0/0 - matching_devs from_slice
2022-01-31T10:24:36.702271Z  INFO stm_html_ui::handler: RAM total KB: 16638595, used 3803736/188162, tot swap: 0, used swap: 0/0 - matching_devs to_value
2022-01-31T10:24:36.702435Z  INFO stm_html_ui::elastic: ES query `matching_devs` deserialized
2022-01-31T10:24:37.690586Z  INFO stm_shared::sqs: Sending msg to SQS /stm_search_stats
2022-01-31T10:24:38.518315Z  INFO stm_shared::sqs: Sent
2022-01-31T10:24:38.518618Z  INFO stm_html_ui::handler: RAM total KB: 16638595, used 3957745/417541, tot swap: 0, used swap: 0/0 - HTML data returned
2022-01-31T10:25:08.520125Z  INFO stm_html_ui::handler: Rendered
2022-01-31T10:25:08.520406Z  INFO stm_html_ui::handler: RAM total KB: 16638595, used 4194887/237142, tot swap: 0, used swap: 0/0 - Tera rendered
2022-01-31T10:25:08.520451Z  INFO stm_html_ui::handler: HTML full: 2682404 bytes
2022-01-31T10:25:09.323232Z  INFO stm_html_ui::handler: HTML mini: 129347 bytes
2022-01-31T10:25:09.323459Z  INFO stm_html_ui::handler: RAM total KB: 16638595, used 4195141/254, tot swap: 0, used swap: 0/0 - HTML minified

In this test where ES data is converted straight into Value there is no much memory use by Tera:

2022-01-31T22:47:44.035319Z  INFO stm_html_ui::handler: RAM total KB: 16638595, used 2340964/489177, tot swap: 0, used swap: 0/0 - HTML data returned
2022-01-31T22:48:15.240377Z  INFO stm_html_ui::handler: Rendered
2022-01-31T22:48:15.240694Z  INFO stm_html_ui::handler: RAM total KB: 16638595, used 2416855/75891, tot swap: 0, used swap: 0/0 - Tera rendered
2022-01-31T22:48:15.240752Z  INFO stm_html_ui::handler: HTML full: 2840100 bytes
2022-01-31T22:48:16.068844Z  INFO stm_html_ui::handler: HTML mini: 128870 bytes
2022-01-31T22:48:16.069060Z  INFO stm_html_ui::handler: RAM total KB: 16638595, used 2425719/8864, tot swap: 0, used swap: 0/0 - HTML minified