Adjust and/or increase scrape times?

[colinmahns]

Currently Planet updates for three scenarios: push/merge actions, pull requests (to test the generation of the website), and a cron set to 12:30 UTC / 09:30 ET / 06:30 PT every day.

From .github/workflows/jekyll.yml:

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]
  schedule:
    # * is a special character in YAML so you have to quote this string
    # We will pull once a day, UTC to start to not blow up our Github actions
    - cron: '30 12 * * *'

Our upstream feeds are currently only sparingly updated, some once a week, some less. I don't think it's worth periodically pulling updates as if we use too many actions we will accrue some costs and quite frankly if there's no updates to the feeds it's a wasted run. But as more chapters start updating their feeds (I've already anecdotally seen a few chapters are now posting updates for the first time in months), I think we can increase this to a couple updates a day. Maybe one in the morning and one in the evening, or even an afternoon pull if we see updates happen from these feeds.

Any suggestions for time? 09:30 ET was selected purely arbitrarily. Happy to adjust this to earlier for catching East Coast commutes and mornings, along with adding a mid or late day scrape.

[victoria-riley-barnett]

Do we have analytics hooked in? Recommend setting that up. Willing to bet theres a clear spike or two and as long as we pull close to but before those, once a day is probably fine.

[colinmahns]

No analytics. We approved but did not start implementing a Plausible instance just due to time, but would love to get that project kicked off if there's interest to help guide some of these decisions!

…

On August 30, 2023 9:45:00 PM EDT, Victoria ***@***.***> wrote: Do we have analytics hooked in? Recommend setting that up. Willing to bet theres a clear spike or two and as long as we pull close to but before those, once a day is probably fine. -- Reply to this email directly or view it on GitHub: #31 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>