Recently accessioned devel Package not reported on stats page

[rdshear]

Reported by @mtmorgan. I don't see AlphaMissenseR on the stats page. Two oddities. It's recently commited, so only in the 'devel' branch. And it doesn't have very many downloads (2 when I checked last week; I was hoping I had increased that by 50% by downloading again...)

[rdshear]

The packages table currently ends at the last published version number, which is currently 3.18. The creation of packages relies on scraping the names and dates of releases from the public web page https://bioconductor.org/about/release-announcements/. But this means that we do not have access to the development version, currently 3.19.

Solution:

1. Add entries to the Models.WebstatsInfo class (the webstats_info table) for current dev version and current production version.
2. Extend the current bioc_web_downloads ingestion process to pull the development package list https://www.bioconductor.org/packages/{version}/{category}/src/contrib/PACKAGES data and to update the packages table as appropriate.
3. Rerun the downloads-to-stats process for the period 2023-10-26 through current.
4. Create process for identifying next devel version number.

[rdshear]

Note: We need to determine if devel packages are deprecated before they get to prod. And if so, how their downloads are handled.

[hpages]

How about using this yaml file for that, as discussed earlier? https://bioconductor.org/config.yaml (see r_ver_for_bioc_ver entry)

[rdshear]

Probably related to #16.

[hpages]

Hey Bob, still no stats for AlphaMissenseR? https://bioconductor.org/packages/stats/bioc/AlphaMissenseR/

FWIW the package had a couple dozens of downloads in 2024 (see https://bioconductor.org/packages/oldstats/bioc/AlphaMissenseR/)

[hpages]

I should add that this breaks the "rank" shield on the package landing page e.g. on:

• https://bioconductor.org/packages/devel/AlphaMissenseR
• https://bioconductor.org/packages/devel/gypsum

Clicking on the shield produces an error 404.

Oh, it seems that even though no stats are available for these packages, they still get ranked (2220/2251). I guess that in the absence of stats they get ranked like if they had zero downloads, so they end up at the lowest rank (32 packages get that same rank). Nice @lshep!

Better than nothing but the real rank would be better @rdshear 😉

[lshep]

yeah the fall back is lowest rank and we tie them all but agreed real rank would be better

[lshep]

@rdshear was the scripts ever adjusted to use the devel list of packages instead of the release so we would catch these newly added packages?

[rdshear]

I have now adjusted the script to use config.yaml instead of scraping HTML from release_announements. I have reprocessed March 2024. Over the next day or two, I will reprocess the logs from the inception of 3.19.

The pages for new packages show up, but they do not show up in the index, probably related to @hpages observation above, which appears to be a logic error in the data model classes. The 404 is a separate matter in the facade layer, which i will deal with at the same time.

[hpages]

Just to note that packages recently added to devel are still not showing up in the new stats. E.g. packages survClust, tidyFlowCore, Pirat, MPAC, and many others were added to BioC 3.20 in June but they're not included in the stats. Should we keep this issue opened until this is actually solved?

[vjcitn]

survClust in devel has no stats. how are we going to fix this?

[rdshear]

packages now up to date.