Federal Register link grouping
Our readers want to find Federal Register documents (Final Rules and Notices of Proposed Rulemaking, known collectively as rules) that changed (or proposed to change) the regulations that they’re reading.
Some regulation sections end with a list of citations to Final Rules that changed them, but it takes tedious work to dig up related NPRMs and corrections. Some regulation sections only include citations to Final Rules from the past 10 or 20 years, and other sections don't have any citations. You can easily miss something, which is not good if you're doing policy analysis.
Example of citations as shown on eCFR (42 CFR 438.3):
- I want to see the most recent NPRMs and Final Rules for all parts in scope, ordered by the most recent publication date of any document.
- For each Final Rule, can you also display the NPRM that it’s based on?
- If an NPRM is based on an earlier NPRM, can you also display that next to the new NPRM?
- I want to see all NPRMs and Final Rules that changed (or proposed to change) any section in this subpart.
- I want to see all NPRMs and Final Rules that changed (or proposed to change) this section.
To help our readers find rules relevant to the regulations they are researching, we provide subpart-by-subpart and section-by-section lists of relevant FR links in a sidebar next to the regulation text.
FR links have relationships to each other, so it’s not sufficient to provide a list of FR links sorted by most recent date. Our readers often need to find an NPRM associated with a Final Rule that changed the subpart or section they are researching. In regulation sidebars, we provide groups of relevant Federal Register documents so they can easily find the NPRM (or other document) that they need, without having to dig around the Federal Register.
This sidebar for 438.3 provides easy access to NPRMs and corrections that are not in the end-of-section citations:
This diagram shows fictitious but realistic examples of complex relationships between FR links. Our users need to navigate these relationships while doing policy research. This is why we built a robust system for grouping and ordering FR links instead of sorting them in raw chronological order.
This system uses our comprehensive database of rules related to Medicaid and CHIP. See Resources linking system: Federal Register links and New regulation part checklist.
Simplified explanation of how the application displays FR links in organized lists (regulation sidebars and homepage):
- Create groups of FR links
- Take the list of items in the Final Rules and Proposed Rules categories.
- In this list of items, find groups of items with docket numbers that start with the same XXX-### pattern (any number of characters + a dash + any number of characters before the next dash). See Federal Register docket number structure.
- Store each group of items.
- Also, a person can create or update groups by hand.
- Treat each ungrouped item as a group of one.
- Select groups for display
- For each group, make a list of the regulation sections and subparts that are referenced in any of the items in that group.
- Store that section list with the group.
- When viewing a regulation section or subpart, retrieve any groups with a matching citation number in their regulation citation list, and display those groups under the heading “Proposed and Final Rules.”
- Sort items for display
- Among all the selected groups, retrieve the group containing the item that has the most recent document publication date. Put that group at the top of the list.
- Do the same thing again – ignoring that initial group, grab and display the group that contains the item with the most recent publication date. Etc.
- Within each group, sort the items in reverse chronological order according to document publication date (newest at the top).
- Filter items for display
- If this is the homepage, only display the three newest groups by publication date.
- If this is a sidebar, show all groups.
When our system imports a new Federal Register link (via our overnight fetch-and-parse job), the system uses each document's docket number to automatically group related items together. The system creates a group from the XXX-### pattern (any number of characters + a dash + any number of characters before the next dash) at the beginning of a docket number.
Technical details: The system’s serializer runs when the parser adds a new document, and when the parser updates an existing document. The system checks if a group exists, and if not it makes a new group. The system runs a method to check for any matching identifiers (shared docket numbers) on existing documents and combines matching groups, creating a group with all of those matching documents. It uses the first group as the primary docket number for the new, combined group and deletes the other groups.
This process sometimes results in creating groups with a large number of semi-related documents. Team members may hand-correct these to split into smaller groups.
Within the Django admin system, we can create new groups and edit or delete existing groups, whether the groups were created manually or automatically.
This section is about what happens in different scenarios. Note: EREGCSC-2767 is a bug with multiple groups, so don't test something that depends on removal with multiple groups until that is fixed.
- If the system imports a new FR link that does not have a docket number, our system does not automatically associate it with a group.
- If a person manually creates an FR link, they may choose not to associate it with a group. For example, an old NPRM may have never had a Final Rule.
FR links without groups always show up as an individual item without “parent” or “child” FR links. Our system:
- Adds it to regulation sidebars if it has matching citations
- Adds it to the homepage in the Rules tab if it was recently-published
If the system imports a new FR link and associates it with a group, and it is the only item in its group, our system:
- Adds it to regulation sidebars if it has matching citations
- Adds it to the homepage in the Rules tab if it was recently-published
If the system imports a new FR link and associates it with a group that contains additional links, and any of the documents in that group have matching regulation citations, that new link will be added to those regulation sidebars, plus any additional matching citations on the new link.
When a resource in a group is saved, either manually or with the parser, the system updates all of the resources in any of the groups that are associated with that link, setting the value for whether the document is a group parent to false. Then it loops through those documents, orders them by date in descending order, and updates the first one of each group, setting the value of whether that first document is a group parent to true.
This group parent value for any group is what determines which document is unindented, if multiple documents in the group are displayed.
- Testing 79 FR 13887, which is currently in CMS-2380-group and appears in the regulation sidebar on 600.100 as the first, non-indented result, of four FR links
- Removing it takes it out of the regulation sidebar, along with all of the other items in that group
- Adding it back to the CMS-2380-group results in it going back to the first, non-indented position in that same sidebar
- Removing 79 FR 14111, which is currently the first indented rule in 600.100, from the 2380 group keeps it in that same sidebar, but it moves it to the bottom even though that does not match its date order
- Adding it back to the CMS-2380-group moves it back to the first indent position
When a resource in a group is saved, either manually or with the parser, the system creates a list of related resources: documents that are in any of the groups that the saved document is in.
Once it has these related resources, it bundles together all of the regulation citations, and subjects from those resources. All the documents in a group get those citations. The regulation sidebar displays groups based on the parent of the group. If the parent is associated with that citation, the group will display on the sidebar where that citation is.
This grouped list of citations does not appear on the FR links themselves; it is an implicit attribute of the group (and visually, it is an implicit part of how the given regulation sidebar displays).