Decision: Regulation subjects
| Thing | Info |
|---|---|
| Relevant features | Subject pages, search, regulation pages |
| Date started | 2024-05-01 |
| Date finished | 2024-05-21 |
| Decision status | Complete |
| Summary of outcome | Decided not to move forward on this |
We believe that if we enabled the following interactions, they could help some of our users with policy research.
- When a person looks up related guidance and rules in a regulation sidebar, they can also visit related subjects to see an overview of recent material in related subjects:
- When a person looks at a subject page, they see relevant regulation sections in the list of items (alongside resources), for example on a page like this:
- When a person searches for some keywords, the regulation sections in the results are annotated with subjects (similar to resources). If they filter the search results by subject, the results include relevant regulations. Example:
However, we don't have data that answers the question "What are the subjects related to a section?" (same question as "What are the sections related to a subject?").
We do have data that answers a different question: "What are the top subjects associated with the resources related to a section?"
Can we use the answer to "What are the top subjects associated with the resources that are tagged to a section?" to also answer "What are the subjects related to a section?"
If not, should we try a text-classification algorithm or build a way to hand-annotate regulation sections with subjects?
Our users need accurate and comprehensive information for their policy research. If we can't provide information that is accurate and comprehensive, we need to either frame the information carefully so that users have the right expectations or not show the information at all. As an example of careful framing, our statute page include links to SSA.gov but notes it was last updated in 2019, and we observed that at least some users do notice that date.
We have limited time and capacity for new feature development, so we need to focus on features that will have a big impact on making policy research easier.
We want to help people arrive on subject pages they might find useful, so we want to find useful ways to integrate relevant subjects into the regulation reading experience.
User stories:
- As a policy researcher, sometimes I need to find guidance and other materials related to the regulation section or subpart that I’m reading. I can look up related guidance and rules in the sidebar using the category system, but I might also find it useful to see an overview of recent material in related subjects.
- This means it's ok if a section about dental benefits has a related subject of EPSDT, because I might be looking at the regulation in the context of EPSDT. I could then go to the related subject and do a search or filter.
- As a policy researcher, sometimes I'm looking at the regulations to try to learn about a topic I have in mind. I may not be sure whether that topic is covered in regulations, statute, or guidance, or where to find the most recent information on the topic. If I can see a relevant subject that reflects what I have in mind, I can click it to learn about recent related resources, and that might help me answer my question.
- As a policy researcher, sometimes I need to look up regulations that I'm less familiar with and figure out how to interpret them. I'm probably interested in how these regulations are currently applied. If I could see a quick overview of recent related guidance, which I could get to by clicking related subjects, that could give me useful context for understanding the regulations.
It could be useful to enable finding regulations by subject because:
- On a combined search page, if you search for some keywords and filter the results by subject, the user could expect to see relevant regulations in the filtered results.
- Some topics are covered in various regulation sections in several parts, so collecting those sections together on a subject page could be helpful.
- Some regulations use language that isn't the everyday term for something - for example, "IMD exclusion" gets 0 search results in regulations, so we provide an imperfect workaround in the form of synonym suggestions:
However, subjects are less necessary for making regulations findable, compared to resources:
- In most cases, the regulations are well-structured with clear section titles.
- Regulations are organized by topic in their own way.
In user research, we've heard more about needing to look up information by subject for statute than regulations.
We can generate lists of related subjects for a section or subpart by counting the subjects associated with related documents. There's a fair bit of imprecision in the data:
- A bunch of documents cover multiple semi-unrelated topics, so they're tagged to multiple subjects and multiple regulation sections. This means that some of those subjects have nothing to do with some of the regulation sections. For example, the CIBs titled "Recent Developments in Medicaid and CHIP Policy", like this one, are a grab bag of topics.
- This is especially a problem with extremely long rules.
- A bunch of documents cover topics that are related to something in a tagged regulation section but not covered by that regulation. For example, top related subjects for 440.120 include Substance Use Disorder, but the section does not cover that topic explicitly.
- The feature is sensitive to changes in the data, and the data is incomplete - we have not tagged all of our documents with a complete set of subjects.
For the purpose of displaying "top subjects of related documents" in sidebars, our SME reviewed the preliminary data. We wanted to figure out if we can have confidence in the accuracy of automated lists. We could:
- Label the component clearly, so that the reader has enough context to reason about the information
- Limit it to showing the top X subjects for any subpart or section (TBD, draft is 5)
- Limit it to counting documents that have fewer than X related subjects, to limit the impact of long documents that cover tons of subjects (TBD, draft is 10)
- Limit it to showing subjects tagged to at least X documents (TBD, could be 5 or 10)
- Hide this component for any subpart or section with zero subjects that meet the criteria
But no matter what, there would be some slippage. We have to decide if it's acceptable slippage.
Subjects tagged to a regulation section would look just like our hand-annotated subjects on documents, so we don't have room for slippage there.
We could build hand-annotation: our devs could enable annotating regulation sections with subjects, and our SME could do the annotations.
We have so many parts, subparts, and sections that it could take a long time to hand-annotate all of them with subjects, and our SME capacity is limited. There is not much value in less-than-complete annotation, because people expect to get comprehensive results if they look up a subject. For example, if a person looked up EPSDT and saw one of the related sections but not all of them, that could be confusing and reduce trust in eRegs.
We could try NLP techniques for associating subjects with regulation sections, such as Fasttext text classification, SageMaker multi-label text classification (limited to 50 categories), or Comprehend multi-label text classification (limited to 100 categories).
We're not expecting to be able to classify documents fully automatically at high quality - to ensure accuracy, classification will always require SME review - but it might be good enough for sidebars.
We've considered enabling hand-annotation of subjects with related regulation and statute citations, to be displayed as a kind of caption at the top of the subject page (not as an item in the results list). That could serve some of the user needs, but it would be a fair bit of annotation work, and we weren't sure if it was useful enough to be worthwhile.
How long would it take to hand-annotate regulation sections with subjects? Could we use that to generate related sections at the subpart level, or would that need to be hand-annotated too?
We're not sure how we would display regulations on the subject pages - they don't have dates, so by default they would go to the end of the list in alphabetical order with the other undated items.
Should we use the top subjects data for annotating sections, for the purpose of displaying regulations on subject pages?
No, our SME does not think this data is usable for this purpose.
No, our SME does not think this data is usable for this purpose.
Could be useful, but it seems like too much work for not enough value.
Maybe, although it's unlikely to be a make-or-break feature for us. Probably more value in figuring out what we can do to help people find what they need in statute.
Could be interesting, but we'd need to revisit this later when we're ready to do some text classification work.
We did some usability testing for single-column search and subject page designs with the concept that regulations would be annotated with subjects, so we would like to re-test them without that aspect to see if the designs still make enough sense.
We implemented some experimental code but it was still buggy (Jira epic); we removed it in a refactor.