Remove UTA Stops And Ridership from SGID

[gregbunce]

Summary

• Internal Dataset Name: Transportation.UTA_StopsAndRidership
• External Dataset Name: UTA Stops and Most Recent Ridership
• ArcGIS Online Url: UTA hosts this layer, and therefore it does not need to be removed/deleted from UGRC's AGOL Org

UTA now has its own ArcGIS Enterprise Portal and is hosting this layer and sharing it with UGRC's SGID (SGID on ArcGIS) via Distributed Collaboration in UGRC's AGOL.

Migration Guide

The goal is to direct our users to UTA's web service, via the SGID Index and the SGID on ArcGIS.

Action items

1. Assign a person who should complete the task by replacing name with their github @name.
2. Check [x] the box when the task is completed and add the date of completion.
3. Strike out all items that do not apply.

Soft Delete

The purpose of the soft delete is to ensure that all of our users and applications have gracefully migrated off of the dataset. Soft deletes will remain in effect for 14 days. During this time, we will have the ability to restore the dataset to its original SGID offering(s). After these 14 days, the item is then ready for a hard delete.

Note: If this dataset is being replaced, then wait until the new data is publicly available before completing these steps:

• Append "(Mature Support)" to the end of the item title in the SGID.META.AGOLItems table (@gregbunce, completed: 2024/07/30)
• Remove all tags other than "Deprecated" in the SDE metadata (@gregbunce, completed: 2024/07/30)
• Add note to SDE description noting when layer will be deleted and any replacement layer (@gregbunce, completed: 2024/07/30)
• Update the SGID Index row to point to UTA's hosted data (@gregbunce, completed: 2024/07/30)
• Change Authoritative field to d in SGID.META.AGOLItems to automatically set the Deprecated AGOL flag. Allow the d to persist through one run of Auditor - currently, Auditor runs daily at 5:00am (@gregbunce, completed: 2024/07/30)
  • After at least one successful run of Auditor:
    • Remove the row from the SGID.META.AGOLItems table. This will trigger the removal of this item in Open SGID (@gregbunce, completed: 2024/08/01)
      - [ ] Unshare the item from its SGID group, which will remove it from the SGID on ArcGIS (@gregbunce, completed: 2024/00/00)
• Check if this layer in the base maps. If so, work with Zach to develop a path forward before you proceed with a Hard Delete (@ZachBeck, completed: 2024/08/12).

Hard Delete

Target Date: 2024/08/15

Hard deletes are final. It is recommended to complete the soft delete process before moving on to these steps. If you decide to skip the soft delete, note that you will need to incorporate some of those steps here.

• Manually remove data from the Internal SGID (@ZachBeck, completed: 2024/08/15)
  • Deprecated database layers should be backed up on Google Drive > AGRC Projects > SGID > deprecated layers > internal_db_sgid_layers (@ZachBeck, completed: 2024/08/15).
    - [ ] Remove ArcGIS Online item: (name, completed: 2024/00/00)
    - Manually delete the AGOL item
    - Deprecated AGOL items can be backed up on Google Drive > AGRC Projects > SGID > deprecated layers > agol_sgid_layers.
• Remove Farm from AGOL connection (@stdavis, completed: 2024/08/26)
  • https://github.com/agrc/warehouse/pull/150
• Update the SGID Index (@gregbunce, completed: 2024/08/15)
• Remove/delete the relevant gis.utah.gov data page (including downloadMetadata.ts) (@ZachBeck, completed: 2024/08/15)
• Update SGID.META.AGOLItems table, remove row (@ZachBeck, completed: 2024/08/15)
• Remove row from SGID.META.ChangeDetection (@stdavis, completed: 2024/08/26)
• Remove data from forklift hashing and receiving (@stdavis, completed: 2024/08/26)
• Remove row from data/hashed/changedetection.gdb/TableHashes (@stdavis, completed: 2024/08/26)

Shelve/Static

Choose one based on situation.

- [ ] Upload to UtahAGRC/AGRC_Shelved folder in AGOL (New shelved item not already in AGOL) (name, completed: 2024/00/00)
- [ ] Move existing AGOL item to AGRC_Shelved AGOL folder (shelving an item already in AGOL) (name, completed: 2024/00/00)
- [ ] Upload to appropriate UtahAGRC/{SGID Category} folder in AGOL (for static datasets) (name, completed: 2024/00/00)

Add record to table.

- [ ] Add record to AGOLItems_shelved table in ArcGIS Online with shelved or static in the CATEGORY field (name, completed: 2024/00/00)

🤖 Automation validation

1. Assign yourself or someone to check the item by replacing name with their github @name.
2. Check [x] the box and add the date of verification 2020/01/01 when the task is verified.
3. Strike out all items that do not apply.

• Remove data from Open SGID (@steveoh, on 2024/09/02)
• sgid-index (@steveoh on 2024/09/02)
• Auditor sets appropriate shelved/static/Deprecated information (@jacobdadams on 2024/00/00)

Are there service dependencies

• Are clients querying this data with the WebAPI?
• Update NG911 client maps with latest data sources (@gregbunce on 2024/08/15)
• Does any other application depend on this data?

Notification

• Twitter (@steveoh)

Group Task Assignments

1. Check [x] the box when you have assigned all the tasks relevant to your group.

• Data Team (@gregbunce)
• Dev Team (@steveoh)
• Cadastre Team (@rkelson)

[ZachBeck]

@gregbunce, @steveoh to me it seems like all this UTA data (trax, commuter rail, bus routes, stops etc) are worth having their own webpages and not just burying them in the sgid index.

[agrc-conductor]

conductor results for tasks - 336

check	status
@gregbunce has completed 1 out of 10 tasks	⛔
@ZachBeck has completed 0 out of 5 tasks	⛔
@stdavis has completed 0 out of 4 tasks	⛔
@steveoh has completed 0 out of 4 tasks	⛔
@jacobdadams has completed 0 out of 1 tasks	⛔
@rkelson has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for Transportation.UTA_StopsAndRidership

check	status
internal sgid	⛔
meta table	⛔
open sgid	⛔
open data	⛔
arcgis online	👍
stewardship
- porter issue link	⛔

[steveoh]

@gregbunce, @steveoh to me it seems like all this UTA data (trax, commuter rail, bus routes, stops etc) are worth having their own webpages and not just burying them in the sgid index.

Ok. Our current convention is to have pages on gis.utah.gov for data that we host. Is there a good enough reason to change that?

[ZachBeck]

No, not necessarily. I just hate to see some data get buried in the index.

[gregbunce]

I'd like to see the sgid search bar (sgid index) become front and center on the data page. to me, the index should hold the most complete view of the sgid.

[rkelson]

IMO I don't think there should be a hard and fast rule for what gets a webpage. I think if the steward would like us to promote their data with a webpage we should continue to do it, within reason. Welands is a good example. I am not sure what the UTA story is but my guess is that it is 'data come full circle'. While the Index is cool a webpage that documents the data (ie. history, codes, how to use, etc.) is still valuable. But if getting our website down to fewer pages is the plan so be it. My 2 cents

[steveoh]

I don't agree with the seemingly negative sentiment that adding items to the sgid index are burying it. It seems to me that the more distributed the SGID becomes the more important the index becomes. If we think we are burying data in it, then we are failing to provide a good service to finding data.

I do think we need a repeatable process that defines what gets a product page. Otherwise it becomes subjective and possible unfair. It is more overhead for us and ultimately more work for you. For instance, who is going to manage the metadata and update history for a product that is not managed by us and how do we want to do that?

The main things we promote on a product page are going to be highly duplicated on the ArcGIS Hub pages since there won't be an open sgid table. The ArcGIS Hub page that possibly exists in our Hub (distributed collaborate) and their Hub would have the most current information and would be discoverable through Hub's search and the index.

[gregbunce]

I also agree that the Hub pages are a good 'replacement' for our data pages, for data that we do not host. The key is working with the stewards to maintain relevant information (odd domains, use cases, update cycles, etc.).

[stdavis]

I wonder if part of the hesitation with the SGID index-only situation is that you have to know what to search for to find something. There's not a great story in our site for browsing datasets that are only in the index and don't have product pages. Maybe we could add links to open data pages in our website category pages for these datasets?

[gregbunce]

So that does bring up an another point. Currently the SGID index does not help the datasets out with search optimization, whereas data pages do.

[nathankota]

What about a simple website page that dynamically lists all of the index layers, to act as both a place to "browse" (or at least view) index datasets, and to help with search optimization?

[steveoh]

There's not a great story in our site for browsing datasets that are only in the index

Should all the items in the index display when their category is selected?

Maybe we could add links to open data pages in our website category pages for these datasets?

We decided during the migration to remove these as most of the existing items were out of date, broken, and not maintained by the steward.

But if they were maintained through the sgid index sheet, with it's validations, that could certainly help with the previous issues.

Currently the SGID index does not help the datasets out with search optimization

I don't see this as being a feature of the SGID index. The hub pages will be indexed by engines already and our outgoing links to their product pages do help their SEO.

[gregbunce]

I don't see this as being a feature of the SGID index. The hub pages will be indexed by engines already and our outgoing links to their product pages do help their SEO.

Glad to hear that. How do our outgoing links work to help their SEO? I was under the assumption that those links needed to be on a web page.

[steveoh]

How do our outgoing links work to help their SEO?

I'm not going to claim to be an SEO expert but I do know that a page with a good score linking to another page boosts the score of the destination page somehow. Search engine crawlers do have the ability to render javascript and the sgid index is a react component. And you are right the link doesn't show up until the box is typed into. So maybe all of that doesn't matter.

The previous sgid index did render every entry on load and was a filtered from that state.

[agrc-conductor]

conductor results for tasks - 336

check	status
@gregbunce has completed 7 out of 9 tasks	⛔
@ZachBeck has completed 0 out of 5 tasks	⛔
@stdavis has completed 0 out of 4 tasks	⛔
@steveoh has completed 0 out of 4 tasks	⛔
@jacobdadams has completed 0 out of 1 tasks	⛔
@rkelson has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for Transportation.UTA_StopsAndRidership

check	status
internal sgid	⛔
meta table	👍
stewardship
- porter issue link	⛔

[agrc-conductor]

conductor results for tasks - 336

check	status
@gregbunce has completed 7 out of 9 tasks	⛔
@ZachBeck has completed 0 out of 5 tasks	⛔
@stdavis has completed 0 out of 4 tasks	⛔
@steveoh has completed 0 out of 4 tasks	⛔
@jacobdadams has completed 0 out of 1 tasks	⛔
@rkelson has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for Transportation.UTA_StopsAndRidership

check	status
internal sgid	⛔
meta table	👍
stewardship
- porter issue link	⛔

[agrc-conductor]

conductor results for tasks - 336

check	status
gregbunce has completed 9 out of 9 tasks	👍
ZachBeck has completed 5 out of 5 tasks	👍
@stdavis has completed 0 out of 4 tasks	⛔
@steveoh has completed 0 out of 4 tasks	⛔
@jacobdadams has completed 0 out of 1 tasks	⛔
@rkelson has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for Transportation.UTA_StopsAndRidership

check	status
internal sgid	👍
meta table	👍
stewardship
- porter issue link	⛔

[gregbunce]

note about this layer and the sgid index: I used the existing row (record) for this layer in the sheet and updated the values with the newest information (name, urls, etc).

[agrc-conductor]

conductor results for tasks - 336

check	status
gregbunce has completed 9 out of 9 tasks	👍
ZachBeck has completed 5 out of 5 tasks	👍
@stdavis has completed 0 out of 4 tasks	⛔
@steveoh has completed 2 out of 4 tasks	⛔
@jacobdadams has completed 0 out of 1 tasks	⛔
@rkelson has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for Transportation.UTA_StopsAndRidership

check	status
internal sgid	⛔
meta table	👍
stewardship
- porter issue link	👍

[agrc-conductor]

conductor results for tasks - 336

check	status
gregbunce has completed 9 out of 9 tasks	👍
ZachBeck has completed 5 out of 5 tasks	👍
stdavis has completed 4 out of 4 tasks	👍
@steveoh has completed 2 out of 4 tasks	⛔
@jacobdadams has completed 0 out of 1 tasks	⛔
@rkelson has completed 0 out of 1 tasks	⛔

[agrc-conductor]

conductor results for Transportation.UTA_StopsAndRidership

check	status
internal sgid	⛔
meta table	👍
stewardship
- porter issue link	👍

[agrc-conductor]

conductor results for tasks - 336

check	status
gregbunce has completed 9 out of 9 tasks	👍
ZachBeck has completed 5 out of 5 tasks	👍
stdavis has completed 4 out of 4 tasks	👍
steveoh has completed 4 out of 4 tasks	👍
@jacobdadams has completed 0 out of 1 tasks	⛔
rkelson has completed 1 out of 1 tasks	👍

[agrc-conductor]

conductor results for Transportation.UTA_StopsAndRidership

check	status
internal sgid	👍
meta table	👍
stewardship
- porter issue link	👍

[agrc-conductor]

conductor results for tasks - 336

check	status
gregbunce has completed 9 out of 9 tasks	👍
ZachBeck has completed 5 out of 5 tasks	👍
stdavis has completed 4 out of 4 tasks	👍
steveoh has completed 4 out of 4 tasks	👍
@jacobdadams has completed 0 out of 1 tasks	⛔
rkelson has completed 1 out of 1 tasks	👍

[agrc-conductor]

conductor results for Transportation.UTA_StopsAndRidership

check	status
internal sgid	👍
meta table	👍
stewardship
- porter issue link	👍