Skip to content

This is a fifty-state survey on open data portals, who has them, and how they're rated. Emphasis is on Kentucky and was the origin of a Kentucky Open Data bill.

License

Notifications You must be signed in to change notification settings

RobWiederstein/open_data_leg

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

73 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Utah open data portal screenshot

The Open Data Movement

Introduction

Increased connectivity and greater bandwidth have provided for an ever growing amount of data access. Those who see this as an opportunity to make more informed and quicker decisions have a competitive edge. Governments are a leading purveyor of data which can often be paired with proprietary data for greater insight. In a 2013 report McKinsey and Company estimated that the value of open data could be 3 trillion dollars globally.

Open data is data that is available to the public for their unrestricted use. All levels and branches of government generate significant data. Examples might include hospital utilization rates, commercial vehicle inspections, and first responder dispatches. When released and accessible in machine readable formats, the data may provide socially valuable insights for effective government and commercially viable applications for software entrepreneurs.

United States' Open Data Developments

The United States renewed its committment to open data with an executive order signed by Barak Obama in 2013. In addition to giving some history of the open data movement, the order included specific implementation goals. One of the goals was for the Office of Management and Budget (OMB) to issue a memorandum implementing the principles in the executive order. The OMB memo directed agencies to use a "machine-readable, open" format, open licenses for end users, and common core, extensible metadata. (The memorandum is helpful and more understandable than statutory provisions.) A more recent development in this area occurred in January, 2019 when President Trump signed the Foundations for Evidence Based Policy Act.

Kentucky Open Data Initiatives & Rankings

Kentucky provides a portal for GIS data, a second portal for transparency data and a third portal for registering and operating a busienss. While there are many data sets available from different Kentucky agencies, there is no centralized portal as in other states for their dissemination, nor are they provided in many situations in a machine-readable or liquid format. The Center for Data Innovation ("CDI") ranked the states on many criteria. Kentucky was ranked 36th overall. When broken down by subject area, Kentucky was ranked

  • 1st for education data
  • 9th in transit systems
  • 12th in building energy efficiency
  • 12th in public access to government information
  • 16th in electronic health records
  • 20th for government financial data
  • 21st in e-government
  • 24th in smart meters
  • 26th information and data processing
  • 28th for e-prescribing
  • 30th in consumer devices
  • 30th in open data portals
  • 30th in statistics jobs
  • 32nd in anti_SLAPP data
  • 33rd in data science community
  • 34th enabling key technology platforms
  • 34th in energy usage data
  • 34th in computer science and statistics AP
  • 35th in open data 500 companies
  • 35th federal funding in for data science R & D funding
  • 36th overall
  • 38th in developing human & business capital
  • 38th for data availability
  • 39th in software service jobs
  • 44th in broadband
  • 45th in STEM degrees
  • 47th in data science job listings
  • 48th for legislative data

Kentucky's Current Technology Structure

Commonwealth Office of Technology

Issues regarding technology are assigned to the Commonwealth Office of Technology. KRS 42.724; KRS 42.726. The office was created within the Finance and Administration Cabinet. KRS 42.724. The head of the office is both its executive director and the chief information officer for the state. While appointed by the Secretary of the Finance and Administration Cabinet, the appointment must be approved by the Governor. KRS 42.724.

Among its many responsibilities, the Office of Technology is to develop

  • "strategies and policies to support and promote . . . electronic public access to information of the Commonwealth." KRS 42.726(2)(c).
  • establish "a central statewide geographic information clearinghouse to maintain map inventories." KRS 42.726(2)(j).
  • provide "staff support and technical assistance to the Geographic Information Advisory Council and the Kentucky Information Technology Advisory Council." KRS 42.726(2)(n).
  • submit an annual report to the LRC in accordance with KRS 57.390.KRS 42.726(5)

The Commonwealth Office of Technology is advised by two statutorily established boards: the Geographic Information Advisory Council and the Kentucky Information Technology Advisory Council.

Executive Branch Reorganization

Kentucky's executive branch underwent signficant reorganization via an executive order entered by Governor Bevin in 2017. The Executive Director of the Commonwealth Office of Technology declared as the Chief Information Officer of all agencies within the Executive Branch of the state. All "agency information technology leads" are to report to both their current cabinet/ agency head and to the Commonwealth Office of Technology's Chief Information Officer.

Creation of the Chief Data Officer

Another significant policy was adopted in 2019. CIO-110 names a Chief Data Officer (CDO) to manage the Commonwealth's data. The CDO is responsible for maintaining standards and best practices with regard to data and acts as the Chief Data Steward as well. Executive branch agencies shall designate at least one Agency Data Steward that (1) reports directly to the Cabinet Secretary or Agency Head; (2) acts as the liaison to the CDO/ODIA; and (3) is responsible for data stewardship within the agency.

Creation of Agency-level Data Stewards

Within CIO-110, the position of "agency data steward" was created. According to the policy, the responsibilities of the position include the following:

  • Assisting in the development of the initial data inventory, including identification of high-value data elements, and shall ensure their agency supports the maintenance of the inventory in a timely fashion.
  • Developing and maintain a data source inventory describing and categorizing the data created or collected by the state agency, including geospatial data used in a state agency’s geographic information system (GIS)
  • Developing and maintain an open data catalog and machine-readable open datasets
  • Developing and maintain an inventory of all interfaces that describes inbound or outbound datasets generated, aggregated, stored, purchased, or shared by the state agency
  • Submitting agency’s data warehousing, data analytics, and data visualization plans to the CDO for approval prior to procurement or execution of such activities
  • Enforcing Commonwealth Data Management Policy, Data Management Standards, and Data Quality Policies and Standards, within the agency
  • Participating in Commonwealth Data Governance and Management workgroups and subcommittees as appropriate
  • Participating in Commonwealth Data Plan activities and lead Agency Data Management planning, including the development of roadmaps and tracking, reporting, and managing roadmap implementation.

All data-sharing partners to the Commonwealth, as well, shall designate a Data Steward (or the equivalent) and provide data inventory and maintenance support within the limits of their data sharing agreement.

Office of Technology Organization Chart

Machine Readable, Liquid and Shareable Formats

According to the Open Data Handbook, machine readable data can be automatically read by a computer like csv (comma separated values), json (java script object notation), or xml (extensible markup language). The opposite of machine readable is human readable and would include non-structured data like handwriting and pdfs. While pdfs are digital, a computer struggles to convert pdf tables into a machine readable format. While the conversion is possible, it requires additional steps and can result in errors. (For example, Tabula can convert pdf tables into a csv format, but it is time consuming and designed for smaller data sets.) In the Kentucky Administrative Regulations, "machine-readable" is defined as "information in the form of magnetic code or optical image that can be processed directly by computers and other related machines." 1 KAR 5:010 Accession of public records by means of electronic data processing procedures.

50 State Comparison Open Data Portals

The sites tend to be of three kinds: transparency, geographic information system (GIS) or the more comprehensive open data portal. Much of the table below was prepared by Meta S. Brown, States Offer Information Resources: 50+ Open Data Portals, Forbes (Apr. 30, 2018). In some states, both the sites for GIS and open data were included. The data were then merged with the open data portal rankings by the Center for Data Innovation ("CDI").

There were six top ranked states. The CDI stated that these six states were highlighted because "they have specific open-data policies, open-data portals and machine-readibility written into their open data policies." Incorporating the requirement of machine-readibility into law allows developers immediate access to usable data. The McKinsey report, mentioned above, uses the phrase "liquid" instead of machine-readable. Too often Kentucky's data, where available, is in a pdf format making it very difficult to convert to a machine-readable format.

state url type data_portal_rank
Hawaii http://data.hawaii.gov/ data 1
Illinois http://data.illinois.gov/ data 1
Maryland https://data.maryland.gov/ data 1
New York https://data.ny.gov/ data 1
Oklahoma http://www.ok.gov/about/data.html data 1
Utah http://www.utah.gov/data/ data 1
Connecticut http://transparency.ct.gov/html/main.asp data 7
Texas https://data.texas.gov/ data 7
New Hampshire http://nhopengovt.org/ data 9
Rhode Island http://www.ri.gov/data/ data 9
California http://data.ca.gov/ data 11
Michigan http://www.michigan.gov/data/ data 11
Missouri http://data.mo.gov/ data 11
New Jersey https://data.nj.gov/ data 11
Oregon http://data.oregon.gov/ data 11
Vermont https://data.vermont.gov/ data 11
Vermont http://geodata.vermont.gov/ data 11
Colorado https://data.colorado.gov/ data 17
Delaware https://delaware.gov/topics/data.shtml data 17
Indiana http://www.stats.indiana.edu/ data 17
Iowa http://data.iowa.gov/ data 17
Maine http://www.maine.gov/data/ data 17
Minnesota http://mn.gov/portal/ data 17
Montana http://data.mt.gov/ data 17
Montana http://gis-mdt.opendata.arcgis.com/ gis 17
Nebraska https://www.nebraskamap.gov/ gis 17
Nebraska http://www.nebraska.gov/government/open-data/ data 17
North Carolina http://www.ncopenbook.gov/ data 17
Ohio http://transparency.ohio.gov/ data 17
Virginia http://data.virginia.gov data 17
Washington http://data.wa.gov/ data 17
Wisconsin https://legis.wisconsin.gov/ltsb/gis/data/ gis 17
Arizona http://openbooks.az.gov/ data 30
Arkansas http://transparency.arkansas.gov/ data 30
Florida https://www.floridahasarighttoknow.myflorida.com/ data 30
Florida http://myflorida-floridadisaster.opendata.arcgis.com/ gis 30
Georgia http://www.open.georgia.gov/ open 30
Idaho http://data.gis.idaho.gov/ gis 30
Kentucky https://kygeoportal.ky.gov/geoportal/catalog/main/home.page gis 30
Kentucky https://transparency.ky.gov/Pages/default.aspx data 30
Mississippi http://opendata.gis.ms.gov/ gis 30
New Mexico http://www.sunshineportalnm.com/ data 30
North Dakota http://www.nd.gov/gis/ gis 30
Pennsylvania https://data.pa.gov/ data 30
South Carolina http://www.sc.gov/Pages/dataDashboard.aspx data 30
South Carolina http://www.gis.sc.gov/data.html gis 30
Tennessee http://tn-tnmap.opendata.arcgis.com/ gis 30
West Virginia http://www.mapwv.gov/help-gisdata.php gis 30
Alabama http://open.alabama.gov/ data 43
Alaska https://data.alaska.gov/ data 43
Kansas http://www.kansas.gov/KanView/ data 43
Louisiana https://wwwcfprd.doa.louisiana.gov/latrac/portal.cfm data 43
Massachusetts http://www.mass.gov/opendata/#/ data 43
South Dakota http://open.sd.gov/ data 43
Wyoming http://geospatialhub.org/ gis 43

Enacted Open Data Legislation in Top States

Hawaii

Hawaii's open data policies are codified in chapter 27. The statute defines "data" and "data sets". Hawaii Rev. Stat. Sec. 27-41.1. It establishes an "information technology steering committee", consisting of eleven members. The 11 members include four from the house, four from the senate, one member appointed by the Chief Justice, one member appointed by the governor, and representatives from the executive branch departments. Hawaii Rev. Stat. Sec. 27-43.

"Each executive branch department shall use reasonable efforts to make appropriate and existing electronic data sets . . . electronically available . . . through the state's open data portal." Hawaii Rev. Stat. Sec. 27-44. The statute specifies that no "new" data sets must be created, nor does it require data sets to be available on demand. It does require that the data sets be updated as "often as is necessary to preserve the integrity and usefulness of the data."

With regard to future liability and potential warranties, the statute states that "[d]ata sets shall be available for informational purposes only. The State does not warrant the fitness of any data set for a particular purpose and shall not be liable for any deficiencies in the completeness or accuracy of any data set, except where the State's conduct would constitute gross negligence, wilful and wanton misconduct, or intentional misconduct." Hawaii Rev. Stat. Sec. 27-44.1.

Hawaii's chief information officer is charged with developing policies and procedures including "[t]echnical requirements with the goal of making data sets available to the greatest number of users and for the greatest number of applications, including whenever practicable, the use of machine readable, nonproprietary technical standards for web publishing; and guidelines for departments to follow in making data sets available." Hawaii Rev. Stat. Sec. 27-44.3.

Illinois

The adopted statute defines "cloud-computing", "data", "open operating standard", "public data", and "voluntary consensus standards body" among others. The implementation portion of the statute reads, "There is hereby established an open operating standard to be known as the Illinois Open Data, for the State of Illinois. Under this operating standard, each agency of State government under the jurisdiction of the Governor shall make available public data sets of public information." Another section requires that all data be located at a single url address and reads, "[p]ublic data sets that are made available on the internet shall be accessible through a single web portal that is linked to data.illinois.gov."

The data sets must be formatted in a way that allows the public to be notified of updates; updated regularly; made available without registration, license requirements, or usage restrictions; and discoverable via external searches. The statute requires an "API" or application programming interface for sending and retrieving data. The plan includes extensive deadlines on the executive branch. It does not use the term "machine-readable", but rather leaves formats to the discretion of the executive. Cloud-computing receives preference as it is considered a more cost effective solution. The data does not include a warranty of any kind and is assigned to the public domain.

Maryland

The Maryland statute defined the following terms: data, data portal, data set, mapping and geographic information systems portal, open data, open data portal. The statute declared the state's policy is that "open data be machine readable and released to the public in ways that make the data easy to find, accessible, and usable . . . ." (While the purpose included "machine readable", the term was not included in the definitions). A Council on Open Data was created with 37 members. The Council's main charge was carry out the express policy of the statute and make recommendations on legislation and regulations.

New York

The New York statute contains many definitions including: data, data set, publishable state data, and technical standard. The enacting portion of the statute provides:

An online open data website for the collection and public dissemination of publishable state data is hereby established in the office of information technology services. The open data website shall be maintained at data.ny.gov . . . . The open data website will provide access to publishable state data that is owned, controlled, collected or otherwise maintained by covered state entities.

The statute also provides a time line with a "data working group" to be formed 45 days after the act becomes law. Within 60 days, each agency of the executive branch designate a data coordinator who can act on behalf of the agency. Within 180 days, the Office of Information Technology shall publish guidelines for the purpose of making public data widely available. Within one year, all publishable state data will be available on the web site.

All "publishable state data" shall be "in a format that permits automated processing." Like the Illinois statute, New York requires the data sets to be formatted in a way that allows the public to be notified of updates; updated regularly; made available without registration, license requirements, or usage restrictions; and discoverable via external searches. The statute also requires an application programming interface like Illinois. New York boasts 1600 data sets as of January, 2019.

Oklahoma

"Open technology standards", "application programming interface", "performance information metrics" and "convenience information sets" are all defined under the Oklahoma statute. 62 Ok. Stat. Sec. 34.11.1(I).

The Chief Information Officer (CIO) is required to "source and submit" to the "State Governmental Technology Applications Review Board" (hereinafter "Technology Board"), employee performance metrics, convenience information sets and other data streams for publication on Oklahoma's open data portal. 62 Ok. Stat. Sec. 34.11.1.1.1(A).

The Technology Board is to establish "open technology standards" and a schedule for state agencies to adopt the standards. The state agencies are to publish and update data sets which "shall be accessible through standardized application programming interfaces and published in standardized formats" like extensible markup language (XML) and comma separated values (CSV). 62 Ok Stat. Sec. 34.11.2

The open data portal should give priority to those data sets that are most often requested by the public.

Utah

Utah updated its approach to public records and data in a 2014 bill. Utah's open record policies are promulgated by a "Transparency Advisory Board" comprised of 13 members and they are charged with specific duties. These duties revolve around what information is included and how it is to be presented to the public.

The 2014 legislation provided for an "information website", meaning a "single Internet website containing public information or links to public information." Regarding the information website, a new responsibility was added but all are set forth in total here:

  • study the establishment of an information website and develop recommendations for its establishment;
  • develop recommendations about how to make public information more readily available to the public through the information website;
  • develop standards to make uniform the format and accessibility of public information posted to the information website; and
  • identify and prioritize public information in the possession of a state agency or political subdivision that may be appropriate for publication on the information website.

In a fiscal note, the approximate one-time cost was $75,000 and the annual cost was $540,000.

Other Helpful resources

resources url note
Best States for Data Innovation https://www.datainnovation.org/2017/07/the-best-states-for-data-innovation/ States ranked for technology.
Center for Data Innovation https://www.datainnovation.org Excellent resource.
Code for America https://www.codeforamerica.org Making government work for people.
NCSL Open Data http://www.ncsl.org/research/telecommunications-and-information-technology/open-data-legislation.aspx 2010 through 2018 Legislation.
Open Knowledge Int'l https://okfn.org Global not-for-profit that promotes free, open data.
White House Open Data Repo https://github.com/project-open-data White House Open Data site.
Socrata https://socrata.com Cloud-based database services for public sector.
Sunlight Found. Open Data Policies https://sunlightfoundation.com/opendataguidelines/ Holding gov't accountable.
U.S. States Open Data Census https://census.usopendata.org User Census of State Datasets

Weaknesses in the Evaluation

This writeup depends heavily on the CDI's rankings. The legal research focused on those states with a high ranking from the CDI. The organization is is headed by Daniel Castro and affilliated with Information Technology and Innovation Foundation (ITIF). The ITIF is a 501(c)(3) and its latest 990 tax return can be found here. The author is unaware to the degree that this organization is affiliated with any particular industry or political group.

Conclusion

Government data is valuable. It belongs to the public and should be widely distributed, easily accessible and intensely comprehensive. Many states have created a centralized location to serve as a repository for this data; formatted it so that it can be shared; and created a board to shepherd the data to the public. While Kentucky has taken strong steps in the creation of specialized portals, Kentucky can do more. Kentucky should centralize its data in an open data portal in machine readable formats. Creating a data-driven state government can spur innovation, improve decision making, drive government transparency and increase civic engagement.

(To suggest improvements or correct errors, please open an issue on github).

About

This is a fifty-state survey on open data portals, who has them, and how they're rated. Emphasis is on Kentucky and was the origin of a Kentucky Open Data bill.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages