Create dataset british_library_hertiage_made_digital_newspapers

[albertvillanova]

• uid: british_library_hertiage_made_digital_newspapers
• type: primary
• description:
  • name: British Library Hertiage Made Digital Newspapers
  • description: This is a collection of copyright-cleared 19th Century newspapers held by the British Library.
  • homepage: https://bl.iro.bl.uk/collections/353c908d-b495-4413-b047-87236d2573e3?locale=en
  • validated: True
• languages:
  • language_names:
    • English
  • language_comments:
  • language_locations:
    • Northern Europe
    • United Kingdom
  • validated: False
• custodian:
  • name: British Library Board
  • in_catalogue:
  • type: A library, museum, or archival institute
  • location: United Kingdom
  • contact_name: Daniel van Strien
  • contact_email: davanstrien@gmail.com
  • contact_submitter: False
  • additional: https://www.bl.uk/
  • validated: False
• availability:
  • procurement:
    • for_download: Yes - it has a direct download link or links
    • download_url: https://bl.iro.bl.uk/collections/353c908d-b495-4413-b047-87236d2573e3?locale=en
    • download_email:
  • licensing:

    • has_licenses: Yes

    • license_text: No Copyright - Other Known Legal Restrictions

      Use of this Item is not restricted by copyright and/or related rights. In one or more jurisdictions, laws other than copyright are known to impose restrictions on the use of this Item. Please refer to the organization that has made the Item available for more information.
      Notices

      Unless expressly stated otherwise, the organization that has made this Item available makes no warranties about the Item and cannot guarantee the accuracy of this Rights Statement. You are responsible for your own use.
      You may find additional information about the copyright status of the Item on the website of the organization that has made the Item available.
      You may need to obtain other permissions for your intended use. For example, other rights such as publicity, privacy or moral rights may limit how you may use the material.
      

      DISCLAIMER The purpose of this statement is to help the public understand how this Item may be used. When there is a (non-standard) License or contract that governs re-use of the associated Item, this statement only summarizes the effects of some of its terms. It is not a License, and should not be used to license your Work. To license your own Work, use a License offered at https://creativecommons.org/

    • license_properties:

      • public domain

    • license_list:

  • pii:
    • has_pii: Yes
    • generic_pii_likely: somewhat likely
    • generic_pii_list:
      • names
      • physical addresses
      • dates (birth, death, etc.)
    • numeric_pii_likely: none
    • numeric_pii_list:
    • sensitive_pii_likely: somewhat likely
    • sensitive_pii_list:
      • political opinions
      • trade-union membership
      • religious or philosophical beliefs
      • racial or ethnic origin
      • health-related data
      • data concerning a person's sex life or sexual orientation
    • no_pii_justification_class:
    • no_pii_justification_text:
  • validated: False
• source_category:
  • category_type: collection
  • category_web:
  • category_media: news articles
  • validated: False
• media:
  • category:
    • text
  • text_format:
    • other
    • ALTO XML
  • audiovisual_format:
  • image_format:
    • .TIFF
  • database_format:
  • text_is_transcribed: Yes - image
  • instance_type: A year of publications for a newspaper title
  • instance_count: 100<n<1K
  • instance_size: 100<n<10,000
  • validated: False
• fname: british_library_hertiage_made_digital_newspapers.json

[cakiki]

#self-assign

[davanstrien]

@cakiki give me a shout if you want any help with this? I am quite familiar with this dataset :)

[cakiki]

@davanstrien You've already helped a lot with your script which I used to download all the data. I'm currently uploading all the .zip files to the hub which will probably take a while.

(For the record the download script is the following: https://github.com/Living-with-machines/hmd_newspaper_dl)

[albertvillanova]

https://huggingface.co/datasets/bigscience-catalogue-data/british_library_heritage_made_digital_newspapers

[cakiki]

Done

[albertvillanova]

Thanks a lot @cakiki!!!

I just left a comment to address this issue later:

This dataset takes too long to load because of the data format inferring. This is due to the compression with zip and should be fixed if compressed with gzip instead.

ds_name = "bigscience-catalogue-data/british_library_heritage_made_digital_newspapers"
ds = load_dataset(ds_name, split="train", streaming=True, use_auth_token=True)

@lhoestq, maybe we should warn about this in the docs?

[cakiki]

Dataset came zipped. Should I convert everything to gzip?

Side question: what compression level would you recommend?

[lhoestq]

The dataset looks fine as ZIP, maybe we could optimize the data format inference so that it doesn't have to iterate over each single zip file. We can decide on a maximum number of files (possibly inside archives) to check for example ? WDYT @albertvillanova ?

[albertvillanova]

PR to fix the issue of taking too long to iterate over all data files:

• Use max number of data files to infer module huggingface/datasets#3407