Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Da 24/reading comprehension #74

Merged
merged 74 commits into from
Dec 4, 2023
Merged

Da 24/reading comprehension #74

merged 74 commits into from
Dec 4, 2023

Commits on Nov 15, 2023

  1. Broken reading-comprehension generators and generator training

    @metric-space
    metric-space committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    298e100 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. Refactor out shared utils and add domain tokenizer training as a opti… 

    …on in direct script tun
    @metric-space
    metric-space committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    ce4a23f View commit details
    Browse the repository at this point in the history

  2. Corrections

    @metric-space
    metric-space committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    9987c06 View commit details
    Browse the repository at this point in the history

  3. Further corrections

    @metric-space
    metric-space committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    b842667 View commit details
    Browse the repository at this point in the history

  4. Add q&a extractor as a util and to the pipeline example script

    @metric-space
    metric-space committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    f6d253a View commit details
    Browse the repository at this point in the history

  5. Util correction and context addition to output

    @metric-space
    metric-space committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    571b41c View commit details
    Browse the repository at this point in the history

  6. Revert previous corrections

    @metric-space
    metric-space committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    5c349ca View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. Chatml outputformat for regex based rc and remove domain keyword

    @metric-space
    metric-space committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    f11797b View commit details
    Browse the repository at this point in the history

  2. Generator additions

    @metric-space
    metric-space committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    21b6210 View commit details
    Browse the repository at this point in the history

  3. More generator corrections and additions

    @metric-space
    metric-space committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    3c49970 View commit details
    Browse the repository at this point in the history

  4. Add train num epochs as an arg to the generator training script

    @metric-space
    metric-space committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    c7b63f6 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2023

  1. Add new dependencies

    @metric-space
    metric-space committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    8a9ec67 View commit details
    Browse the repository at this point in the history

  2. 1. json dump when writing to file 

    2. regex-based-gen now creates a domain tokenizer if both domain sentencepiece model and domain text (explicitly) is not given
3. attempt at pipeline
    @metric-space
    metric-space committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    c0c530f View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2023

  1. Post pipeline test run corrections

    @metric-space
    metric-space committed Nov 19, 2023
    Configuration menu
    Copy the full SHA
    fcdfa3b View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2023

  1. Remove (explicit) use of ConstantLengthDataset

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    0f7b7e0 View commit details
    Browse the repository at this point in the history

  2. Lift state management to a higer level and some corrections

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    7515b0a View commit details
    Browse the repository at this point in the history

  3. Add option to save dataset as huggingface dataset

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    75c7811 View commit details
    Browse the repository at this point in the history

  4. Training script corrections

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    979408c View commit details
    Browse the repository at this point in the history

  5. Trainer script cleanup and corrections

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    42f77f5 View commit details
    Browse the repository at this point in the history

  6. Add cloud friendly logger

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    de7f60c View commit details
    Browse the repository at this point in the history

  7. Reformatted synthetic dataset generation + corrections

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    ead9e7b View commit details
    Browse the repository at this point in the history

  8. More formatting for the synth-gen script

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    3efa61b View commit details
    Browse the repository at this point in the history

  9. More corrections to synth-gen

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    1d39491 View commit details
    Browse the repository at this point in the history

  10. Regex-gen changes and banner addition

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    3abc635 View commit details
    Browse the repository at this point in the history

  11. Missing comma

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    980cc6a View commit details
    Browse the repository at this point in the history

  12. Type hint all the functions in utils

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    1868423 View commit details
    Browse the repository at this point in the history

  13. Lightly refactor the pipeline

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    ce9b352 View commit details
    Browse the repository at this point in the history

  14. Address proper negation of lags

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    3760fc5 View commit details
    Browse the repository at this point in the history

  15. Switch out generator for iterator when type hinting

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    fe85703 View commit details
    Browse the repository at this point in the history

  16. Util typing correction

    @metric-space
    metric-space committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    331eaed View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2023

  1. Correct all linting issues

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    8c1a35a View commit details
    Browse the repository at this point in the history

  2. Pipeline corrections

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    e77d273 View commit details
    Browse the repository at this point in the history

  3. More corrections for output type of generator

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    2f91c25 View commit details
    Browse the repository at this point in the history

  4. More corrections to the pipeline

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    3b8e270 View commit details
    Browse the repository at this point in the history

  5. Appeasing the linter for the pipeline code

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    a37b5d3 View commit details
    Browse the repository at this point in the history

  6. Appeasing the linter for llm synth script

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    280e2dd View commit details
    Browse the repository at this point in the history

  7. Appeasing the linter for the training script

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    3facf92 View commit details
    Browse the repository at this point in the history

  8. Linter based corrections for utils

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    3f50761 View commit details
    Browse the repository at this point in the history

  9. More appeasing of the linter and work arounds

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    8bb10c4 View commit details
    Browse the repository at this point in the history

  10. Incorporate csv reading and associated changes

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    42330eb View commit details
    Browse the repository at this point in the history

  11. Unicode decoding revisit

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    194cf91 View commit details
    Browse the repository at this point in the history

  12. More fixes

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    451015b View commit details
    Browse the repository at this point in the history

  13. Forgot to put in replace line

    @metric-space
    metric-space committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    2e3fd08 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2023

  1. Better logging and removal of statefile and more corrections

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    9a430e5 View commit details
    Browse the repository at this point in the history

  2. Add missing general spm input validation line to pipeline script

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    3eb3b44 View commit details
    Browse the repository at this point in the history

  3. More validation lines for pipeline

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    14f65c5 View commit details
    Browse the repository at this point in the history

  4. More corrections

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    55f03ef View commit details
    Browse the repository at this point in the history

  5. Banner correction, corrections

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    4d5802d View commit details
    Browse the repository at this point in the history

  6. Start of README.md and add general sentencepiece model to resources

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    3f17238 View commit details
    Browse the repository at this point in the history

  7. Add defaults for cli args

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    88a74c1 View commit details
    Browse the repository at this point in the history

  8. Add more detail to README.md

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    4ab1211 View commit details
    Browse the repository at this point in the history

  9. Add defaults to function

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    b6fa0ae View commit details
    Browse the repository at this point in the history

  10. Defaults

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    644c294 View commit details
    Browse the repository at this point in the history

  11. README.md for rc pipeline

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    9c2f00d View commit details
    Browse the repository at this point in the history

  12. transformers version dependency constraint

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    1d4ec97 View commit details
    Browse the repository at this point in the history

  13. alpha -> beta

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    9171734 View commit details
    Browse the repository at this point in the history

  14. Better warning message

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    1af81cd View commit details
    Browse the repository at this point in the history

  15. Correct description in README.md

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    3039050 View commit details
    Browse the repository at this point in the history

  16. Stream arg correction for trainer

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    4d93d4a View commit details
    Browse the repository at this point in the history

  17. Add prompt link to README

    @metric-space
    metric-space committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    5fb1252 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2023

  1. Add general spm to resources

    @metric-space
    metric-space committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    958f1f4 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2023

  1. - Better input content generator (deals with directory of csv(s)) 

    - remove default wandb value
- new logging statement
- change of terminology for logging (files -> texts)
    @metric-space
    metric-space committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    749f11c View commit details
    Browse the repository at this point in the history

  2. Vocab size second-try and key error fix

    @metric-space
    metric-space committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    903ba18 View commit details
    Browse the repository at this point in the history

  3. Correct logging

    @metric-space
    metric-space committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    bc63323 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2023

  1. Update dalm/pipelines/reading_comprehension_pipeline.py 

    Co-authored-by: Traun Leyden <[email protected]>
    @metric-space @tleyden
    metric-space and tleyden authored Dec 2, 2023
    Configuration menu
    Copy the full SHA
    a6f0e6e View commit details
    Browse the repository at this point in the history

  2. Update dalm/pipelines/reading_comprehension_pipeline.py 

    Co-authored-by: Traun Leyden <[email protected]>
    @metric-space @tleyden
    metric-space and tleyden authored Dec 2, 2023
    Configuration menu
    Copy the full SHA
    0b89e1d View commit details
    Browse the repository at this point in the history

  3. Update dalm/datasets/reading_comprehension_generation/synthetic_based.py 

    Co-authored-by: Traun Leyden <[email protected]>
    @metric-space @tleyden
    metric-space and tleyden authored Dec 2, 2023
    Configuration menu
    Copy the full SHA
    340b969 View commit details
    Browse the repository at this point in the history

  4. Update dalm/datasets/reading_comprehension_generation/utils.py 

    Co-authored-by: Traun Leyden <[email protected]>
    @metric-space @tleyden
    metric-space and tleyden authored Dec 2, 2023
    Configuration menu
    Copy the full SHA
    9ae906d View commit details
    Browse the repository at this point in the history

  5. Update dalm/pipelines/reading_comprehension_pipeline.py 

    Co-authored-by: Traun Leyden <[email protected]>
    @metric-space @tleyden
    metric-space and tleyden authored Dec 2, 2023
    Configuration menu
    Copy the full SHA
    1d3ed52 View commit details
    Browse the repository at this point in the history

  6. Update dalm/pipelines/reading_comprehension_pipeline.py 

    Co-authored-by: Traun Leyden <[email protected]>
    @metric-space @tleyden
    metric-space and tleyden authored Dec 2, 2023
    Configuration menu
    Copy the full SHA
    a7ab91c View commit details
    Browse the repository at this point in the history

  7. Corrections

    @metric-space
    metric-space committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    9cb16ee View commit details
    Browse the repository at this point in the history

  8. Post linting

    @metric-space
    metric-space committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    4025e45 View commit details
    Browse the repository at this point in the history

  9. Update README with suggested corrections

    @metric-space
    metric-space committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    39b7f1e View commit details
    Browse the repository at this point in the history

  10. grammar corrections

    @metric-space
    metric-space committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    81a2ea0 View commit details
    Browse the repository at this point in the history