Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Idefics 3! #32473

Open
wants to merge 51 commits into
base: main
Choose a base branch
from
Open

Add Idefics 3! #32473

wants to merge 51 commits into from
+4,512 −0

Commits on Sep 18, 2024

  1. Add Idefics 3!

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    842a28d View commit details
    Browse the repository at this point in the history

  2. fixes to make both pipelines identical

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    afce007 View commit details
    Browse the repository at this point in the history

  3. fix for quantized models

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    3e3b31d View commit details
    Browse the repository at this point in the history

  4. First pass at the review

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    9c8ffc4 View commit details
    Browse the repository at this point in the history

  5. remove vocab size from the main config (it's still in the text_config)

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    7e3d7a6 View commit details
    Browse the repository at this point in the history

  6. hot fix for merve

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    dd99bca View commit details
    Browse the repository at this point in the history

  7. Apply suggestions from code review 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    ddac9ec View commit details
    Browse the repository at this point in the history

  8. re-add model_type for text_config

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    188bb76 View commit details
    Browse the repository at this point in the history

  9. remove support for old_cache

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    43fb214 View commit details
    Browse the repository at this point in the history

  10. remove hidden_size from main config

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    c9e0d85 View commit details
    Browse the repository at this point in the history

  11. rename idefics3 HF repo

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    1b2b89c View commit details
    Browse the repository at this point in the history

  12. few changes suggested in the PR

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    6ff766f View commit details
    Browse the repository at this point in the history

  13. fix to input_data_format computation

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    11c2e1a View commit details
    Browse the repository at this point in the history

  14. remove overwrite of _autoset_attn_implementation following @zucchini-nlp 

     suggestion
    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    c1048ed View commit details
    Browse the repository at this point in the history

  15. improve example

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    a163564 View commit details
    Browse the repository at this point in the history

  16. few improvements from amy's review

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    6f0a479 View commit details
    Browse the repository at this point in the history

  17. big change to enable processing input images as numpy arrays

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    8361fce View commit details
    Browse the repository at this point in the history

  18. Changes to the code to uniformize processor kwargs

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    32970d0 View commit details
    Browse the repository at this point in the history

  19. image processing tests

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    c504f00 View commit details
    Browse the repository at this point in the history

  20. image processing tests fixes and some bugs they discovered

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    a914e41 View commit details
    Browse the repository at this point in the history

  21. addressed review comments from Yoni

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    6722d13 View commit details
    Browse the repository at this point in the history

  22. fix modeling tests

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    0533eda View commit details
    Browse the repository at this point in the history

  23. remove special tokens that are not special

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b034091 View commit details
    Browse the repository at this point in the history

  24. fixes tests

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    47fb7ce View commit details
    Browse the repository at this point in the history

  25. skip failing tests - they also fail for idefics2

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    4032a6f View commit details
    Browse the repository at this point in the history

  26. added paper and readded the tests with multi gpu, who knows

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    757e834 View commit details
    Browse the repository at this point in the history

  27. Update docs/source/en/model_doc/idefics3.md 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    7797279 View commit details
    Browse the repository at this point in the history

  28. Apply suggestions from code review 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b478124 View commit details
    Browse the repository at this point in the history

  29. review amy until image_processing_idefics3

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    ada6219 View commit details
    Browse the repository at this point in the history

  30. last comments from Amy

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    164fbe8 View commit details
    Browse the repository at this point in the history

  31. review amy

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    000c8ea View commit details
    Browse the repository at this point in the history

  32. Update src/transformers/models/idefics3/image_processing_idefics3.py 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    4d02e0c View commit details
    Browse the repository at this point in the history

  33. Update src/transformers/models/idefics3/modeling_idefics3.py 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    3bf03c2 View commit details
    Browse the repository at this point in the history

  34. Update docs/source/en/model_doc/idefics3.md 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    57bfd51 View commit details
    Browse the repository at this point in the history

  35. doc improvement - amy review

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    63b1d7f View commit details
    Browse the repository at this point in the history

  36. fix runtime error during fine-tuning

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    6325fbc View commit details
    Browse the repository at this point in the history

  37. amy's review

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    76b8892 View commit details
    Browse the repository at this point in the history

  38. Update src/transformers/models/idefics3/image_processing_idefics3.py 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    9a20306 View commit details
    Browse the repository at this point in the history

  39. Update src/transformers/models/idefics3/image_processing_idefics3.py 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    3129920 View commit details
    Browse the repository at this point in the history

  40. Update src/transformers/models/idefics3/modeling_idefics3.py 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    e1a10b3 View commit details
    Browse the repository at this point in the history

  41. ruff

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    4c3756f View commit details
    Browse the repository at this point in the history

  42. amy's comment on the order

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    fbaf07e View commit details
    Browse the repository at this point in the history

  43. ruff ruff

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    87fa179 View commit details
    Browse the repository at this point in the history

  44. fix copies

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    23d4cf8 View commit details
    Browse the repository at this point in the history

  45. square images when they are not splitted

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    9e925b9 View commit details
    Browse the repository at this point in the history

  46. ruff :(

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    215b636 View commit details
    Browse the repository at this point in the history

  47. Update src/transformers/models/idefics3/image_processing_idefics3.py 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    2967974 View commit details
    Browse the repository at this point in the history

  48. Update tests/models/idefics3/test_processing_idefics3.py 

    Co-authored-by: amyeroberts <[email protected]>
    @andimarafioti @amyeroberts
    andimarafioti and amyeroberts committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    ee041bf View commit details
    Browse the repository at this point in the history

  49. fix small bug introduced in refactor

    @andimarafioti
    andimarafioti committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    4aad266 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. amy's image processing changes

    @andimarafioti
    andimarafioti committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    f1ae8ae View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. fixes peft tests and ruff

    @andimarafioti
    andimarafioti committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    39d88b2 View commit details
    Browse the repository at this point in the history