Add support for markdown file input

[tkhongsap]

Hi there,
I hope this works this time.

1. Added markdown_extractor.py for extracting content from Markdown files.
2. Modified content_extractor.py Markdown parsing features.
3. Added and updated unit tests (test_markdown_extractor.py, test_content_parser.py) and mock data for Markdown files.
4. Kept API calls to Gemini and OpenAI as they are, with the language still set to English.
5. Updated README.md

I ran I ran " poetry run pytest" and had no errors;

[souzatharsis]

Please remove any audio or transcripts generated during tests

…

On Mon, Oct 21, 2024, 7:46 AM tkhongsap ***@***.***> wrote: Hi there, I hope this works this time. 1. Added markdown_extractor.py for extracting content from Markdown files. 2. Modified content_extractor.py Markdown parsing features. 3. Added and updated unit tests (test_markdown_extractor.py, test_content_parser.py) and mock data for Markdown files. 4. Kept API calls to Gemini and OpenAI as they are, with the language still set to English. 5. Updated README.md I ran I ran " poetry run pytest" and had no errors; ------------------------------ You can view, comment on, or merge this pull request online at: #93 Commit Summary - 984dcd7 <984dcd7> Updated README, modified content parser and tests, added markdown extractor. File Changes (55 files <https://github.com/souzatharsis/podcastfy/pull/93/files>) - *M* README.md <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5> (15) - *A* data/markdown/01-Prompt-Engineering.md <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-b0c811004bfa3a8ea5821bb8a61f2b2b8d3fadbabf21c5be11036d5f9b026026> (501) - *M* podcastfy/client.py <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-b9748df95849c730d5a23f5f987a42c889427eca44c27ef3cdc52d9c9d101519> (2) - *M* podcastfy/content_parser/content_extractor.py <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-877ae2f080ea9e9cf27f5004f74aebcdca47bf7b20350db6c94e6624315abe92> (20) - *A* podcastfy/content_parser/markdown_extractor.py <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-75eea75fa6e70787ebe7a862471b017329bba08dc8bfe92ed0321cd9f2cd4cff> (71) - *A* tests/data/audio/podcast_111fc2ce7bf34a6e8f17bc444f1a830e.mp3 <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-29c6328692dc4b325a493531d596a5781ca612930b57d9fad38141c24953b301> (0) - *A* tests/data/audio/podcast_3ce535c98f1f4e6ca8e40f6183f74e5d.mp3 <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-7c05b74614fe7fb9404c5a2357e44c33f8239d50a99e577ac6d9463c4e7a9000> (0) - *A* tests/data/audio/podcast_7a83c9f12c6844fdb473ea783e2104d4.mp3 <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-4a80b1a4b14d58c17c70e31e8f9eaf5acfc6c03f496ffd79e48949780c542007> (0) - *A* tests/data/audio/podcast_829bcbe557274040a77609683bc9d46e.mp3 <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-dc2dc7a7c04c677b967fd79c63a13f6c52e2fec22b6db5c1c1b8ca9841cfb9c6> (0) - *A* tests/data/audio/podcast_9cee601aa46843078878728e0cec2867.mp3 <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-ae3db3560512bc74eeda1c080385c347cb76fdb776004aa97057da03a44c31f1> (0) - *A* tests/data/audio/podcast_a56e5aa3cabf490a8c9d8e887a553844.mp3 <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-7c14ea99ea874bff5ad7752585c297e49fc2036864a86c6ac8fa18203534490c> (0) - *A* tests/data/audio/podcast_ac1c9d9519664cf89884815dbe8698c3.mp3 <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-ff5842e25d28e672c63b0e569e75f67be906f867faa21b5544097896ba3d2d35> (0) - *A* tests/data/audio/podcast_da247d4a92334b98a8d453a0441a7d80.mp3 <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-33ec50b1e64564b30a9b753c0503d6d09614465bcc3cab63d64934804122b477> (0) - *A* tests/data/markdown/file.md <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-e845a9beaa1ae6b6019777756756c8de4b7f7629abe9da426ee090f080b30584> (501) - *A* tests/data/mock/markdown.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-8f7c68a3744a6c28d9e9c206c0df375d27c3b6af9abe38a6eca35b7586b43786> (9) - *A* tests/data/transcripts/test_transcript.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-17a1365bc08e2ae7cd837aa1a3527b90e93753333021f4e6dfa2f1983dc0e6ff> (1) - *A* tests/data/transcripts/transcript_00a8a3f89eb9404183a3714ee7be9818.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-1ab27ee9949601a7d3d33be92da6cb52928f9458a0fa2943fe2e4aec85e6e8e8> (21) - *A* tests/data/transcripts/transcript_03ad6444c34e4affa5b978027bab7680.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-cafe8c294c99982fd6b24896c0deb7bfa1fc3bfa48786ebb64d597810efe801a> (21) - *A* tests/data/transcripts/transcript_1608ae49f650424395ecd6831007b644.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-d9248541259ea929ed63095bab6b67b1e6dcf57fa3f752b4ef4f71bcadf65161> (21) - *A* tests/data/transcripts/transcript_1df0a81afd5c41a89d03d6ad4d8fee8d.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-be034c8056d5f1403698caa5f9cc7c92fc9e3b513a1edf64fbe71829c8e3aa27> (23) - *A* tests/data/transcripts/transcript_356a5a0f22304265821b09aa05c644e3.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-aadafab05442ea3157583b9e9b794f788cda4cf39c6ae7b50409a0340ad6bf64> (33) - *A* tests/data/transcripts/transcript_37e810bcbcf44cb7958ab5476fd002ab.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-bd1d2fd5132c103471af8932d9185b8c75921a4329865e7abb25772c12a9e47f> (31) - *A* tests/data/transcripts/transcript_39568ce7ea7e42dc96db04db5ec6b6dd.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-7772cc61a1c542c2e3830c8b087f99fade07c5e4970bc32a9526ae216047e39f> (23) - *A* tests/data/transcripts/transcript_3b6e0eaf73eb4ed1a2d19eaefb60c95c.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-ce6751bc771470a6dcb4adbc9922e90c0ddc9c817a574c823a15f3746cccf3a0> (21) - *A* tests/data/transcripts/transcript_45a49bf0e00041c385b70230989dd018.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-3b8def126840133b32ffa6de8b70b2da99d7fdc88c72d296602203cdc8bcb40c> (81) - *A* tests/data/transcripts/transcript_49216ccf72694bb1842b533f886c90b1.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-c486dda54b18f7f92fce1ae034716d2b84e8c3497236d5f1cc51fe0bcf9289c6> (21) - *A* tests/data/transcripts/transcript_51dcd5082d71473d98dce5e75154b441.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-097c38d409cec4fff8bdd6c5a540494332995b336cbd4e2ee6f3491d44890381> (25) - *A* tests/data/transcripts/transcript_5f3360f6d26546c0b0545ec9dd377f32.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-1a1dee857b983ab9f3b65d97108bfdb25acc6aa0c17d0de4c82dbc2187f23965> (31) - *A* tests/data/transcripts/transcript_617a3145307a4fffb76ac59e579b3ff3.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-f317a2076e5958027d3f41d14ad29ef6285623793c656e876037257f2dfbf674> (29) - *A* tests/data/transcripts/transcript_61b91191746147879ffda2bf5d2ba001.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-f80fdc387581f5df13fd1cf6f5a0a512584b694ef4af92a573d3723d113dc663> (23) - *A* tests/data/transcripts/transcript_68b734ca04344eac9f086faa31e19c61.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-366fbeadb806475deed3641e4547eeb5d8da2ef1c4ff4add1d38316622da767e> (33) - *A* tests/data/transcripts/transcript_6b4d3d7b15fc4e8480f8879593bfb3e7.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-0a706e37da81188bdbfad19622b2f86baff944093f66061ac7d4f8f5863da44a> (73) - *A* tests/data/transcripts/transcript_6dbe0d9694934c329b66d3dc0e7128cc.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-11f53df317759868c65610bfac29b53011c1fc28b5534098aa3c433999b6c1d3> (57) - *A* tests/data/transcripts/transcript_7d12af26d77c4f299d614112ec81ffca.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-c7a390c26e084afbd16c6d94d60766ef429b5227aa4590317823a83cf1aba65b> (71) - *A* tests/data/transcripts/transcript_803d5a55b6ba49a08dfa982029c36233.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-6756b6f36283bf4b32dae8d974fd6b7e42f095f40ecccd05694b7843af1467fa> (37) - *A* tests/data/transcripts/transcript_81d7f3ec39e54b8ebadeeb41d069b24a.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-67c451c4c1143e0f918196edc93302b6300be74d70df3cf58e738ed37141631b> (49) - *A* tests/data/transcripts/transcript_88546a623fb54a2693d1824cb4b58122.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-cf254b38cec3e11c167df9659d46f1caebfc3b47d9aa40d43b91568c5d7e9ea1> (23) - *A* tests/data/transcripts/transcript_8966e646ef904a2798a399787f820c37.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-653d6d830af7ebe6bc42c5ed56f404854af5a1ce15ee69f929d5b808bf1dc2c6> (33) - *A* tests/data/transcripts/transcript_8b74a02cca9c4057a166a5ec8bde42a3.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-b7d369c1abe694246da9695f521bdbe76bceb511ed52f5f2ceeb7a62744cea1e> (19) - *A* tests/data/transcripts/transcript_913ae2244a7e49a1921f3419f03933a8.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-a6878bbfd39a43cbc40d8913c2c0c87ed47502d990df9eca108facbcabcb5a34> (21) - *A* tests/data/transcripts/transcript_aa5b15b7ee9a462f8017630d6ee84345.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-59ca8b9a8a8b33f8b132bd1df1e4e3da65b2c266d4189ec0c48c84a2ec2f06dc> (29) - *A* tests/data/transcripts/transcript_ac70a015bf024ca5ab4f664754e04641.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-23a8867be6d88ee08fe74b7d6b1ea01dc72569e25f472cca5c9473b6ac5ecb53> (15) - *A* tests/data/transcripts/transcript_b275fe0584384a17bef5a0065c949180.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-d0dd525e3f1f667bb6d5dcf195911e8c094d1351e6aa22bee21bbcd5d4719a70> (17) - *A* tests/data/transcripts/transcript_b57d3af997824b379a814f0b86e460d6.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-37b2a6c53b742f13e304dd865b0a73430ae52a8d432832e2897be15c1a422aec> (21) - *A* tests/data/transcripts/transcript_b9ff91d383874d37bca67565baf0d702.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-4eb4e39dbf30d69c722b630903afb956d15c73a9f2309c8f49d4a88b53251a05> (25) - *A* tests/data/transcripts/transcript_bcf1eeef974546cd88cd04338369279e.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-d0fa0a8a993d1c4403e67489dee292d23668b060e029960410756a049795f764> (21) - *A* tests/data/transcripts/transcript_c2a29b3c8bb34431b8ec53ed268c01ff.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-e2a950ab00e0653486c2395ffb8b16fbbb82a44ba6aa2f873b64b12eaa2f0cbd> (45) - *A* tests/data/transcripts/transcript_c5e990ac634d4a828af5aee60106a805.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-b0742d95d2c3cae66001055a3d2b8992ba9be0f388560f549464568ec28e3057> (21) - *A* tests/data/transcripts/transcript_c6e8844661954a78aed61f8ad0441400.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-b22cc0e7a67aca5b35fa8a4658dbf507432934be4f4a6b6c590f16e8db6ff48b> (21) - *A* tests/data/transcripts/transcript_d5a3346df71143e5a5cd01b4c808b512.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-233e45aafd9962ab7b57cc5fd598a98a8c478cf02f0958eb43a70763ac004398> (21) - *A* tests/data/transcripts/transcript_db4b38ff07784680805db11c041b28dd.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-c5cf2527c79b44636cd9303435c55cddae874272ba006d95f73ddc5c50e22957> (25) - *A* tests/data/transcripts/transcript_e2548f4267194ccdac2081727fed1abc.txt <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-a4cc927d0de0190aef2724c175618dcaefa00da61b3a0b41b2e8bca37dbace4e> (41) - *M* tests/test_content_parser.py <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-11cec7e50b2cefadd491384ab4799405c3d0cb8917be475f0f8b38006943ebcd> (29) - *A* tests/test_markdown_extractor.py <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-3f28409a287559cb4f03a2d71ea54e9e2b823d7c4011bfd4e1b04ccf3697fb6b> (68) - *M* usage/cli.md <https://github.com/souzatharsis/podcastfy/pull/93/files#diff-38f28c521d68550896b47c281e268bfaba5064e66853f6268f143ef6ddc7199e> (18) Patch Links: - https://github.com/souzatharsis/podcastfy/pull/93.patch - https://github.com/souzatharsis/podcastfy/pull/93.diff — Reply to this email directly, view it on GitHub <#93>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTMY3JSNFOLUELDA3EBKFTZ4TEPHAVCNFSM6AAAAABQJYBBRKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYYDCOJXGQZDMMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[tkhongsap]

Removed audio transcript files from tests.

[souzatharsis]

Great! I'll merge it as soon as I'm back at keyboard later this week.

…

On Mon, Oct 21, 2024, 7:28 PM tkhongsap ***@***.***> wrote: Removed audio transcript files from tests. — Reply to this email directly, view it on GitHub <#93 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTMY3KDQYTK3PSF7SOEGLTZ4VWYRAVCNFSM6AAAAABQJYBBRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRXG42TOMJTGY> . You are receiving this because you commented.Message ID: ***@***.***>

[souzatharsis]

Tests on github don't pass because you forgot to add some mock data. At least the following:

tests/data/mock/website.md and
tests/data/mock/file.txt

[tkhongsap]

@souzatharsis Apologies for the delay, I was out of town. I’ve updated the tests as requested and added the necessary mock data files. All tests passed successfully on my side with pytest.

Here’s a summary of the test results:

tests\test_audio.py .ss [ 9%] tests\test_client.py ......s.s.. [ 42%] tests\test_content_parser.py ....s [ 57%] tests\test_genai_podcast.py .... [ 69%] tests\test_generate_podcast.py ....... [ 90%] tests\test_markdown_extractor.py ... [100%]

[souzatharsis]

Hi Pytest tests failed.

ModuleNotFoundError: No module named 'markdown'

Could you please review?

[tkhongsap]

@souzatharsis Hi there,
below are my updates:

• Updated requirements.txt to include Markdown==3.7 for Python versions between 3.11 and 4.0.
• Verified all tests passed locally with poetry run pytest.

[souzatharsis]

Hi, I recommend instead integration with docling instead of building our own content parsers from scratch:

https://github.com/DS4SD/docling

Improved support, maintainability as well as coverage of courses since docling supports (PDF, DOCX, PPTX, Images, HTML, AsciiDoc, Markdown)

There is an open issue for that if you are excited about it #138