Image conversion not parsing correctly

[mailtoshwetha09]

I have the image which is in pdf. I have exported images and trying to convert to json or markdown, where it is not extracting the content of the image correct.

Convert the current PNG file

            conv_results = doc_converter.convert_all([Path(png_path)])  # Ensure png_path is wrapped in a list

            for res in conv_results:
                out_path = Path("scratch")
                lgmsg.logmessage(lgmsg.info, f"Document {res.input.file.name} converted.")

                # Export the document to different formats
                md_file_path = out_path / f"{res.input.file.stem}.md"
                json_file_path = out_path / f"{res.input.file.stem}.json"
                yaml_file_path = out_path / f"{res.input.file.stem}.yaml"

                with md_file_path.open("w") as fp:
                    fp.write(res.document.export_to_markdown())

                with json_file_path.open("w") as fp:
                    fp.write(json.dumps(res.document.export_to_dict()))

                with yaml_file_path.open("w") as fp:
                    fp.write(yaml.safe_dump(res.document.export_to_dict()))

                # Log the saved output paths
                lgmsg.logmessage(lgmsg.info, f"Saved markdown output to: {md_file_path}")
                lgmsg.logmessage(lgmsg.info, f"Saved JSON output to: {json_file_path}")
                lgmsg.logmessage(lgmsg.info, f"Saved YAML output to: {yaml_file_path}")

sample json, here text values are encoded where i was expecting the same text as in png and i have to call LLM for translating the content

{
"bbox": {
"l": 590.3333129882812,
"t": 114.33333587646484,
"r": 659.6666870117188,
"b": 96.66666412353516,
"coord_origin": "BOTTOMLEFT"
},
"row_span": 1,
"col_span": 1,
"start_row_offset_idx": 5,
"end_row_offset_idx": 6,
"start_col_offset_idx": 3,
"end_col_offset_idx": 4,
"text": "3,021 @A",
"column_header": false,
"row_header": false,
"row_section": false
},{
"bbox": {
"l": 469.6666564941406,
"t": 94.66666412353516,
"r": 536.6666870117188,
"b": 77,
"coord_origin": "BOTTOMLEFT"
},
"row_span": 1,
"col_span": 1,
"start_row_offset_idx": 6,
"end_row_offset_idx": 7,
"start_col_offset_idx": 2,
"end_col_offset_idx": 3,
"text": "5,564 {8A",
"column_header": false,
"row_header": false,
"row_section": false
}, {
"bbox": {
"l": 350.3333435058594,
"t": 94,
"r": 416.6666564941406,
"b": 77,
"coord_origin": "BOTTOMLEFT"
},
"row_span": 1,
"col_span": 1,
"start_row_offset_idx": 6,
"end_row_offset_idx": 7,
"start_col_offset_idx": 1,
"end_col_offset_idx": 2,
"text": "4,638 '8A",
"column_header": false,
"row_header": false,
"row_section": false
},

[dolfim-ibm]

Bitmap figures are currently not processed, i.e. the output will contain their location (page, bbox, etc) and, possibly, their caption.

We are working on new features which would

1. return the OCR text in the figure (no interpretation yet)
2. use a visual llm to describe the figure. see Integrate image understanding pipeline #192
3. use a dedicated small model for charts understanding (in development)