PDF2HTML CONVERSION- IPCC LONGER REPORT

[Gayathrijonnalagadda]

in test_make_ipcc_HTML

added IPCC longer report to the list of the chapters
tests run successfully
output file : tidied.HTML
but failed to keep the style in @gayathri's system, @petermr's looks fine.

TO DO

write a test to retain styles

[sunagparasu]

Downloaded and tried to run pyami on linux. These were the errors presented:

Import "lxml" could not be resolved Pylance (reportMissingImports) [Ln 10, Col 8]

Import "lxml.etree" could not be resolved Pylance(reportMissingImports) [Ln 11, Col 8]

Import "lxml.html" could not be resolved Pylance(reportMissingImports) [Ln 12, Col 8]

Import "test.test_all" could not be resolved Pylance (reportMissingImports) [Ln 14, Col 8]

Import "pdfplumber" could not be resolved Pylance(reportMissingImports) [Ln 17, Col 8]

Import "test.resources" could not be resolved Pylance(reportMissingImports) [Ln 30, Col 6]

Import "test.test_all" could not be resolved Pylance (reportMissingImports) [Ln 33, Col 6]

Import "pdfminer.high_level" could not be resolved Pylance(reportMissingImports) [Ln 743, Col 14]

Import "pdfminer" could not be resolved Pylance(reportMissingImports) [Ln 805, Col 16]

Import "pdfminer.image" could not be resolved Pylance(reportMissingImports) [Ln 806, Col 14]

Import "pdfminer.high_level" could not be resolved Pylance (reportMissingImports) [Ln 807, Col 14]

Import "pdfminer.high_level" could not be resolved Pylance(reportMissingImports) [Ln 846, Col 14]

[petermr]

Thanks

On Fri, Mar 24, 2023 at 11:27 AM Gayathrijonnalagadda < ***@***.***> wrote: in test_make_ipcc_HTML added IPCC longer report to the list of the chapters tests run successfully output file : tidied.HTML *but failed to keep the style in @gayathri <https://github.com/gayathri>'s system, @petermr <https://github.com/petermr>'s looks fine.*

Can you output a file which fails to show the style. It's just possible it's your browser environment

TO DO write a test to retain styles — Reply to this email directly, view it on GitHub <#38>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFTCS3XZ7DVSDTVF22GL6DW5WALBANCNFSM6AAAAAAWGN4TCQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

I will post more , integrated, code over the weekend. This may not solve *your* problem. We will use the community to find out whether others have the problem.

…

-- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK

[petermr]

try pip uninstall lxml pip install lxml and the same for other non-PMR libraries. P.

…

On Fri, Mar 24, 2023 at 1:05 PM sunagparasu ***@***.***> wrote: Downloaded and tried to run pyami on linux. These were the errors presented: Import "lxml" could not be resolved Pylance (reportMissingImports) [Ln 10, Col 8] Import "lxml.etree" could not be resolved Pylance(reportMissingImports) [Ln 11, Col 8] Import "lxml.html" could not be resolved Pylance(reportMissingImports) [Ln 12, Col 8] Import "test.test_all" could not be resolved Pylance (reportMissingImports) [Ln 14, Col 8] Import "pdfplumber" could not be resolved Pylance(reportMissingImports) [Ln 17, Col 8] Import "test.resources" could not be resolved Pylance(reportMissingImports) [Ln 30, Col 6] Import "test.test_all" could not be resolved Pylance (reportMissingImports) [Ln 33, Col 6] Import "pdfminer.high_level" could not be resolved Pylance(reportMissingImports) [Ln 743, Col 14] Import "pdfminer" could not be resolved Pylance(reportMissingImports) [Ln 805, Col 16] Import "pdfminer.image" could not be resolved Pylance(reportMissingImports) [Ln 806, Col 14] Import "pdfminer.high_level" could not be resolved Pylance (reportMissingImports) [Ln 807, Col 14] Import "pdfminer.high_level" could not be resolved Pylance(reportMissingImports) [Ln 846, Col 14] — Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFTCSZVHTPHH35PBPMHGWDW5WL3BANCNFSM6AAAAAAWGN4TCQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK

[sunagparasu]

Attempt #2 to run py4ami

• Opened the terminal and switched to a virtual environment

• Entered py4ami PDF --makehtml --infile chapter03.pdf --outdir /home/anything/Downloads/ into the terminal and received the following error:

usage: py4ami [-h] [-v] {DICT,GUI,HTML,PDF,PROJECT} ...

py4ami: error: unrecognized arguments: --makehtml

Traceback (most recent call last):

  File "/home/anything/anaconda3/envs/semanticClimate/bin/py4ami", line 8, in <module>
    sys.exit(main())

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/pyamix.py", line 1263, in main
    pyamix.run_command(sys.argv[1:])

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/pyamix.py", line 343, in run_command
    self.parse_and_run_args(args)

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/pyamix.py", line 358, in parse_and_run_args
    self.args = self.make_substitutions_create_arg_tuples(arglist, parser)

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/pyamix.py", line 542, in make_substitutions_create_arg_tuples

    raise ValueError(f"bad command arguments {parsed_args} (see log output)")

ValueError: bad command arguments SystemExitFail_2 (see log output)

• Cloned the semanticClimate repository within pyami and ran the argument python3 -m py4ami.ami_pdf --inpath /home/anything/semanticClimate/ipcc/wg3/Chapter03/fulltext.pdf --outdir /home/anything/semanticClimate/ipcc/wg3/Chapter03/ and recieved the following errors:

running PDFArgs main

module_stem ami_pdf

 sys.argv ['/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/ami_pdf.py', '--inpath', '/home/anything/semanticClimate/ipcc/wg3/Chapter03/fulltext.pdf', '--outdir', '/home/anything/semanticClimate/ipcc/wg3/Chapter03/']

WARNING:py4ami.util:AbstractArgs ADDED ARGS ['/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/ami_pdf.py', '--inpath', '/home/anything/semanticClimate/ipcc/wg3/Chapter03/fulltext.pdf', '--outdir', '/home/anything/semanticClimate/ipcc/wg3/Chapter03/']

WARNING:root:sys.argv ['/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/ami_pdf.py', '--inpath', '/home/anything/semanticClimate/ipcc/wg3/Chapter03/fulltext.pdf', '--outdir', '/home/anything/semanticClimate/ipcc/wg3/Chapter03/']
argv: ['--inpath', '/home/anything/semanticClimate/ipcc/wg3/Chapter03/fulltext.pdf', '--outdir', '/home/anything/semanticClimate/ipcc/wg3/Chapter03/']

WARNING:root:**** raw pages: 1_9999999

Traceback (most recent call last):
  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/ami_pdf.py", line 1872, in parse_and_process_1
    pdf_args.parse_and_process()

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/util.py", line 387, in parse_and_process
    self.parse_and_process1(argv_)

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/util.py", line 394, in parse_and_process1
    self.process_args()

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/ami_pdf.py", line 886, in process_args
    if not self.check_input():

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/ami_pdf.py", line 912, in check_input
    raise FileNotFoundError(f"input file/path does not exist: ({self.inpath}")

FileNotFoundError: input file/path does not exist: (/home/anything/semanticClimate/ipcc/wg3/Chapter03/fulltext.pdf

***Cannot run pyami***; see output for errors: input file/path does not exist: (/home/anything/semanticClimate/ipcc/wg3/Chapter03/fulltext.pdf

• Unable to identify the solution, tried checking if test_pdf.py ran smoothly on visual studio code and came across this error:

Import "test.test_all" could not be resolved Pylance (reportMissingImports) [Ln 14, Col 8]

Import "test.resources" could not be resolved Pylance(reportMissingImports) [Ln 30, Col 6]

Import "test.test_all" could not be resolved Pylance (reportMissingImports) [Ln 33, Col 6]

• I believe each of these errors will need to be fixed separately, and I'd be happy to work on it.

[petermr]

A. Run

py4ami PDF --help

Is there a --makehtml in the options?

If there isn't and you entered it, this is what caused the error. I think it converts automatically (shouldn't really do this implicitly). Try without the --makehtml.

This is what we will address on Thursday

B. The repo has
https://github.com/petermr/semanticClimate/blob/main/ipcc/ar6/wg3/Chapter03/fulltext.pdf
you have checked it out it so it should run.

I don't understand this - we can debug tomorrow.

[sunagparasu]

• There isn't a --makehtml in the options, However that's included in the example given when you run py4ami -h or py4ami --help . Here's what I get when I run py4ami -h :

usage: py4ami [-h] [-v] {DICT,GUI,HTML,PDF,PROJECT} ...

Py4AMI: create, manipulate, use CProject 
----------------------------------------

Py4AMI is the largest collection of functionality in the AMI system.It contains executable code and libraries to manage complex documents.
A key structure is the corpus (CProject directory) which contains a list of subdirectories (CTrees) which themselves contain many different document features (text, tables, images, graphics.
Py4AMI can create, fill, manipulate, transform many of the components including PDF, HTML, TXT, images, CSV.

The subcommands:

  DICT <options>      # create/edit/search dictionaries
  GUI <options>       # run tkinter GUI (prototype)
  HTML <options>      # create/edit HTML
  PDF <options>       # convert PDF into HTML and images
  PROJECT <options>   # create and transform a corpus of documents

After installation, run 
  py4ami <subcommand> <options>

Examples (# foo is a comment):
  py4ami        # runs help
  py4ami -h     # runs help
  py4ami PDF -h # runs PDF help
  py4ami PDF --makehtml --infile foo.pdf --outdir bar/ # converts PDF to HTML
  py4ami PROJECT --project foodir/ # converts all PDF in foodir to CTrees

----------------------------------------

positional arguments:
  {DICT,GUI,HTML,PDF,PROJECT}
                        subcommands

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show version 0.0.45

run:
        py4ami <subcommand> <args>
          where subcommand is in   {DICT,GUI,HTML,PDF,PROJECT} and args depend on subcommand

• However, there is an option within the subcommand PDF named pdf2html which should convert pdfs to the html format. Upon your suggestion, I selected fulltext.pdf in the semanticClimate repository as the input file. Running py4ami PDF --pdf2html --infile /home/anything/semanticClimate/ipcc/ar6/wg3/Chapter03/fulltext.pdf --outdir /home/anything/Downloads/ gave the following error:

WARNING:symbol.ini: environment variable $PYAMI_HOME not set, defaulting to bundled config.ini in /home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami

usage: py4ami PDF [-h]
                  [--debug {words,lines,rects,curves,images,tables,hyperlinks,annots}]
                  [--flow FLOW] [--footer FOOTER] [--header HEADER]
                  [--imagedir IMAGEDIR] [--indir INDIR]
                  [--inform INFORM [INFORM ...]] [--inpath INPATH]
                  [--infile INFILE] [--instem INSTEM] [--maxpage MAXPAGE]
                  [--offset OFFSET] [--outdir OUTDIR] [--outpath OUTPATH]
                  [--outstem OUTSTEM] [--outform OUTFORM]
                  [--pdf2html {pdfminer,pdfplumber}]
                  [--pages PAGES [PAGES ...]] [--resolution RESOLUTION]
                  [--template TEMPLATE]

py4ami PDF: error: argument --pdf2html: expected one argument

Traceback (most recent call last):
  File "/home/anything/anaconda3/envs/semanticClimate/bin/py4ami", line 8, in <module>
    sys.exit(main())

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/pyamix.py", line 1263, in main
    pyamix.run_command(sys.argv[1:])

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/pyamix.py", line 343, in run_command
    self.parse_and_run_args(args)

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/pyamix.py", line 358, in parse_and_run_args
    self.args = self.make_substitutions_create_arg_tuples(arglist, parser)

  File "/home/anything/anaconda3/envs/semanticClimate/lib/python3.8/site-packages/py4ami/pyamix.py", line 542, in make_substitutions_create_arg_tuples

    raise ValueError(f"bad command arguments {parsed_args} (see log output)")

ValueError: bad command arguments SystemExitFail_2 (see log output)

• My guess is that these are documentation issues as you're recalling that there isn't an explicit option for converting pdf to html but the documentation for py4ami mentions a pdf2html argument (running py4ami PDF -h mentions a --pdf2html argument as well). I hope we can resolve these issues tomorrow.