Text highlighting using char_span

[DWHowes]

I am loading one or more text files (as specified by flist) into a list (data_list). Each entry in the list is a dictionary composed of a spacy doc object and a currently empty Label list.

File Access

    def merge_data_files(self, flist:list):
        nlp = sp.blank("en")
        for file in flist:
            with open(file, 'r', encoding='utf-8') as f:
                while line := f.readline():
                    self.data_list.append({'Doc': nlp(line.rstrip()), 'Labels': []})

The list data_list is initialized as an empty list in the init method of the class.

In my app, I can click through each entry in the list, displaying the text of the entry in my edit window (a QTextEdit widget).

In the displayed text, I wish to partially highlight a span of tokens and have the highlighted text snap to the beginning of the first partially selected token and the end of the last one. This is the code I'm using

    def lftBtnRel(self):
        cursor = self.caseWindow.textCursor()
        if cursor.hasSelection():
            span = self.doc.char_span(cursor.selectionStart(), 
                                      cursor.selectionEnd(), 
                                      alignment_mode="expand")
            print("span = %s, start position = %s, end position = %s" % (span.text, span.start_char, span.end_char))
            cursor.setPosition(span.start_char, QTextCursor.MoveMode.MoveAnchor)
            cursor.movePosition(QTextCursor.MoveOperation.NextCharacter, 
                                QTextCursor.MoveMode.KeepAnchor, 
                                span.end_char - span.start_char)
            self.caseWindow.setTextCursor(cursor)

This code is called from my event filter, when I catch the left mouse button release.

The highlighting is managed correctly on every line of the file except the first one. For the first line, there is an offset in the start_char returned from char_span. This behaviour manifests for the first line of each file that has been merged into data_list.

For example, using the following text (first line from the file I've attached).

In the first part of this book, and particularly in Chapters 4–7, I have presented the basis of this book’s argument by presenting some of the salient characteristics of my approach, such as taking a long-term ex-ante perspective, learning from the past about the present for the future, using complex systems thinking, etc.

• If I highlight the 't' in 'the' print returns : span = , start position = 4, end position = 4 [nothing is highlighted]
• If I highlight the 'h' in 'the' print returns: span = the, start position = 4, end position = 7 [highlighted text is 'he ']
• If I highlight 'e f' (so it should highlight the span 'the first') print returns: span = the, start position = 4, end position = 7 [highligthed text is 'he ']

I've eliminated a file read error, as I've read in the file two different ways (using readline and a csv reader) with the same results. If there is a non-displaying character starting the text file, I haven't been able to find it (I've attached one of the texts files)

If I'm not handling the edge case correctly, I'm not sure what I'm doing wrong. Any help or ideas will be greatly appreciated.

an_outline_of_human_socioenvironmental_coevolution.txt

[DWHowes]

I've figured out the problem.

When the text files were originally saved, they were encoded as UTF-8-BOM. Changing the encoding to UTF-8 fixed the problem I describe above.