Added LookUp error handling during encoding detection.

[ggcr]

Please check Issue #411

I go into details on how to reproduce and the reasoning behind my simple fix.

---

In the current implementation of parse_file, the exception handling only catches UnicodeDecodeError.

NeMo-Curator/tutorials/dapt-curation/code/docbuilder.py

Lines 275 to 288 in 7272ca0

	# Open the file and read its content. Determine the encoding using cchardet. Skip over binary files.
	with zip_ref.open(file_info, "r") as file:
	content = file.read()
	# Determine the encoding of the file
	encoding = chardet.detect(content)["encoding"]
	if not encoding:
	return None
	try:
	content = content.decode(encoding)
	except UnicodeDecodeError:
	# If the file cannot be decoded, return None
	return None

This can be updated to also catch LookupError.

except (UnicodeDecodeError, LookupError):
    return None

While this works, it will still trigger an error when prompted with an encoding not available in the runtime system. Would be very nice to parse line by line instead, this way we would only skip the line and not ditch the whole github repo from the curation process. However, parsing the file line by line might introduce a lot of overhead for big repos.