Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix whitespace typo for Bert Question Answerer test in README.md #967

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ghostfly23333
Copy link

In the Bert QA testcase provided by README.md, I think the context is seperated by different types of whitespace. I also print the results of the subwords of tokens. It shows these.

context: The Amazon rainforest, alternatively, the Amazon Jungle, also known in English as Amazonia, is a moist broadleaf tropical rainforest in the Amazon biome that covers most of the Amazon basin of South America. This basin encompasses 7,000,000 km2 (2,700,000 sq mi), of which 5,500,000 km2 (2,100,000 sq mi) are covered by the rainforest. This region includes territory belonging to nine nations.

orig_tokens: {The Amazon}, {rainforest,}, {alternatively,}, {the Amazon}, {Jungle,}, {also}, {known}, {in}, {English}, {as Amazonia,}, {is}, {a moist}, {broadleaf tropical rainforest in}, {the Amazon}, {biome that}, {covers}, {most}, {of}, {the Amazon}, {basin of}, {South}, {America.}, {This}, {basin}, {encompasses}, {7,000,000 km2 (2,700,000 sq mi),}, {of}, {which}, {5,500,000 km2 (2,100,000 sq mi)}, {are}, {covered}, {by}, {the}, {rainforest.}, {This}, {region}, {includes}, {territory}, {belonging}, {to}, {nine}, {nations.}

So the tokens are not correctly been seperated by the absl::StrSplit() in

orig_tokens_ = absl::StrSplit(context, absl::ByChar(' '), absl::SkipEmpty());

I check the hex values of the origin context string it shows typos about whitespace in the context and I patch the issue.

@ghostfly23333
Copy link
Author

Hi @lu-wang-g

Asking here for code review as I'm not allowed to assign you as reviewer in the reviewers sidebar.

Also if the example is ought to be the origin version, feel free to comment below!
Have a good day!
ghostfly23333

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant