-
Notifications
You must be signed in to change notification settings - Fork 2
Post_Processing
KwonTaeYang edited this page May 24, 2021
·
1 revision
- F1 score와 EM의 차이가 많이 남
- answer의 결과를 볼 때 필요없는 조사가 존재
- ex)
![](https://github.com/TEAM-IKYO/Open-Domain-Question-Answering/raw/main/etc/img/post_processing_img1.png)
from konlpy.tag import Mecab, Kkma, Hannanum
def last_preprocessing():
mecab = Mecab()
kkma = Kkma()
hannanum = Hannanum()
with open("../outputs/test_dataset/predictions.json", encoding="utf-8") as json_file:
json_data = json.load(json_file)
for k, v in json_data.items():
if mecab.pos(v)[-1][-1] in {"JX", "JKB", "JKO", "JKS", "ETM", "VCP", "JC"}:
json_data[k] = json_data[k][:-len(mecab.pos(v)[-1][0])]
elif v[-1] == "의":
# 지울 수 있는 `의` 인지 check
if kkma.pos(v)[-1][-1] == "JKG" or mecab.pos(v)[-1][-1] == "NNG" or hannanum.pos(v)[-1][-1] == "J":
json_data[k] = json_data[k][:-1]
with open("../outputs/test_dataset/proprecessing_predictions.json", "w", encoding="utf-8") as make_file:
json.dump(json_data, make_file, indent="\t", ensure_ascii=False)
- 필요 없는 조사 제거 완료
- F1 score와 EM의 차이가 줄어듦
![](https://github.com/TEAM-IKYO/Open-Domain-Question-Answering/raw/main/etc/img/post_processing_img2.png)