Skip to content
This repository has been archived by the owner on Jan 15, 2024. It is now read-only.

Commit

Permalink
[DOC] add int8 command (#1174)
Browse files Browse the repository at this point in the history
  • Loading branch information
xinyu-intel authored Feb 25, 2020
1 parent 50e5278 commit 27331cb
Showing 1 changed file with 12 additions and 12 deletions.
24 changes: 12 additions & 12 deletions scripts/bert/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -223,22 +223,22 @@ version of `mxnet-mkl <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/inde
Sentence Classification
+++++++++++++++++++++++

+-----------+-------------------+---------------+---------------+---------+---------+-----+---------+
| Dataset | Model | FP32 Accuracy | INT8 Accuracy | FP32 F1 | INT8 F1 | Log | Command |
+===========+===================+===============+===============+=========+=========+=====+=========+
| MRPC | bert_12_768_12 | 87.01 | 87.01 | 90.97 | 90.88 | | |
+-----------+-------------------+---------------+---------------+---------+---------+-----+---------+
| SST-2 | bert_12_768_12 | 93.23 | 93.00 | | | | |
+-----------+-------------------+---------------+---------------+---------+---------+-----+---------+
+-----------+-------------------+---------------+---------------+---------+---------+------------------------------------------------------------------------------------------------------------------------+
| Dataset | Model | FP32 Accuracy | INT8 Accuracy | FP32 F1 | INT8 F1 | Command |
+===========+===================+===============+===============+=========+=========+========================================================================================================================+
| MRPC | bert_12_768_12 | 87.01 | 87.01 | 90.97 | 90.88 |`command <https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/bert/calibration_MRPC_base_mx1.6.0b20200125.sh>`__ |
+-----------+-------------------+---------------+---------------+---------+---------+------------------------------------------------------------------------------------------------------------------------+
| SST-2 | bert_12_768_12 | 93.23 | 93.00 | | |`command <https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/bert/calibration_SST_base_mx1.6.0b20200125.sh>`__ |
+-----------+-------------------+---------------+---------------+---------+---------+------------------------------------------------------------------------------------------------------------------------+

Question Answering
++++++++++++++++++

+-----------+-------------------+---------+---------+---------+---------+-----+---------+
| Dataset | Model | FP32 EM | INT8 EM | FP32 F1 | INT8 F1 | Log | Command |
+===========+===================+=========+=========+=========+=========+=====+=========+
| SQuAD 1.1 | bert_12_768_12 | 81.18 | 80.32 | 88.58 | 88.10 | | |
+-----------+-------------------+---------+---------+---------+---------+-----+---------+
+-----------+-------------------+---------+---------+---------+---------+----------------------------------------------------------------------------------------------------------------------------+
| Dataset | Model | FP32 EM | INT8 EM | FP32 F1 | INT8 F1 | Command |
+===========+===================+=========+=========+=========+=========+============================================================================================================================+
| SQuAD 1.1 | bert_12_768_12 | 81.18 | 80.32 | 88.58 | 88.10 |`command <https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/bert/calibration_squad1.1_base_mx1.6.0b20200125.sh>`__ |
+-----------+-------------------+---------+---------+---------+---------+----------------------------------------------------------------------------------------------------------------------------+

For all model settings above, we use a subset of evaluation dataset for calibration.

Expand Down

0 comments on commit 27331cb

Please sign in to comment.