Models fail to load

[peter-ciccolo]

I'm getting the following error in the final step of the quickstart:

unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file

Input line:
./main -m models/bigcode/gpt_bigcode-santacoder-ggml.bin -p "def fibonnaci(" --top_k 0 --top_p 0.95 --temp 0.2

Output:

main: seed = 1687068338
starcoder_model_load: loading model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'
starcoder_model_load: n_vocab = 49280
starcoder_model_load: n_ctx   = 2048
starcoder_model_load: n_embd  = 2048
starcoder_model_load: n_head  = 16
starcoder_model_load: n_layer = 24
starcoder_model_load: ftype   = 1003
starcoder_model_load: qntvr   = 1
starcoder_model_load: ggml ctx size = 1794.97 MB
starcoder_model_load: memory size =   768.00 MB, n_mem = 49152
starcoder_model_load: unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file
main: failed to load model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'

Notable differences from the sample output:

• starcoder_model_load: ftype = 1 in my output vs starcoder_model_load: ftype = 3
  (quanitzed models were produced with ./quantize models/bigcode/gpt_bigcode-santacoder-ggml.bin models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin 3; non-quanitzed model fails with a similar error)
• starcoder_model_load: qntvr = 1 in my output vs. no info on qntvr in the sample output

Other notes:

• this is running on a 2019 Intel MBP, not an M1
• conda list is reproduced below in case I'm somehow missing a dependency

# Name                    Version                   Build  Channel
accelerate                0.20.3             pyhd8ed1ab_0    conda-forge
blas                      1.0                         mkl
brotlipy                  0.7.0           py310hca72f7f_1002
bzip2                     1.0.8                h1de35cc_0
ca-certificates           2023.5.7             h8857fd0_0    conda-forge
certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h6c40b1e_3
charset-normalizer        2.0.4              pyhd3eb1b0_0
click                     8.0.4           py310hecd8cb5_0
cryptography              39.0.1          py310hf6deb26_2
dataclasses               0.8                pyh6d0b6a4_7
filelock                  3.9.0           py310hecd8cb5_0
future                    0.18.3          py310hecd8cb5_0
huggingface_hub           0.15.1                     py_0    huggingface
idna                      3.4             py310hecd8cb5_0
importlib-metadata        6.0.0           py310hecd8cb5_0
importlib_metadata        6.0.0                hd3eb1b0_0
intel-openmp              2023.1.0         ha357a0b_43547
joblib                    1.2.0           py310hecd8cb5_0
libcxx                    14.0.6               h9765a3e_0
libffi                    3.4.4                hecd8cb5_0
libgfortran               5.0.0           11_3_0_hecd8cb5_28
libgfortran5              11.3.0              h9dfd629_28
libopenblas               0.3.21               h54e7dc3_0
libprotobuf               3.20.3               hfff2838_0
libuv                     1.44.2               h6c40b1e_0
llvm-openmp               14.0.6               h0dcd299_0
mkl                       2023.1.0         h59209a4_43558
mkl-service               2.4.0           py310h6c40b1e_1
mkl_fft                   1.3.6           py310h3ea8b11_1
mkl_random                1.2.2           py310h3ea8b11_1
ncurses                   6.4                  hcec6c5f_0
ninja                     1.10.2               hecd8cb5_5
ninja-base                1.10.2               haf03e11_5
numpy                     1.24.3          py310h827a554_1
numpy-base                1.24.3          py310ha186be2_1
openssl                   3.1.1                h8a1eda9_1    conda-forge
packaging                 23.0            py310hecd8cb5_0
pip                       23.1.2          py310hecd8cb5_0
protobuf                  3.20.3          py310hcec6c5f_0
psutil                    5.9.5           py310h90acd4f_0    conda-forge
pycparser                 2.21               pyhd3eb1b0_0
pyopenssl                 23.0.0          py310hecd8cb5_0
pysocks                   1.7.1           py310hecd8cb5_0
python                    3.10.11              h5ee71fb_3
python_abi                3.10                    2_cp310    conda-forge
pytorch                   1.13.1          cpu_py310h9e40b02_0
pyyaml                    6.0             py310h6c40b1e_1
readline                  8.2                  hca72f7f_0
regex                     2022.7.9        py310hca72f7f_0
requests                  2.29.0          py310hecd8cb5_0
sacremoses                master                     py_0    huggingface
setuptools                67.8.0          py310hecd8cb5_0
six                       1.16.0             pyhd3eb1b0_1
sqlite                    3.41.2               h6c40b1e_0
tbb                       2021.8.0             ha357a0b_0
tk                        8.6.12               h5d9f67b_0
tokenizers                0.11.4          py310h8776b5c_1
tqdm                      4.65.0          py310h20db666_0
transformers              4.28.1                     py_0    huggingface
typing-extensions         4.6.3           py310hecd8cb5_0
typing_extensions         4.6.3           py310hecd8cb5_0
tzdata                    2023c                h04d1e81_0
urllib3                   1.26.16         py310hecd8cb5_0
wheel                     0.38.4          py310hecd8cb5_0
xz                        5.4.2                h6c40b1e_0
yaml                      0.2.5                haf1e3a3_0
zipp                      3.11.0          py310hecd8cb5_0
zlib                      1.2.13               h4dc903c_0