We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[2024-08-24 15:53:49,278] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible Starting DeepSpeed-MII instance for model /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/... Deployment name: mixtral-8x7b-instruct-v0.1 [2024-08-24 15:53:57,845] [INFO] [server.py:38:__init__] Hostfile /job/hostfile not found, creating hostfile. [2024-08-24 15:53:57,845] [INFO] [server.py:38:__init__] Hostfile /job/hostfile not found, creating hostfile. [2024-08-24 15:53:57,846] [INFO] [server.py:110:_launch_server_process] msg_server launch: ['deepspeed', '-i', 'localhost:0,1,2,3,4,5,6,7', '--master_port', '29500', '--master_addr', 'localhost', '--no_ssh_check', '--no_local_rank', '--no_python', '/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:53:57,846] [INFO] [server.py:110:_launch_server_process] msg_server launch: ['deepspeed', '-i', 'localhost:0,1,2,3,4,5,6,7', '--master_port', '29500', '--master_addr', 'localhost', '--no_ssh_check', '--no_local_rank', '--no_python', '/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:53:57,847] [INFO] [server.py:110:_launch_server_process] msg_server launch: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--load-balancer', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:53:57,847] [INFO] [server.py:110:_launch_server_process] msg_server launch: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--load-balancer', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:53:59,468] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-24 15:53:59,916] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible [2024-08-24 15:54:02,848] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:02,848] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:03,585] [WARNING] [runner.py:202:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. [2024-08-24 15:54:03,586] [INFO] [runner.py:568:main] cmd = /databricks/python3/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNCwgNSwgNiwgN119 --master_addr=127.0.0.1 --master_port=29500 --no_python --no_local_rank --enable_each_rank_log=None /local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python -m mii.launch.multi_gpu_server --deployment-name mixtral-8x7b-instruct-v0.1 --load-balancer-port 50050 --restful-gateway-port 51080 --restful-gateway-host localhost --restful-gateway-procs 32 --server-port 50051 --zmq-port 25555 --model-config eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9 [2024-08-24 15:54:05,442] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible [2024-08-24 15:54:07,848] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:07,848] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... Starting load balancer on port: 50050 About to start server Started [2024-08-24 15:54:10,210] [INFO] [launch.py:139:main] 0 NCCL_SOCKET_IFNAME=eth [2024-08-24 15:54:10,210] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]} [2024-08-24 15:54:10,210] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=8, node_rank=0 [2024-08-24 15:54:10,210] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]}) [2024-08-24 15:54:10,210] [INFO] [launch.py:164:main] dist_world_size=8 [2024-08-24 15:54:10,211] [INFO] [launch.py:168:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 [2024-08-24 15:54:10,211] [INFO] [launch.py:256:main] process 44848 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,212] [INFO] [launch.py:256:main] process 44849 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,212] [INFO] [launch.py:256:main] process 44850 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,213] [INFO] [launch.py:256:main] process 44851 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,213] [INFO] [launch.py:256:main] process 44852 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,214] [INFO] [launch.py:256:main] process 44853 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,214] [INFO] [launch.py:256:main] process 44854 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJx *** WARNING: max output size exceeded, skipping output. *** FO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:53,284] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00007-of-00019.safetensors [2024-08-24 15:54:53,654] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00007-of-00019.safetensors [2024-08-24 15:54:54,172] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00007-of-00019.safetensors [2024-08-24 15:54:54,447] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:54,585] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:55,147] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:55,404] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:55,502] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00007-of-00019.safetensors [2024-08-24 15:54:55,877] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:56,042] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:56,885] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:57,371] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:57,440] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:54:57,590] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:57,861] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:57,861] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:57,871] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:58,092] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:58,705] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:58,819] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:59,656] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:55:00,017] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:00,287] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:00,711] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:00,918] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:55:01,250] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:01,304] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:02,212] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:02,861] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:02,861] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:03,469] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:04,154] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:07,007] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:07,144] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:07,407] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:07,862] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:07,862] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:08,656] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:08,806] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:08,812] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:09,105] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:10,825] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:12,263] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:12,275] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:12,487] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:12,863] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:12,863] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:13,154] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:13,960] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:14,207] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:15,573] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:15,870] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:17,863] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:17,863] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:19,297] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:19,557] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:19,850] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:20,426] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:21,166] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:21,210] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:22,465] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:22,864] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:22,864] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:23,444] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:26,525] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:26,546] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:26,714] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:26,953] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:27,865] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:27,865] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:28,504] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:28,522] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:29,286] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:29,777] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:32,866] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:32,866] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:33,819] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:34,124] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:34,626] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:34,664] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:34,888] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:35,127] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:35,146] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:37,568] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:37,866] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:37,866] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:38,367] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:39,525] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:39,867] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:39,948] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:40,296] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:40,866] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:42,328] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:42,867] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:42,867] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:47,868] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:47,868] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:51,288] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:52,868] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:52,868] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:54,629] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:57,733] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:57,869] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:57,869] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:58,901] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:58,957] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:59,381] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:59,516] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:56:01,163] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. Starting server on port: 50055 About to start server Starting server on port: 50054 Starting server on port: 50057 Starting server on port: 50058 Started Starting server on port: 50052 Starting server on port: 50056 About to start server About to start server About to start server Started Started Starting server on port: 50053 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR E0000 00:00:1724514961.707826 44855 chttp2_server.cc:1118] UNKNOWN:No address added out of total 1 resolved for '[::]:50058' {created_time:"2024-08-24T15:56:01.707823091+00:00", children:[UNKNOWN:Failed to add any wildcard listeners {created_time:"2024-08-24T15:56:01.707808181+00:00", children:[UNKNOWN:Unable to configure socket {fd:121, created_time:"2024-08-24T15:56:01.70776102+00:00", children:[UNKNOWN:bind: Address already in use (98) {created_time:"2024-08-24T15:56:01.707724179+00:00"}]}, UNKNOWN:Unable to configure socket {fd:121, created_time:"2024-08-24T15:56:01.707805861+00:00", children:[UNKNOWN:bind: Address already in use (98) {created_time:"2024-08-24T15:56:01.70780179+00:00"}]}]}]} Started [rank7]: Traceback (most recent call last): [rank7]: File "<frozen runpy>", line 198, in _run_module_as_main [rank7]: File "<frozen runpy>", line 88, in _run_code [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/launch/multi_gpu_server.py", line 105, in <module> [rank7]: main() [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/launch/multi_gpu_server.py", line 100, in main [rank7]: serve_inference(inference_pipeline, port) [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/grpc_related/modelresponse_server.py", line 291, in serve_inference [rank7]: _do_serve(ModelResponse(async_pipeline=async_pipeline), port) [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/grpc_related/modelresponse_server.py", line 281, in _do_serve [rank7]: server.add_insecure_port(f"[::]:{port}") [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/grpc/_server.py", line 1473, in add_insecure_port [rank7]: return _common.validate_port_binding_result( [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/grpc/_common.py", line 181, in validate_port_binding_result [rank7]: raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address) [rank7]: RuntimeError: Failed to bind to address [::]:50058; set GRPC_VERBOSITY=debug environment variable to see detailed error message. About to start server Started About to start server Started Starting server on port: 50051 About to start server [2024-08-24 15:56:02,870] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:56:02,870] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... Started [2024-08-24 15:56:03,243] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44848 [2024-08-24 15:56:03,630] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44849 [2024-08-24 15:56:04,007] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44850 [2024-08-24 15:56:04,385] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44851 [2024-08-24 15:56:04,805] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44852 [2024-08-24 15:56:05,183] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44853 [2024-08-24 15:56:05,603] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44854 [2024-08-24 15:56:06,060] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44855 [2024-08-24 15:56:06,060] [ERROR] [launch.py:325:sigkill_handler] ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] exits with return code = 1 [2024-08-24 15:56:07,871] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:56:07,871] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/entrypoints/openai_api_server.py", line 506, in <module> mii.serve(app_settings.model_id, File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/api.py", line 179, in serve import_score_file(mii_config.deployment_name, DeploymentType.LOCAL).init() File "/tmp/mii_cache/mixtral-8x7b-instruct-v0.1/score.py", line 33, in init mii.backend.MIIServer(mii_config) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/backend/server.py", line 50, in __init__ self._wait_until_server_is_live(processes, File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/backend/server.py", line 65, in _wait_until_server_is_live raise RuntimeError( RuntimeError: server crashed for some reason, unable to proceed
Model seems to load but then server fails to start.
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Model seems to load but then server fails to start.
The text was updated successfully, but these errors were encountered: