Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't start ['clip_t'] when starting the server. #924

Open
Charlestar opened this issue Jul 18, 2023 · 4 comments
Open

Can't start ['clip_t'] when starting the server. #924

Charlestar opened this issue Jul 18, 2023 · 4 comments

Comments

@Charlestar
Copy link

Hi! I'm trying to deploy clip-as-service on my wsl2, which is Ubuntu 22.04.2 and with CUDA 12.2 for wsl installed.
The python env has python=3.8 and pytorch=2.0.1.
After I installed clip_server and clip_client, I run python -m clip_server as the README shows, however, it just download the 'ViT-B-32.pt' model and failed to continue. Here is the error log:
Snipaste_2023-07-17_20-52-28
Please tell me how to fix this problem, thanks!
@jemmyshin
Copy link
Contributor

Hi, what's your output of jina -vf?

Can you set JINA_LOG_LEVEL=debug and start the server again?

@Charlestar
Copy link
Author

Hi, thank you for your reply! here is my jina -vf output after set JINA_LOG_LEVEL=debug. By restart the server, I find the problem is GRPC not ready (see the output below), then I try to install grpcio and grpcio-tools from conda, but it's still don't work, what should I do next?

(jina) istar@DESKTOP:~$ jina -vf
DeprecationWarning: pkg_resources is deprecated as an API (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:121)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2870)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2870)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2349)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2870)
DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/pkg_resources/__init__.py:2870)
- jina 3.19.0
- docarray 0.21.1
- jcloud 0.2.12
- jina-hubble-sdk 0.39.0
- jina-proto 0.1.27
- protobuf 3.20.3
- proto-backend cpp
- grpcio 1.47.5
- pyyaml 6.0
- python 3.8.17
- platform Linux
- platform-release 5.15.90.1-microsoft-standard-WSL2
- platform-version #1 SMP Fri Jan 27 02:56:13 UTC 2023
- architecture x86_64
- processor x86_64
- uid 91769515676
- session-id 62682054-2529-11ee-ab13-00155de3ae9c
- uptime 2023-07-18T13:10:16.805832
- ci-vendor (unset)
- internal False
* JINA_DEFAULT_HOST (unset)
* JINA_DEFAULT_TIMEOUT_CTRL (unset)
* JINA_DEPLOYMENT_NAME (unset)
* JINA_DISABLE_UVLOOP (unset)
* JINA_EARLY_STOP (unset)
* JINA_FULL_CLI (unset)
* JINA_GATEWAY_IMAGE (unset)
* JINA_GRPC_RECV_BYTES (unset)
* JINA_GRPC_SEND_BYTES (unset)
* JINA_HUB_NO_IMAGE_REBUILD (unset)
* JINA_LOG_CONFIG (unset)
* JINA_LOG_LEVEL debug
* JINA_LOG_NO_COLOR (unset)
* JINA_MP_START_METHOD (unset)
* JINA_OPTOUT_TELEMETRY (unset)
* JINA_RANDOM_PORT_MAX (unset)
* JINA_RANDOM_PORT_MIN (unset)
* JINA_LOCKS_ROOT (unset)
* JINA_K8S_ACCESS_MODES (unset)
* JINA_K8S_STORAGE_CLASS_NAME (unset)
* JINA_K8S_STORAGE_CAPACITY (unset)
* JINA_STREAMER_ARGS (unset)

(jina) istar@DESKTOP:~$ python -m clip_server
DEBUG  clip_t/rep-0@484 Setting signal handlers                                            [07/18/23 13:23:05]
DEBUG  clip_t/rep-0@484 Signal handlers already set
DEBUG  clip_t-replica-set@480 Waiting for ReplicaSet to start successfully                 [07/18/23 13:23:05]
DEBUG  gateway/rep-0@485 Setting signal handlers                                           [07/18/23 13:23:05]
⠋ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 -:--:--DEBUG  gateway/rep-0@485 Signal handlers already set
DEBUG  gateway-replica-set@480 Waiting for ReplicaSet to start successfully                [07/18/23 13:23:05]
⠋ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 -:--:--DEBUG  gateway/rep-0@485 adding connection for deployment clip_t/heads/0 to                [07/18/23 13:23:05]
       grpc://0.0.0.0:60420
DEBUG  gateway/rep-0@485 create_connection connection for clip_t to grpc://0.0.0.0:60420
DEBUG  gateway/rep-0@485 create_connection connection for clip_t to grpc://0.0.0.0:60420
DEBUG  gateway/rep-0@485 connection for deployment clip_t/heads/0 to grpc://0.0.0.0:60420
       added
DEBUG  gateway/rep-0@485 Setting up GRPC server
INFO   gateway/rep-0@485 start server bound to 0.0.0.0:51000
DEBUG  gateway/rep-0@485 Running GatewayRuntime warmup
DEBUG  gateway/rep-0@485 starting warmup task for deployment clip_t
DEBUG  gateway/rep-0@485 server bound to 0.0.0.0:51000 started
DEBUG  gateway/rep-0@485 GRPC server setup successful
DEBUG  gateway/rep-0@480 ready and listening                                               [07/18/23 13:23:06]
DEBUG  gateway-replica-set@480 ReplicaSet started successfully                             [07/18/23 13:23:06]
DEBUG  gateway@480 Deployment started successfully                                         [07/18/23 13:23:06]
⠹ Waiting clip_t gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:00UserWarning: `docs` annotation must be a type hint, got DocumentArray instead, you should maybe remove the string annotation. Default valueDocumentArray will be used instead. (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/jina/serve/executors/__init__.py:131)
⠙ Waiting clip_t... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 0:00:00DeprecationWarning: Importing from timm.models.layers is deprecated, please import via timm.layers (raised from /home/istar/anaconda3/envs/jina/lib/python3.8/site-packages/timm/models/layers/__init__.py:49)
⠴ Waiting clip_t... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 0:00:10DEBUG  clip_t/rep-0@484 <clip_server.executors.clip_torch.CLIPEncoder object at            [07/18/23 13:23:16]
       0x7fd25709aee0> is successfully loaded!
DEBUG  clip_t/rep-0@484 Setting up GRPC server
INFO   clip_t/rep-0@484 start server bound to 0.0.0.0:60420
DEBUG  clip_t/rep-0@484 server bound to 0.0.0.0:60420 started
DEBUG  clip_t/rep-0@484 GRPC server setup successful
DEBUG  clip_t/rep-0@480 Checking readiness to 0.0.0.0:60420 with protocol GRPC             [07/18/23 13:23:16]
⠙ Waiting clip_t... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 0:00:10DEBUG  clip_t/rep-0@484 recv _status request
DEBUG  gateway/rep-0@485 completed warmup task in 10.483339071273804s.                     [07/18/23 13:23:16]
DEBUG  clip_t/rep-0@480 Exception: <AioRpcError of RPC that terminated with:               [07/18/23 13:23:17]
               status = StatusCode.DEADLINE_EXCEEDED
               details = "Deadline Exceeded"
               debug_error_string =
       "{"created":"@1689657797.204551546","description":"Deadline
       Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81…
       >
DEBUG  clip_t/rep-0@480 Server on 0.0.0.0:60420 with protocol GRPC is not yet ready
DEBUG  clip_t/rep-0@480 Checking readiness to 0.0.0.0:60420 with protocol GRPC
DEBUG  clip_t/rep-0@480 Exception: <AioRpcError of RPC that terminated with:               [07/18/23 13:23:18]
               status = StatusCode.DEADLINE_EXCEEDED
               details = "Deadline Exceeded"
               debug_error_string =
       "{"created":"@1689657798.313920705","description":"Deadline
       Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81…
       >
DEBUG  clip_t/rep-0@480 Server on 0.0.0.0:60420 with protocol GRPC is not yet ready
... (repeat the GRPC not ready message)

@ZiniuYu
Copy link
Member

ZiniuYu commented Jul 18, 2023

Unfortunately, we currently do not have access to Windows devices for testing purposes. While we investigate this issue further, we recommend trying the following troubleshooting steps:

  1. Clear the model cache: Clearing the model cache can help resolve any potential caching conflicts. It normally stores at ~/.cache

  2. Configure the VPN: If you are using a VPN, ensure that it is properly configured and not causing any network-related issues. Double-check the VPN settings and try reconnecting or using a different VPN server if possible.

  3. Use a Linux device or Docker image: If you have access to a Linux device or a Docker image, try running the application on that platform. This can help determine if the issue is specific to Windows or if it's a more general problem.

  4. Utilize the SaaS CLIP service via inference.jina.ai: As an alternative, you can leverage our SaaS CLIP service for inference by visiting inference.jina.ai. This allows you to perform inference tasks without relying on local setups.

We appreciate your patience while we work on resolving this issue. If you have any further questions or concerns, please don't hesitate to reach out to us.

@datduyng
Copy link

I have similar issue but seems related to some permission issue because I see some 403 when downloading the model

DEBUG  gateway/rep-0@32033 starting warmup task for deployment clip_o                                         
DEBUG  gateway/rep-0@32033 server bound to 0.0.0.0:51000 started                                              
DEBUG  gateway/rep-0@32033 GRPC server setup successful                                                       
objc[32033]: +[__NSTimeZone initialize] may have been in progress in another thread when fork() was called.
objc[32033]: +[__NSTimeZone initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
DEBUG  gateway/rep-0@32023 ready and listening                                             [09/18/23 23:12:13]
DEBUG  gateway-replica-set@32023 ReplicaSet started successfully                                              
DEBUG  gateway@32023 Deployment started successfully                                       [09/18/23 23:12:13]
UserWarning: `docs` annotation must be a type hint, got DocumentArray instead, you should maybe remove the string annotation. Default valueDocumentArray will be used instead. (raised from /opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg/jina/serve/executors/__init__.py:265)
Failed to download 
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual
.onnx with <HTTPError 403: 'Forbidden'> at the 0th attempt
Failed to download 
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual
.onnx with <HTTPError 403: 'Forbidden'> at the 1th attempt
Failed to download 
https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual
.onnx with <HTTPError 403: 'Forbidden'> at the 2th attempt
  textual.onnx 0.0% • 0/100 bytes • ? • -:--:--
                                               
CRITI… clip_o/rep-0@32032 can not load the executor from {"jtype": "CLIPEncoder", "metas": [09/18/23 23:12:15]
       {"py_modules": ["clip_server.executors.clip_onnx"]}}                                                   
ERROR  clip_o/rep-0@32032 RuntimeError('Failed to download                                                    
       https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276…                    
       within retry limit 3') during 'WorkerRuntime' initialization                                           
        add "--quiet-error" to suppress the exception details                                                 
       Traceback (most recent call last):                                                                     
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 140, in run                                                                                       
           runtime = AsyncNewLoopRuntime(                                                                     
                     ^^^^^^^^^^^^^^^^^^^^                                                                     
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 90, in __init__                                                                                   
           self._loop.run_until_complete(self.async_setup())                                                  
         File                                                                                                 
       "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11…                    
       line 653, in run_until_complete                                                                        
           return future.result()                                                                             
                  ^^^^^^^^^^^^^^^                                                                             
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 270, in async_setup                                                                               
           self.server = self._get_server()                                                                   
                         ^^^^^^^^^^^^^^^^^^                                                                   
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 192, in _get_server                                                                               
           return GRPCServer(                                                                                 
                  ^^^^^^^^^^^                                                                                 
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 34, in __init__                                                                                   
           super().__init__(**kwargs)                                                                         
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 56, in __init__                                                                                   
           self._request_handler = req_handler or self._get_request_handler()                                 
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                 
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 81, in _get_request_handler                                                                       
           return self.req_handler_cls(                                                                       
                  ^^^^^^^^^^^^^^^^^^^^^                                                                       
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 139, in __init__                                                                                  
           self._load_executor(                                                                               
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 354, in _load_executor                                                                            
           self._executor: BaseExecutor = BaseExecutor.load_config(                                           
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^                                           
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 792, in load_config                                                                               
           obj = JAML.load(tag_yml, substitute=False, runtime_args=runtime_args)                              
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                              
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 174, in load                                                                                      
           r = yaml.load(stream, Loader=get_jina_loader_with_runtime(runtime_args))                           
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                           
         File "/opt/homebrew/lib/python3.11/site-packages/yaml/__init__.py", line 81, in                      
       load                                                                                                   
           return loader.get_single_data()                                                                    
                  ^^^^^^^^^^^^^^^^^^^^^^^^                                                                    
         File "/opt/homebrew/lib/python3.11/site-packages/yaml/constructor.py", line 51,                      
       in get_single_data                                                                                     
           return self.construct_document(node)                                                               
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                               
         File "/opt/homebrew/lib/python3.11/site-packages/yaml/constructor.py", line 55,                      
       in construct_document                                                                                  
           data = self.construct_object(node)                                                                 
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                 
         File "/opt/homebrew/lib/python3.11/site-packages/yaml/constructor.py", line 100,                     
       in construct_object                                                                                    
           data = constructor(self, node)                                                                     
                  ^^^^^^^^^^^^^^^^^^^^^^^                                                                     
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 582, in _from_yaml                                                                                
           return get_parser(cls, version=data.get('version', None)).parse(                                   
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                   
         File                                                                                                 
       "/opt/homebrew/lib/python3.11/site-packages/jina-3.20.3-py3.11-macosx-13-arm64.egg…                    
       line 46, in parse                                                                                      
           obj = cls(                                                                                         
                 ^^^^                                                                                         
         File

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jemmyshin @ZiniuYu @datduyng @Charlestar