[Bug]: Client sent an HTTP request to an HTTPS server. #28032

gaoyuan5251 · 2023-10-31T02:39:43Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster): standalone 2.3.1
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): 
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

With Milvus-operator, deploy milvus single-node mode, but rely on Etcd and minio external clusters.

Expected Behavior

milvus standalone should be running.

Steps To Reproduce

1. vim milvs-external.yaml
2. kubectl apply -f milvs-external.yaml
3. kubectl describe milvus >desc.txt
4.kubectl logs hll-release-milvus-standalone-d57df84d8-qdmms > log.txt

### Milvus Log

[milvs-external.txt](https://github.com/milvus-io/milvus/files/13213385/milvs-external.txt)
[desc.txt](https://github.com/milvus-io/milvus/files/13213398/desc.txt)
[log.txt](https://github.com/milvus-io/milvus/files/13213393/log.txt)



### Anything else?

yanliang567 · 2023-10-31T07:51:10Z

looks like the config yaml is not parsed correctly. @haorenfsa could you please take a look?
/assign @haorenfsa
/unassign

gaoyuan5251 · 2023-11-01T06:27:12Z

@haorenfsa @yanliang567 bro, is anyone looking at this issue?

haorenfsa · 2023-11-03T09:57:00Z

@gaoyuan5251 Sry for the delay, cloud you provide the output of kubectl get configmap hll-release -o yaml?

gaoyuan5251 · 2023-11-06T03:46:52Z

@haorenfsa ok,
configmap.txt

gaoyuan5251 · 2023-11-06T03:49:40Z

@haorenfsa cc, kubectl describe milvus hll-release > status.txt
status.txt

Is this because the test machine is out of memory?

haorenfsa · 2023-11-06T05:00:01Z

Is this because the test machine is out of memory?

looks no.

@haorenfsa cc, kubectl describe milvus hll-release > status.txt status.txt

@yanliang567 looks the configmap is rendered correctly. @zhagnlu have you met this error before?

gaoyuan5251 · 2023-11-06T06:08:12Z

Is this because the test machine is out of memory?

looks no.

@haorenfsa cc, kubectl describe milvus hll-release > status.txt status.txt

@yanliang567 looks the configmap is rendered correctly. @zhagnlu have you met this error before?

...
Status:
Components Deploy Status:
Standalone:
Generation: 1
Image: milvusdb/milvus:v2.3.1
Status:
Conditions:
Last Transition Time: 2023-11-06T03:23:34Z
Last Update Time: 2023-11-06T03:23:34Z
Message: Deployment does not have minimum availability.
Reason: MinimumReplicasUnavailable
Status: False
Type: Available
Last Transition Time: 2023-11-06T03:33:35Z
Last Update Time: 2023-11-06T03:33:35Z
Message: ReplicaSet "hll-release-milvus-standalone-64b987fcf5" has timed out progressing.
Reason: ProgressDeadlineExceeded
Status: False
Type: Progressing
Observed Generation: 1
Replicas: 1
Unavailable Replicas: 1
Updated Replicas: 1
...
What does this part of the message mean? @haorenfsa

zhagnlu · 2023-11-07T04:00:30Z

#27853 same error. check deploy and config params about k8s

yanliang567 · 2023-11-07T04:19:03Z

/assign @zhagnlu
/unassign @haorenfsa

gaoyuan5251 · 2023-11-07T06:03:43Z

#27853 same error. check deploy and config params about k8s
bro, Could you mind help me check, what is wrong with my yaml file? @zhagnlu
milvs-external.yml.txt

gaoyuan5251 · 2023-11-07T06:38:27Z

kubectl logs hll-release-milvus-standalone-6c747c8948-kqdjl > logs.txt
logs.txt

[2023/11/07 06:33:46.053 +00:00] [ERROR] [querynode/service.go:129] ["QueryNode init error: "] [error="Endpoint url cannot have fully qualified paths."] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:129\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:223\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"]
[2023/11/07 06:33:46.053 +00:00] [ERROR] [components/query_node.go:54] ["QueryNode starts error"] [error="Endpoint url cannot have fully qualified paths."] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"]
panic: Endpoint url cannot have fully qualified paths.

@zhagnlu idk why this error is reported

xiaofan-luan · 2023-11-07T06:54:42Z

@gaoyuan5251
Did you enable SSL on your minio or S3?

gaoyuan5251 · 2023-11-07T06:57:02Z

@gaoyuan5251 Did you enable SSL on your minio or S3?

Nope @xiaofan-luan

xiaofan-luan · 2023-11-07T06:57:27Z

@zhagnlu @zwd1208

xiaofan-luan · 2023-11-07T06:57:42Z

@gaoyuan5251 Did you enable SSL on your minio or S3?

Nope @xiaofan-luan

you are not enable ssl, what about your minio config?

xiaofan-luan · 2023-11-07T06:58:07Z

https://github.com/zhagnlu/milvus-store-tool

this tool could help you to verify your S3 connectivity

gaoyuan5251 · 2023-11-07T06:58:44Z

@xiaofan-luan I uploaded my yaml file at the top so you can take a look

gaoyuan5251 · 2023-11-07T07:00:04Z

#27853 same error. check deploy and config params about k8s
bro, Could you mind help me check, what is wrong with my yaml file? @zhagnlu
milvs-external.yml.txt

@xiaofan-luan cc

haorenfsa · 2023-11-07T07:45:33Z

@gaoyuan5251 Cloud you use the minio client command mc to check the if the minio server address 10.102.235.153:80 is available?

gaoyuan5251 · 2023-11-07T07:48:14Z

@gaoyuan5251 Cloud you use the minio client command mc to check the if the minio server address 10.102.235.153:80 is available?

mc tree hllminio
hllminio
└─ hll-milvus
└─ k8s部署

@haorenfsa

gaoyuan5251 · 2023-11-07T07:52:36Z

@gaoyuan5251 Cloud you use the minio client command mc to check the if the minio server address 10.102.235.153:80 is available?

我重新部署了minio集群，cluster-ip变更了。但是可以看到 minio 服务使用上是没问题的。

mc config host list
gcs
URL : https://storage.googleapis.com
AccessKey : YOUR-ACCESS-KEY-HERE
SecretKey : YOUR-SECRET-KEY-HERE
API : S3v2
Path : dns

hllminio
URL : http://10.108.111.59:80
AccessKey : XS1emrJC6zxv65tV
SecretKey : XS1emrJC6zxv65tV
API : s3v4
Path : auto

haorenfsa · 2023-11-07T07:54:42Z

@gaoyuan5251 cloud you also get the output of kubectl exec -it deploy/hll-release-milvus-standalone -- cat /milvus/configs/milvus.yaml when the pod is running?

gaoyuan5251 · 2023-11-07T07:55:32Z

gaoyuan5251 · 2023-11-07T07:56:48Z

@gaoyuan5251 cloud you also get the output of kubectl exec -it deploy/hll-release-milvus-standalone -- cat /milvus/configs/milvus.yaml when the pod is running?

@haorenfsa
kubectl exec -it deploy/hll-release-milvus-standalone -- cat /milvus/configs/milvus.yaml
Defaulted container "standalone" out of: standalone, config (init)
error: unable to upgrade connection: container not found ("standalone")

haorenfsa · 2023-11-07T07:56:57Z

you can first delete the old pod kubectl delete pod hll-release-milvus-standalone-6c747c8948-kqdjl, kubernetes will automatically start a new pod for you. then try this again

kubectl exec -it deploy/hll-release-milvus-standalone -- cat /milvus/configs/milvus.yaml

gaoyuan5251 · 2023-11-07T07:58:35Z

you can first delete the old pod kubectl delete pod hll-release-milvus-standalone-6c747c8948-kqdjl, kubernetes will automatically start a new pod for you. then try this again

@haorenfsa done

haorenfsa · 2023-11-07T07:59:53Z

keep trying, until you got a yaml output ...

gaoyuan5251 · 2023-11-07T08:08:10Z

milvus.yaml.txt

@haorenfsa done

gaoyuan5251 · 2023-11-07T08:32:10Z

@haorenfsa Isn't the reason obvious? etcd and minio, which build their own clusters, seem to have no problem

haorenfsa · 2023-11-07T08:49:19Z

@haorenfsa Isn't the reason obvious? etcd and minio, which build their own clusters, seem to have no problem

very strange, I run the same code with your uploaded configuration, and no error occurs

haorenfsa · 2023-11-07T09:04:11Z

@gaoyuan5251 looks like the k8s service minio in the same namespace will sets env of MINIO_PORT as an unexpected value. try deploy the minio in different namespace with milvus instead.

ref:
#28215 (comment)

haorenfsa · 2023-11-07T09:05:47Z

Or you can just delete the k8s service minio and rename it to other name than just minio, like minio-service or so

gaoyuan5251 · 2023-11-09T06:49:53Z

Or you can just delete the k8s service minio and rename it to other name than just minio, like minio-service or so

Sry for the delay, thank u @haorenfsa
After I move minio to a different namespace in k8s, milvus is worked.
but idk why it work ,hh

haorenfsa · 2023-11-10T03:29:08Z

@gaoyuan5251 it's an issue of the minio SDK used by milvus. The SDK uses the env var MINIO_PORT to overwrite its config. You created a service named minio, and kubernetes auto generates a $servicename_PORT(which is also MINIO_PORT) env, value as tcp://svcIP:svcPort for every pod in the same namespace. so your configuration of minio port which is 80 got overwritten by tcp://svcIP:svcPort.

So long store short, don't name your MinIO service as minio, where ever you uses MinIO.

gaoyuan5251 · 2023-11-10T03:35:37Z

@gaoyuan5251 it's an issue of the minio SDK used by milvus. The SDK uses the env var MINIO_PORT to overwrite its config. You created a service named minio, and kubernetes auto generates a $servicename_PORT(which is also MINIO_PORT) env, value as tcp://svcIP:svcPort for every pod in the same namespace. so your configuration of minio port which is 80 got overwritten by tcp://svcIP:svcPort.

So long store short, don't name your MinIO service as minio, where ever you uses MinIO.

ok, i get it, thank u so much ~ @haorenfsa

stale · 2023-12-17T03:39:01Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

wangjn163 · 2024-04-17T03:46:35Z

Or you can just delete the k8s service minio and rename it to other name than just minio, like minio-service or so

我用这个解决了问题

Niceies · 2025-03-11T12:26:50Z

@haorenfsa @yanliang567

Request to open

This problem is not a problem with minio sdk. minio sdk does not get MINIO_PORT from the environment variable. It is that envsoure is configured to read the environment variable when Milvus is started. However, the KeyFormatter logic that matches the MILVUS environment variable seems to lack a consideration that the environment variable that does not start with MILVUS_ can also be hit.

pkg/util/paramtable/base_table.go

func (bt *BaseTable) init() {
	formatter := func(key string) string {
		ret := strings.ToLower(key)
		ret = strings.TrimPrefix(ret, "milvus.")
		ret = strings.ReplaceAll(ret, "/", "")
		ret = strings.ReplaceAll(ret, "_", "")
		ret = strings.ReplaceAll(ret, ".", "")
		return ret
	}
....
}

I think it is correct to filter out the environment variables of milvus when setting the environment variables. After I added the following code, my milvus started normally.

pkg/config/env_source.go

func NewEnvSource(KeyFormatter func(string) string) EnvSource {
	es := EnvSource{
		configs:      typeutil.NewConcurrentMap[string, string](),
		KeyFormatter: KeyFormatter,
	}

	for _, value := range os.Environ() {
		rs := []rune(value)
		in := strings.Index(value, "=")
		key := string(rs[0:in])
		value := string(rs[in+1:])
		if !strings.HasPrefix(key, "MILVUS_") {              # Add a variable to determine whether to skip non-milvus
			continue
		}
		envKey := KeyFormatter(key)
		es.configs.Insert(key, value)
		es.configs.Insert(envKey, value)
	}
	return es
}

xiaofan-luan · 2025-03-11T23:41:52Z

@haorenfsa @yanliang567

Request to open

This problem is not a problem with minio sdk. minio sdk does not get MINIO_PORT from the environment variable. It is that envsoure is configured to read the environment variable when Milvus is started. However, the KeyFormatter logic that matches the MILVUS environment variable seems to lack a consideration that the environment variable that does not start with MILVUS_ can also be hit.

pkg/util/paramtable/base_table.go
func (bt *BaseTable) init() {
	formatter := func(key string) string {
		ret := strings.ToLower(key)
		ret = strings.TrimPrefix(ret, "milvus.")
		ret = strings.ReplaceAll(ret, "/", "")
		ret = strings.ReplaceAll(ret, "_", "")
		ret = strings.ReplaceAll(ret, ".", "")
		return ret
	}
....
}
I think it is correct to filter out the environment variables of milvus when setting the environment variables. After I added the following code, my milvus started normally.

pkg/config/env_source.go
func NewEnvSource(KeyFormatter func(string) string) EnvSource {
	es := EnvSource{
		configs:      typeutil.NewConcurrentMap[string, string](),
		KeyFormatter: KeyFormatter,
	}

	for _, value := range os.Environ() {
		rs := []rune(value)
		in := strings.Index(value, "=")
		key := string(rs[0:in])
		value := string(rs[in+1:])
		if !strings.HasPrefix(key, "MILVUS_") {              # Add a variable to determine whether to skip non-milvus
			continue
		}
		envKey := KeyFormatter(key)
		es.configs.Insert(key, value)
		es.configs.Insert(envKey, value)
	}
	return es
}

What is your env looks like?
Milvus is running under docker and all the params should be adopted by milvus.

I believe you need to set minio.address and minio.port

Niceies · 2025-03-12T01:40:16Z

@xiaofan-luan
I am running in k8s mode, and then connecting to the dependent storage components. What I mean is that it is a better way to obtain parameter settings if we only use environment variables starting with the special identifier Milvus to match, otherwise there may be conflicts with system variables.

At first, I set the externalS3 address through --set, as shown in the figure, but it failed to start, as shown in the title of this issue. Then I found that the problem was that the port was concatenated in the minio configuration initialization address, resulting in an illegal format. In fact, the mino.port here uses the value in the environment variable (tcp://10.192.122.189:9000)

Niceies · 2025-03-12T02:49:35Z

@xiaofan-luan
In addition, the minio service appears because the current version of the minio-operator built-in code fixes this content. If you want to deploy minio and milvus in the same namespace, there will inevitably be problems. I tried to start milvus by reading only the configuration file instead of reading the environment variable, but I checked the config.skipEnv option in the milvus code and could not be configured externally.

This is minio-operator code:

gaoyuan5251 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 31, 2023

gaoyuan5251 assigned yanliang567 Oct 31, 2023

sre-ci-robot assigned haorenfsa and unassigned yanliang567 Oct 31, 2023

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 31, 2023

yanliang567 added this to the 2.3.3 milestone Oct 31, 2023

sre-ci-robot assigned zhagnlu and unassigned haorenfsa Nov 7, 2023

yanliang567 modified the milestones: 2.3.3, 2.3.4 Nov 16, 2023

stale bot added the stale indicates no udpates for 30 days label Dec 17, 2023

stale bot closed this as completed Dec 24, 2023

lhx692135353 mentioned this issue Apr 23, 2024

报错Endpoint url cannot have fully qualified paths zilliztech/milvus-operator#109

Closed

jagirard mentioned this issue Dec 16, 2024

[Bug]: Alternative solutions or potential update for the MinIO operator issue with Milvus #38504

Closed

1 task

Niceies mentioned this issue Mar 11, 2025

[Bug]: panic: Endpoint url cannot have fully qualified paths. #40471

Open

1 task

[Bug]: Client sent an HTTP request to an HTTPS server. #28032

[Bug]: Client sent an HTTP request to an HTTPS server. #28032

Comments

gaoyuan5251 commented Oct 31, 2023 • edited Loading

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

yanliang567 commented Oct 31, 2023

gaoyuan5251 commented Nov 1, 2023

haorenfsa commented Nov 3, 2023

gaoyuan5251 commented Nov 6, 2023

gaoyuan5251 commented Nov 6, 2023 • edited Loading

haorenfsa commented Nov 6, 2023

gaoyuan5251 commented Nov 6, 2023

zhagnlu commented Nov 7, 2023

yanliang567 commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

xiaofan-luan commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

xiaofan-luan commented Nov 7, 2023

xiaofan-luan commented Nov 7, 2023

xiaofan-luan commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

haorenfsa commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

haorenfsa commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

haorenfsa commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

haorenfsa commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

gaoyuan5251 commented Nov 7, 2023

haorenfsa commented Nov 7, 2023

haorenfsa commented Nov 7, 2023

haorenfsa commented Nov 7, 2023

gaoyuan5251 commented Nov 9, 2023

haorenfsa commented Nov 10, 2023 • edited Loading

gaoyuan5251 commented Nov 10, 2023 • edited Loading

stale bot commented Dec 17, 2023

wangjn163 commented Apr 17, 2024

Niceies commented Mar 11, 2025 • edited Loading

Request to open

pkg/util/paramtable/base_table.go

pkg/config/env_source.go

xiaofan-luan commented Mar 11, 2025

Request to open

pkg/util/paramtable/base_table.go

pkg/config/env_source.go

Niceies commented Mar 12, 2025 • edited Loading

Niceies commented Mar 12, 2025

gaoyuan5251 commented Oct 31, 2023 •

edited

Loading

gaoyuan5251 commented Nov 6, 2023 •

edited

Loading

haorenfsa commented Nov 10, 2023 •

edited

Loading

gaoyuan5251 commented Nov 10, 2023 •

edited

Loading

Niceies commented Mar 11, 2025 •

edited

Loading

Niceies commented Mar 12, 2025 •

edited

Loading