Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Common: Support import the exist file in object storage. #3677

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

baifachuan
Copy link
Contributor

@baifachuan baifachuan commented Nov 27, 2024

#3663

What problem does this PR solve?

Briefly describe what this PR aims to solve. Include background context that will help reviewers understand the purpose of the PR.

Type of change

  • New Feature (non-breaking change which adds functionality)

@baifachuan baifachuan changed the title Common: Support the exist file in object storage. Common: Support import the exist file in object storage. Nov 27, 2024
@baifachuan
Copy link
Contributor Author

image image

image

@KevinHuSh KevinHuSh added the ci Continue Integration label Nov 27, 2024
@yingfeng
Copy link
Member

Hi, thanks for your contribution!
This PR will block CI, could you please figure out the reason and submit fix? Thank you!

@Feiue
Copy link
Contributor

Feiue commented Dec 18, 2024

2024-12-18 11:08:14 ragflow-server  | Traceback (most recent call last):
2024-12-18 11:08:14 ragflow-server  |   File "/ragflow/api/apps/storage_app.py", line 35, in list_storage_keys
2024-12-18 11:08:14 ragflow-server  |     files = STORAGE_IMPL.list(None, dir)
2024-12-18 11:08:14 ragflow-server  |   File "/ragflow/rag/utils/minio_conn.py", line 62, in list
2024-12-18 11:08:14 ragflow-server  |     data = [{"name": key.object_name, "size": key.size, "etag": key.etag, "owner": key.owner_name} for key in keys]
2024-12-18 11:08:14 ragflow-server  |   File "/ragflow/rag/utils/minio_conn.py", line 62, in <listcomp>
2024-12-18 11:08:14 ragflow-server  |     data = [{"name": key.object_name, "size": key.size, "etag": key.etag, "owner": key.owner_name} for key in keys]
2024-12-18 11:08:14 ragflow-server  |   File "/ragflow/.venv/lib/python3.10/site-packages/minio/api.py", line 3111, in _list_objects
2024-12-18 11:08:14 ragflow-server  |     response = self._execute(
2024-12-18 11:08:14 ragflow-server  |   File "/ragflow/.venv/lib/python3.10/site-packages/minio/api.py", line 437, in _execute
2024-12-18 11:08:14 ragflow-server  |     region = self._get_region(bucket_name)
2024-12-18 11:08:14 ragflow-server  |   File "/ragflow/.venv/lib/python3.10/site-packages/minio/api.py", line 494, in _get_region
2024-12-18 11:08:14 ragflow-server  |     response = self._url_open(
2024-12-18 11:08:14 ragflow-server  |   File "/ragflow/.venv/lib/python3.10/site-packages/minio/api.py", line 423, in _url_open
2024-12-18 11:08:14 ragflow-server  |     raise response_error
2024-12-18 11:08:14 ragflow-server  | minio.error.S3Error: S3 operation failed; code: NoSuchBucket, message: The specified bucket does not exist, resource: /bucket, request_id: 181226A477AEFD17, host_id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, bucket_name: bucket

I have already added the relevant information for import_bucket in both service_conf.yaml and service_conf.yaml.template, but it seems the error still occurs.

@@ -5,16 +5,19 @@
from botocore.client import Config
import time
from io import BytesIO
from rag import settings
from rag.settings import s3_logger
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting a message here that s3_logger does not exist in the settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continue Integration
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants