S3 container sometimes doesn't resolve from PHP container

[mikelittle]

Sometimes, whether from a brand-new stack or an existing stack, uploading images results in an error that boils down to the PHP container being unable to resolve the MinIO s3 emulation container.

Here's a brain dump of notes from a brief investigation into the problem. Note, sometimes the S3 container does resolve, but the Tachyon one doesn't. This has only occurred on one machine. A previous macBook does not exhibit the problem (but then my current one didn't used to either). It doesn't occur (for me) on Linux (Ubuntu 24.04).

• Found fatal error uploading an image as author

qatest-php  | [07-Aug-2024 09:35:43 UTC] PHP Warning:  Uncaught exception 'Aws\S3\Exception\S3Exception' with message 'Error executing "ListObjectsV2" on "https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777"; AWS HTTP error: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777'
qatest-php  |
qatest-php  | GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777 in /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:275
qatest-php  | Stack trace:
qatest-php  | #0 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(205): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array)
qatest-php  | #1 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(157): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
qatest-php  | #2 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(267): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
qatest-php  | #3 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(180): GuzzleHttp\Handler\CurlMultiHandler->processMessages()
qatest-php  | #4 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(206): GuzzleHttp\Handler\CurlMultiHandler->tick()
qatest-php  | #5 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
qatest-php  | #6 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn()
qatest-php  | #7 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #8 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #9 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #10 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #11 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #12 /usr/src/app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
qatest-php  | #13 /usr/src/app/vendor/aws/aws-sdk-php/src/ResultPaginator.php(138): Aws\AwsClient->execute(Object(Aws\Command))
qatest-php  | #14 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(52): Aws\ResultPaginator->valid()
qatest-php  | #15 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(69): Aws\map(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #16 [internal function]: Aws\flatmap(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #17 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-stream-wrapper.php(695): Generator->valid()
qatest-php  | #18 [internal function]: S3_Uploads\Stream_Wrapper->dir_readdir()
qatest-php  | #19 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-plugin.php(676): scandir('s3://s3-qatest/...')
qatest-php  | #20 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): S3_Uploads\Plugin->get_files_for_unique_filename_file_list(NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #21 /usr/src/app/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters(NULL, Array)
qatest-php  | #22 /usr/src/app/wordpress/wp-includes/functions.php(2667): apply_filters('pre_wp_unique_f...', NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #23 /usr/src/app/wordpress/wp-admin/includes/file.php(984): wp_unique_filename('s3://s3-qatest/...', 'zed1-img_1777.j...', NULL)
qatest-php  | #24 /usr/src/app/wordpress/wp-admin/includes/file.php(1105): _wp_handle_upload(Array, Array, '2024-08-07 09:3...', 'wp_handle_uploa...')
qatest-php  | #25 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(1322): wp_handle_upload(Array, Array, '2024-08-07 09:3...')
qatest-php  | #26 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(269): WP_REST_Attachments_Controller->upload_from_file(Array, Array, '2024-08-07 09:3...')
qatest-php  | #27 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(158): WP_REST_Attachments_Controller->insert_attachment(Object(WP_REST_Request))
qatest-php  | #28 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1230): WP_REST_Attachments_Controller->create_item(Object(WP_REST_Request))
qatest-php  | #29 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1063): WP_REST_Server->respond_to_request(Object(WP_REST_Request), '/wp/v2/media', Array, NULL)
qatest-php  | #30 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(439): WP_REST_Server->dispatch(Object(WP_REST_Request))
qatest-php  | #31 /usr/src/app/wordpress/wp-includes/rest-api.php(420): WP_REST_Server->serve_request('/wp/v2/media')
qatest-php  | #32 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): rest_api_loaded(Object(WP))
qatest-php  | #33 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters(false, Array)
qatest-php  | #34 /usr/src/app/wordpress/wp-includes/plugin.php(565): WP_Hook->do_action(Array)
qatest-php  | #35 /usr/src/app/wordpress/wp-includes/class-wp.php(418): do_action_ref_array('parse_request', Array)
qatest-php  | #36 /usr/src/app/wordpress/wp-includes/class-wp.php(813): WP->parse_request('')
qatest-php  | #37 /usr/src/app/wordpress/wp-includes/functions.php(1336): WP->main('')
qatest-php  | #38 /usr/src/app/wordpress/wp-blog-header.php(16): wp()
qatest-php  | #39 /usr/src/app/index.php(21): require('/usr/src/app/wo...')
qatest-php  | #40 {main}
qatest-php  |
qatest-php  | Next Aws\S3\Exception\S3Exception: Error executing "ListObjectsV2" on "https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777"; AWS HTTP error: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777 in /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:196
qatest-php  | Stack trace:
qatest-php  | #0 /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php(97): Aws\WrappedHttpHandler->parseError(Array, Object(GuzzleHttp\Psr7\Request), Object(Aws\Command), Array)
qatest-php  | #1 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(209): Aws\WrappedHttpHandler->Aws\{closure}(Array)
qatest-php  | #2 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(174): GuzzleHttp\Promise\Promise::callHandler(2, Array, NULL)
qatest-php  | #3 /usr/src/app/vendor/guzzlehttp/promises/src/RejectedPromise.php(49): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}(Array)
qatest-php  | #4 /usr/src/app/vendor/guzzlehttp/promises/src/TaskQueue.php(52): GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}()
qatest-php  | #5 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(167): GuzzleHttp\Promise\TaskQueue->run()
qatest-php  | #6 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(206): GuzzleHttp\Handler\CurlMultiHandler->tick()
qatest-php  | #7 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
qatest-php  | #8 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn()
qatest-php  | #9 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #10 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #11 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #12 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #13 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #14 /usr/src/app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
qatest-php  | #15 /usr/src/app/vendor/aws/aws-sdk-php/src/ResultPaginator.php(138): Aws\AwsClient->execute(Object(Aws\Command))
qatest-php  | #16 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(52): Aws\ResultPaginator->valid()
qatest-php  | #17 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(69): Aws\map(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #18 [internal function]: Aws\flatmap(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #19 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-stream-wrapper.php(695): Generator->valid()
qatest-php  | #20 [internal function]: S3_Uploads\Stream_Wrapper->dir_readdir()
qatest-php  | #21 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-plugin.php(676): scandir('s3://s3-qatest/...')
qatest-php  | #22 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): S3_Uploads\Plugin->get_files_for_unique_filename_file_list(NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #23 /usr/src/app/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters(NULL, Array)
qatest-php  | #24 /usr/src/app/wordpress/wp-includes/functions.php(2667): apply_filters('pre_wp_unique_f...', NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #25 /usr/src/app/wordpress/wp-admin/includes/file.php(984): wp_unique_filename('s3://s3-qatest/...', 'zed1-img_1777.j...', NULL)
qatest-php  | #26 /usr/src/app/wordpress/wp-admin/includes/file.php(1105): _wp_handle_upload(Array, Array, '2024-08-07 09:3...', 'wp_handle_uploa...')
qatest-php  | #27 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(1322): wp_handle_upload(Array, Array, '2024-08-07 09:3...')
qatest-php  | #28 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(269): WP_REST_Attachments_Controller->upload_from_file(Array, Array, '2024-08-07 09:3...')
qatest-php  | #29 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(158): WP_REST_Attachments_Controller->insert_attachment(Object(WP_REST_Request))
qatest-php  | #30 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1230): WP_REST_Attachments_Controller->create_item(Object(WP_REST_Request))
qatest-php  | #31 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1063): WP_REST_Server->respond_to_request(Object(WP_REST_Request), '/wp/v2/media', Array, NULL)
qatest-php  | #32 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(439): WP_REST_Server->dispatch(Object(WP_REST_Request))
qatest-php  | #33 /usr/src/app/wordpress/wp-includes/rest-api.php(420): WP_REST_Server->serve_request('/wp/v2/media')
qatest-php  | #34 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): rest_api_loaded(Object(WP))
qatest-php  | #35 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters(false, Array)
qatest-php  | #36 /usr/src/app/wordpress/wp-includes/plugin.php(565): WP_Hook->do_action(Array)
qatest-php  | #37 /usr/src/app/wordpress/wp-includes/class-wp.php(418): do_action_ref_array('parse_request', Array)
qatest-php  | #38 /usr/src/app/wordpress/wp-includes/class-wp.php(813): WP->parse_request('')
qatest-php  | #39 /usr/src/app/wordpress/wp-includes/functions.php(1336): WP->main('')
qatest-php  | #40 /usr/src/app/wordpress/wp-blog-header.php(16): wp()
qatest-php  | #41 /usr/src/app/index.php(21): require('/usr/src/app/wo...')
qatest-php  | #42 {main}
qatest-php  |   thrown in /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php on line 196
qatest-php  | [07-Aug-2024 09:35:43 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 41943040 bytes) in /usr/src/app/vendor/humanmade/s3-uploads/inc/class-plugin.php on line 676

• Further testing.
• Tried connecting to S3 from the docker shell - didn't work. Same from v19

$ composer server ssh
www-data@a7c1f17b35b0:/usr/src/app$ curl "https://altis19.altis.dev/s3-altis19/"
curl: (7) Failed to connect to altis19.altis.dev port 443 after 42 ms: Couldn't connect to server
www-data@a7c1f17b35b0:/usr/src/app$

• testing inside qa-test v20 gave this

www-data@08674edaddfe:/usr/src/app$ curl -if 'http://s3:9000/minio/health/live'
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 0
Content-Security-Policy: block-all-mixed-content
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Bucket-Region: us-east-1
X-Amz-Request-Id: 17E97BFE1103A8B1
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Date: Wed, 07 Aug 2024 15:31:03 GMT

www-data@08674edaddfe:/usr/src/app$

• testing qatest v20 again. FQDN for S3 does not work

www-data@08674edaddfe:/usr/src/app$ curl "http://s3-qatest.altis.dev:9000/s3-qatest/"
curl: (7) Failed to connect to s3-qatest.altis.dev port 9000 after 4 ms: Couldn't connect to server

• but short name does work

www-data@08674edaddfe:/usr/src/app$ curl "http://s3:9000/s3-qatest/"
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>s3-qatest</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter></Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>www-data@08674edaddfe:/usr/src/app$

• same problem with qatest. Also, s3-uploads command broken too

$ composer server cli -- s3-uploads upload-directory tmp uploads
Error: Error executing "PutObject" on "https://s3-qatest.altis.dev/s3-qatest/uploads/zed1-img_1922.jpg"; 
  AWS HTTP error: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: 
  Couldn\'t connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/uploads/zed1-img_1922.jpg

Acceptance criteria:

• ...

[roborourke]

@mikelittle I wouldn't expect the FQDN to work within the containers so I think that's ok. There might be a configuration issue then, by which WP is trying to use the FQDN to interact with S3 rather than the container link name and port (the short name). The FQDN is purely for use via the host. S3 Uploads should be using the short URL or s3.localhost to speak to Minio from the PHP container, if it isn't then that's the problem.

Looks like it should be available at s3.localhost from the PHP container too. I recall the the S3 Uploads plugin and AWS SDK had some odd behaviour around how it resolve S3 URLs, it's possible this issue has crept in with an AWS SDK update.

[mikelittle]

Ah, OK. That was my expectation too, but I couldn't figure out where it was picking up the FQDN from. I assumed it had always used the full name and worked somehow before.
If it might be something an AWS SDK update has done, that gives me a pointer to track this down. Still, odd that it works on a brand-new installation on an old Mac, and continues to work on new installations on linux VMs.

[roborourke]

It might be worth tinkering with this code a bit

https://github.com/humanmade/altis-local-server/blob/master/inc/namespace.php#L28-L37

I would try adding some breakpoints with xdebug to see which URL its trying to use. It might arrive at the FQDN as part of some attempts to retry or try different versions of the S3 endpoint.

I can't actually remember why the S3 endpoint is set to the FQDN, I thought it should be the internal container link URL e.g. s3 or s3.localhost. It was a fairly complex piece to get working. Odd that it could be inconsistent like this.

Maybe could factor in docker engine versions? Are they same between Mac and Linux?

@abhishek-kaushik is this happening on all your Altis instances or only some? What about on a clean install?

[abhishek-kaushik]

Yes @roborourke I am facing this issue on all of my stacks