fix: only fall back to listing prefixes on 404 errors

This logic would previously swallow all errors, including things like
DNS resolution failures. If the path was indeed a path and not a prefix,
that would result in the path being dropped, since the prefix wouldn't
include any further files.

Fixes #18242.

Author: colinmarc

datafusion/datasource/src/url.rs

@@ -252,7 +252,10 @@ impl ListingTableUrl {
                     .boxed(),
                 // If the head command fails, it is likely that object doesn't exist.
                 // Retry as though it were a prefix (aka a collection)
-                Err(_) => list_with_cache(ctx, store, &self.prefix).await?,
+                Err(object_store::Error::NotFound { .. }) => {
+                    list_with_cache(ctx, store, &self.prefix).await?
+                }
+                Err(e) => return Err(e.into()),
             }
         };