s3-like compatibility

[zifeo]

while gocloud seems to support s3ForcePathStyle=true, it does not seem possible to use it with pmtiles:

2024/01/04 00:45:10 main.go:128: Failed to create new server, open bucket s3://REDACTED?awssdk=v2&endpoint=REDACTED&region=us-east-1&s3ForcePathStyle=true: unknown query parameter "s3ForcePathStyle"

[bdon]

If you are inputting the URLs can you use virtual host-style URLs instead? AWS has been trying to deprecate path style URLs for a while https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access so it seem like they should be 1:1 equivalent.

[zifeo]

@bdon Thanks for getting back, unfortunately our storage provider does not offer virtual ones, only path style are supported.

[bdon]

Which storage provider or storage engine is this so others can reproduce?

[zifeo]

@bdon Openstack Swift, but I suppose you can also use minio to replicate.

[bdon]

PR here: #113

Currently we force the use of awssdkv2, but v1 is necessary to support those flags and work with Minio. Defaulting to v1 shouldn't have any functional impact. The alternative is to add more storage implementation-specific logic to go-pmtiles but that would defeat the purpose of using the gocloud abstraction.

Aligns with suggested way of using Minio in gocloud docs: https://gocloud.dev/howto/blob/#s3-compatible

[bdon]

I've merged it into main, please see if it fixes the issue and I'll make a minor release.

[zifeo]

@bdon confirmed, thanks a lot!

[bdon]

Released in https://github.com/protomaps/go-pmtiles/releases/tag/v1.12.0

[bdon]

FYI, AWS is putting aws-sdk-go v1 into maintenance mode; v2 will be the default going forward: https://github.com/google/go-cloud/releases/tag/v0.39.0

For the next minor version I am bumping the gocloud version

[zifeo]

@bdon Thanks for letting me know, what is the expect change? It seems unclear to me

[bdon]

Based on those release notes Gocloud will be removing aws-sdk-v1 in early 2025.

The AWS SDK V2 definitely supports forcing path-style S3 buckets, not quite sure how it's exposed through gocloud S3 connection strings.

[zifeo]

Seems to be the same: https://github.com/google/go-cloud/blob/master/aws/aws.go#L99

[bdon]

1.22 defaults to awsv2: https://github.com/protomaps/go-pmtiles/releases/tag/v1.22.0

[zifeo]

@bdon upgraded and it worked out of the box :)