how to grant s3 temp permissions when using pyiceberg?

[chengchengpei]

Question

I am looking into https://py.iceberg.apache.org/configuration/#s3.
i tried s3.access-key-id and s3.secret-access-key. they works well. but now, i am thinking how to grant temp permissions.

i am looking into s3.session-token. I have two questions:

1. can i specify s3.session-token only without s3.access-key-id and s3.secret-access-key?
2. how to get the token by aws cli?

I have similar questions about s3.role-arn:
1, how to get this by aws cli?

I know some questions are not relevant to pyiceberg... Any comments welcomed. Thanks

[jayceslesar]

My org uses SSO and we use IAM for cloud runtimes but for local runtimes something like the following:

aws configure export-credentials --profile YOUR_PROFILE_NAME --format env

Will get them into a parseable format

[kevinjqliu]

pyiceberg passes those s3 configs to the underlying filesystem
for pyarrow we use pyarrow.fs.S3FileSystem
for fsspec we use s3fs.S3FileSystem

I think your question is mostly related to

1. how to get different permissions from aws
2. how to pass those permissions to pyiceberg/filesystem

#2 can be found in
https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem
https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

#1 im not sure what the best practice is from AWS, but im sure there are tons of resources online about it