Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove all versions of a file form the S3 bucket #9171

Merged
merged 6 commits into from
May 27, 2020
Merged

Remove all versions of a file form the S3 bucket #9171

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
831e3f0
Remove all versions of a file form the S3 bucket
May 24, 2020
56af880
Remove TODO that was not addressed for a long time
May 24, 2020
1a38a3f
Return os.ErrNotExist when no versions are found
May 24, 2020
a48ccfa
Return warning instead of error to hide issues during cluster teardown
May 24, 2020
b565122
Remove delete markers also from S3 bucket
May 24, 2020
9675692
Implement RemoveAll() for S3 paths
May 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 28 additions & 6 deletions util/pkg/vfs/s3fs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,15 +92,37 @@ func (p *S3Path) Remove() error {

klog.V(8).Infof("removing file %s", p)

request := &s3.DeleteObjectInput{}
request.Bucket = aws.String(p.bucket)
request.Key = aws.String(p.key)
request := &s3.ListObjectVersionsInput{
Bucket: aws.String(p.bucket),
Prefix: aws.String(p.key),
}

_, err = client.DeleteObject(request)
response, err := client.ListObjectVersions(request)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBD: will this require new IAM permissions?

if err != nil {
// TODO: Check for not-exists, return os.NotExist
return fmt.Errorf("error listing versions %s: %v", p, err)
hakman marked this conversation as resolved.
Show resolved Hide resolved
}

// Sometimes S3 will return paginated results if there are too many versions for an object.
// This is unlikely with current use cases, but a warning should be triggered in case it ever occurs.
if aws.BoolValue(response.IsTruncated) {
klog.Warningf("too many versions for %s", p)
}

return fmt.Errorf("error deleting %s: %v", p, err)
for _, version := range response.Versions {
klog.V(8).Infof("removing file %s version %q", p, aws.StringValue(version.VersionId))

request := &s3.DeleteObjectInput{
Bucket: aws.String(p.bucket),
Key: aws.String(p.key),
VersionId: version.VersionId,
}

_, err = client.DeleteObject(request)
if err != nil {
// TODO: Check for not-exists, return os.NotExist
hakman marked this conversation as resolved.
Show resolved Hide resolved

return fmt.Errorf("error deleting %s version %q: %v", p, aws.StringValue(version.VersionId), err)
}
}

return nil
Expand Down