Searching Amazon S3 files using the aws cli

I recently had a need to search Amazon S3 files and was very excited to discover how powerful the aws cli tool was when combined with the jmes query language

The first step was to install the aws cli tool. There is a graphic installer available for macOS, but I installed via the command line:

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

I checked to make sure it had worked, and then removed the installer:

aws --version
rm AWSCLIV2.pkg

The second step was configuration – just a matter of issuing a command and then providing response to configuration questions (key, secret, region, etc).

aws configure

Once that was done, it was very easy to query files. Here are a few basic examples:

Search S3 files by owner

aws s3api list-objects --bucket your-bucket --query "Contents[?Owner.DisplayName == 'someone']"

Search S3 files not by owner

aws s3api list-objects --bucket your-bucket --query "Contents[?Owner.DisplayName != 'someoneelse']"

Search S3 files by key

aws s3api list-objects --bucket your-bucket --query "Contents[?contains(Key, 'path/to/your/file')]"

Search S3 files by date

aws s3api list-objects --bucket your-bucket --query "Contents[?LastModified > '2020-03-26']"

Keep in mind that, as far as I know, the searching is happening on your end, so the requests can be a bit slow, but still a heck of lot faster and more effective than manually combing through files in Transmit which is what I wondered if I might have to do before checking to see what other options were available!

The cli has options that allow for various ouput formats, coloured output, and so on.

Happy searching!