Suppose you have an amazon s3 bucket that has around hundreds or thousands of filenames in it. So what is the easiest way to get a text file that contains lists of all the filenames in that amazon s3 bucket? Using AWS Command Line Interface (CLI) AWS have their own Command Line Tools. You can list all the files, in the aws s3 bucket using the command. Aws s3 ls path/to/file. List files in Amazon s3 bucket on wordpress page.
Active2 years, 1 month ago
$begingroup$I'm using the Amazon S3 Java SDK to fetch a list of files in a (simulated) sub-folder. This code is rather standard (
AWSConfiguration
is a class that contains a bunch of account specific values):Now this list will include objects like
/images/cars/default.png
as well as /images/cars/ford/Default.png
(because they both contain the same prefix). To list only the objects that are directly inside the /images/cars/
'folder' I have the following function (in a class called S3Asset)This looks at the full key for any
/
after the prefix as a clue that it is inside a sub-folder. This lets me iterate over the objects with the following code (I'm trimming the prefix for clarity):As near as I can tell this is the cleanest way to do this but it has one characteristic that I don't like. If I'm looking a root level folder I'm requesting the names of all files in all sub-folders only to iterate over them and learn that there is only one object in the actual root level folder. I've considered associating a key with the value being the full path of the folder, which would allow me to request objects with a predictable key instead of the prefix, but the major downside to this is that the key would have to be generated in code and therefor assets uploaded directly in to the S3 Bucket (through the management console) would not have this key. Anyone have a better idea?
palacsint29.3k99 gold badges7171 silver badges154154 bronze badges
Jason SperskeJason Sperske
$endgroup$3 Answers
$begingroup$In the
ListObjectsRequest
javadoc there is a method called withDelimiter(String delimiter)
. Adding .withDelimiter('/')
after the .withPrefix(prefix)
call then you will receive only a list of objects at the same folder level as the prefix (avoiding the need to filter the returned ObjectListing
after the list was sent over the wire).Some notes about the code: Ets 2 patch 1 3 keygen musica.
1, I'd extract out to a local variable for the
ListObjectsRequest
instance:It's easier to read.
2,
root_size
should be rootSize
. (Regarding to the Java Coding Conventions.)3, I would use String.contains instead of
indexOf
. It's more meaningful, easier to read since you don't have to use the -1
magic number.4, In the last snippet I'd create a local variable for the
key
:5, Furthermore, I'd move the
length
call inside the helper method:Note the input check. (Effective Java, Second Edition, Item 38: Check parameters for validity)
The multiple calls of
length
could look redundant and slow but premature optimization is not a good thing (see Effective Java, Second Edition, Item 55: Optimize judiciously). If you check the source of java.lang.String
, you will find this:String
is immutable, so it's easy to cache its length and JDK does it for you.29.3k99 gold badges7171 silver badges154154 bronze badges
$endgroup$$begingroup$That's an old question, but I just've find out that with a new version of amazonaws '1.9.22' you can call 'getCommonPrefixes' to get a list of all available prefixes. You need to use delimiter either.
So, for a structure like 'prefix/aaa/111' it will return 'prefix/aaa' level items.
S3 Bucket List
UPDATE:
And a java version (original one uses scala)
cyrillkcyrillk
$endgroup$$begingroup$I got nothing returned when I used
alecxewithDelimiter('/')
. It turned out that in order to use delimiter, I also needs to end my prefix with /
. In other words, withPrefix('prefix/abc')
will get all objects under prefix/abc
, but for delimiter to work, use withPrefix('prefix/abc/')
instead.15.5k55 gold badges4040 silver badges8383 bronze badges
BarryBarry
$endgroup$protected by Jamal♦Jul 28 '17 at 4:10
Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).
Would you like to answer one of these unanswered questions instead?
Would you like to answer one of these unanswered questions instead?