Amazon S3 Bucket Explorer

• Amazon S3 Bucket Explorer
• Amazon S3 Bucket Explorer 2.0
• S3 Bucket Explorer
• Amazon S3 Bucket Explorer Specs
• Download S3 Explorer

Amazon S3 Bucket Explorer. A web based folder/bucket explorer in ReactJS framework consuming S3 REST API. Specifications: On Page Load: Only folders/files which are present at the root of bucket are visible. If folder is clicked: It Expands if it is in collapsed state. Collapses if it is in expanded state. Find helpful customer reviews and review ratings for Bucket Explorer Software for Amazon S3 - Simple Storage Service, with DVD at Amazon.com. Read honest and unbiased product reviews from our users.

I have been working hard to improve this blog loading speed and overall site performance. one of the method adopted was to move all my blog media files majority of which are images to my Amazon S3 bucket. which in turn are distributed by Cloudflare CDN.

Afterward, i did some testing using Google PageSpeed Insights and GTmetrix Website Speed and Performance Optimization tools. Performance score for the former was ranging from 68 – 80 and that of the latter was page speed of 84% and YSlow 64%

One of the contributing factor to this low performance score was due to the absent of expiry date or a maximum age in the HTTP headers for my static resources especially images served from Amazon S3. Unlike the images served by my main web host, the images over at Amazon weren't getting served with any expires headers at all. Below are screenshot of Google PageSpeed Insights and GTmetrix Performance result spotting caching absent.

What is browser caching?

Every time a browser loads a webpage it has to download all the web files to properly display the page. This includes all the HTML, CSS, JavaScript and images. Browser caching can help by storing some of these files locally in the user's browser. Their first visit to your site will take the same time to load, however when that user revisits your website, refreshes the page, or even moves to a different page of your site, they already have some of the files they need locally.
This means the amount of data the user's browser has to download is less, and fewer requests need to be made to your server. The result? Decreased page load times.

Why cache Amazon S3 images

• Reduction in Amazon S3 Bill – One of the criteria use by Amazon s3 in calculating your bill is the number of GET request to your files. though it relatively cheap, if you run a high traffic site, those little cent per GET request can quickly accumulate. setting caching will reduce the number of request to images in your s3 bucket which in turn significantly reduce your bill.
• Fast website loading speed – Aside the reduction in bill, it also significantly increase website loading speed.
  Say you set cache control Header for s3 images to expire in 3 months time. when a user visit your website, the static images are downloaded from Amazon S3 servers and get saved inside his browser's cache. if that person is to visit your site, rather than download the images again, the browser uses the images it has cached.

To fully understand the concept of caching, i suggest your read Caching Tutorial by Mark Nottingham.

Setting cache for Amazon s3 Images

Reading Google pageSpeed insights on leverage browser caching, Google said they prefer Expires (for example 'Expires, Fri, 30 Oct 1998 14:19:41 GMT') over Cache-Control: max-age (for example 'Cache-Control: max-age=3600') because it is more widely supported.

• The use of Amazon s3 third-party applications

  After some frantic Googling, i stumbled on Bucket Explorer with a comprehensive tutorial on How to Set Cache Control Header for Amazon S3 Object. though it has lots of cool s3 feature, i was put off because it wasn't free. i wasn't ready to spend $69.99. this led me to searching for a freeware and eventually settled for CloudBerry Explorer Freeware.

  How to set s3 HTTP headers via CloudBerry Explorer

  1. Grab a copy of CloudBerry Explorer, install and open it.
  2. Next is adding your Amazon S3 account. click on File ->Amazon S3, Add a display name of your choice followed by your s3 Access and Secrete keys.
     
  3. Say you want to set cache for all images in a given bucket or folder, navigate to the bucket, right-click and select Set HTTP Headers
     
  4. Click on the Add button, in Http Header, select either Cache-Control or Expires. mind you, Google recommend we use latter, though the former is OK.

     if you chose Cache-Control, specify in seconds, when the cache should last. say we want the images to be cache for one month (60x60x24x30) which is 2592000 secs.
     

     if you chose Expires, specify the Date and time you want the cache to expire.

  5. Click the OK button. after which, wait for a moment for the Cache header to be processed.
  6. To confirm if it works, navigate to any image in your bucket, repeat step 3 to view the image HTTP header or use this nifty HTTP header checker tool.
     

Aside, Bucket and Cloudberry explorer, there is S3 Browser a freeware Windows client for Amazon S3 and Amazon CloudFront.

• Another option to consider is the use of this PHP script by Lalit to upload and set HTTP headers.

If you have any contribution as regard setting up HTTP headers for Amazon s3 static resources, or you have a question to add, feel free to drop a comment, we'll love to hear from you.

• Update April 15, 2019:
  Include a enhancement to the UDF to work with HTTP and HTTPS simultaneously;
  Include a enhancement to the UDF to work with URL encoded and NOT encoded (it's necessary comment/uncomment the line).

Today I'll explain step-by-step how to calculate the signature to authenticate and download a file from the Amazon S3 Bucket service without third-party adapters.

Request

In summary this interface receive download URL, Bucket, AccessKeyID, SecretAccessKey, Token and AWSRegion, a mapping calculate the signature with this information and sent to REST Adapter, the signature and anothers parameters are insert in HTTP header.

Some information for calculate the signature are provide another service, this post explain only how to calculate, but is possible implemented enhancements, for example, create a rest/soap lookup to get a Token and SecretAccessKey.

Response

The response is a file, and the REST Adapter don't work with format different of XML or JSON, then you will need convert the file to binary, and this content are insert in a tag of XML. For this conversion I recommend a module adapter FormatConversionBean developed by @engswee.yeoh

Request mapping

For the request mapping you need create a two structures, one for inbound and another for outbound.

Inbound

Outbound

After create the structures for the request mapping (data type, message type, etc), you need create a message mapping.

Now you need to map the fields, pay attention to the next steps for configurations the roles.

Roles for Message Mapping

• Fields XAmzSecurityTokenand Urlare mapped directly….

• Field XAmzSha256 is mapped with a constant value e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 (this string is a hash of a null value)

• Field XAmzDate is mapped with a CurrentDate (format yyyyMMdd'T'HHmmss'Z') function…

• Field ContentType is mapped with a constant value application/x-www-form-urlencoded…

• Field Host is mapped with a UDF or ConstantValue.

The Host is a result of concatenation of the Bucket+'.s3.amazonaws.com',
so you can use a ConstantValue (eu01-s3-store.s3.amazonaws.com for example), which receives the bucket and returns the Host

• Field Authorizathion is mapped with a UDF.

In field Authorization you have insert the signature calculated with the UDF below.

You also need to create some methods, which will be used by UDF in signing.

and import the packages…

Amazon S3 Bucket Explorer

After developed the UDF, its necessary configure with the inbound values.

Note: the format of CurrentDate is yyyyMMdd'T'HHmmss'Z'.

Now save and Activate the Request mapping.

Response mapping

Amazon S3 Bucket Explorer 2.0

The response mapping it's simple and not necessary many explanation.

Configure the interface normally …

After created Request/Response Mapping, build the Operation Mapping and a Integrated Configuration normally. The Communication Channel can be of any type that is synchronous, but the Receiver must be to type rest and configured as below.

Receiver Communication Channel

Now you need configure the Receiver Channel, for this the values generates in request message mapping are storage in variables, and this variables are used in the communication channel.

Now the variables storage are used in the HTTP Header, here you configure how the canonical request is create.

It's necessary configure the REST Operation, for this case the operation is GET.

And finally configure the module adapter FormatConversionBean to converte the file in b64string.

IMPORTANTE: The module adapter FormatConversionBean isn't standard, and you need deploy if you have not already, for more information and download of module you can access here.

Save and active all objects, now we going test!

Fill in all the fields correctly in the interface and call the created service, the response should be the file in b64string format.

If you analyze the log of request messages, the parameters are populated in the HTTP header and communication has succeeded (HTTP 200)

and the response (the file) is converted to b64 string.

S3 Bucket Explorer

That's all! I hope I have collaborated and I am waiting for your feedback on this post.

Amazon S3 Bucket Explorer Specs

References

How to Calculate AWS Signature Version 4
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

Download S3 Explorer

Module Adapter FormatConversionBean
https://blogs.sap.com/2015/03/25/formatconversionbean-one-bean-to-rule-them-all/

PI REST Adapter – Define custom http header elements
https://blogs.sap.com/2015/04/14/pi-rest-adapter-define-custom-http-header-elements/