Amazon S3 REST Wrapper

Posted At : September 7, 2006 4:17 PM | Posted By : Joe Danziger
Related Categories: ColdFusion,Code | 16016 Views

As my first foray into open-source code, I'm releasing a wrapper for interacting with Amazon's Simple Storage Service (S3) via REST. The wrapper is packaged as a CFC and has the following methods:

init(accessKeyID, secretAccessKey) - initialize CFC (both parameters required).
getBuckets() - List all buckets.
putBucket(bucketName) - create a new bucket.
getBucket(bucketName, prefix, marker, maxKeys) - get contents of a bucket (prefix is optional and matches on the beginning of a key, marker is optional and results start from there, maxKeys is optional and restricts the number of objects returned).
deleteBucket(bucketName) - delete a bucket (bucket must be empty).
putObject(bucketName, fileKey, contentType, HTTPtimeout) - puts an object into a bucket (HTTPtimeout is in seconds).
getObject(bucketName, fileKey, minutesValid) - get link to an object (minutesValid is optional and defaults to 60).
deleteObject(bucketName, fileKey) - delete an object from a bucket.

A simple test script is included which demonstrates the use of the CFC. You must insert your Amazon S3 access keys in the first 2 lines in s3test.cfm, then just pull it up in a browser.

This is an initial release. Future plans include support for Access Control Lists. If you need something else added, let me know.

This script should run on both ColdFusion MX 6 and 7, let me know if you run into any problems.

The current version is 1.1 and you can visit the project page and download here.

Comments (17) |

Print |

Send |

del.icio.us |

Digg It! |

Linking Blogs

Related Blog Entries

Amazon S3 & Smartermail REST Wrappers v1.2 (October 4, 2006)

Comments (Comment Moderation is enabled. Your comment will not appear until approved.)

[Add Comment]

Great script Joe. Just what I needed to automate my uploads to Amazon S3 from the site. Keep up the fantastic work.

# Posted By Steve | 5/11/07 6:11 PM

Thanks for this amazing script!! I was just wondering if the custom tag cf_hmac could be more updated.. i noticed that it's code from 2003 and all these algo thingies are in MX6 and 7 aren't they?? I have no clue how that tag works but maybe someone could update it.. and make it into a CFC??
Thanks again for this amazing script.. i'm stoked at the moment!!

By the way.. you guys did know you can also reach your files thru the following construction... http://bucketnamet.s3.amazonaws.com/name-of-the-fi...

# Posted By Tjarko Rikkerink | 5/26/07 7:32 AM

Hey Everyone.
While recently working on a project using Amazon S3, I developed the need to be able to edit the ACL of the buckets I was adding to my S3 account.
Following on from the great work by Joe, I have added ACL compatibility to the putBucket method.
For those of you who are interested, you can download the edited version here - http://www.stevehicksonline.com/2007/06/04/amazon-...
Steve

# Posted By Steve | 6/4/07 5:27 PM

I followed the readme and got everything installed, but when I hit the test page I get: An error occured while Searching an XML document.
Prefix must resolve to a namespace:

Has anyone ever seen this before? Id really like to start using s3

# Posted By anthony | 6/7/07 2:29 AM

Nevermind, I hadnt clicked the link in my email to verify my account, working great now!

As for these "folders", anyone know if you somehow have subfolders within a folder?

# Posted By anthony | 6/7/07 2:50 AM

It would be nice to have full support to put to and get from "subdirectories" see: http://developer.amazonwebservices.com/connect/mes...

# Posted By anthony | 6/7/07 4:06 AM

Hi, I actually changed the complete use of the md5 custom tags and such.. you can read that on my blog. With ColdFusion MX 7 you don't need those customtags anymore to hash the keys.

# Posted By Tjarko Rikkerink | 6/21/07 4:16 PM

Looks interesting. Thanks for sharing your work. My only suggestion would be to add some error handling for when authorization fails. It currently just returns an xml search error, which isn't very intuitive.

# Posted By Chris S | 8/30/07 9:22 AM

As a simple test, I just tried storing s3.zip in a bucket. Uploading seemed to work fine, but when I tried to download it using the link generated by s3test.cfm I get the error "The request signature we calculated does not match the signature you provided. Check your key and signing method."

# Posted By Chris S | 8/30/07 9:34 AM

I'm getting that error about half the time. It happens for both ascii and binary files. I don't see a pattern in the errors, since I'll get the error for one link, then I'll refresh the link and it'll work, then a minute later it'll be broken again. I'd really like to start using S3 but if it's only going to work 50% of the time, it's not going to be worth it. Any thoughts?

# Posted By Chris S | 8/30/07 10:20 AM

Double-checked R's solution, and found that that bug was still not fixed in the posted code.

@Joe, could you please update the code? You said you fixed it in 1.2, but the only link I can find is to the older version.

# Posted By Chris S | 8/30/07 11:44 AM

@Chris S -- not sure if it's the same issue, but I've found that URLEncoding the signature in the getObject function eliminated the problem of links that only work half the time. I changed this line:

<cfset timedAmazonLink = "http://s3.amazonaws.com/#arguments.bucketName#/#ar...(signature)#">
Hope that helps...

# Posted By Jack Welde | 9/23/07 1:47 AM

I've also found that uploading is extremely slow, that this wrapper is next to useless for large files. For example, uploading a 3MB image consistently times out after 5 minutes. I installed Jungledisk, an S3 wrapper for the desktop, and was able to upload the exact same file in 38 seconds.

# Posted By Chris S | 11/8/07 11:07 AM

Everything works perfectly except I cant delete a bucket....I am getting a "403 Forbidden" . I can put an object into the bucket view it and delete it. I have tried URLEncoding the Hex2Bin as well.

Here is the function:
<cffunction name="deleteBucket" access="public" output="false" returntype="string"
            description="Deletes a bucket.">
      <cfargument name="bucketName" type="string" required="yes">

      <cfset var signature = "">
      <cfset var dateTimeString = GetHTTPTimeString(Now())>

      
      <cfset var cs = "DELETE\n\n\n#dateTimeString#\n/#arguments.bucketName#">

      
      <cfset var fixedData = replace(cs,"\n","#chr(10)#","all")>

      
      <cf_hmac hash_function="sha1" data="#fixedData#" key="#variables.secretAccessKey#">

      
      <cfset signature = URLEncodedFormat(ToBase64(Hex2Bin("#digest#")))>


      
      <cfhttp method="DELETE" url="http://s3.amazonaws.com/#arguments.bucketName#&quo...; charset="utf-8">
         <cfhttpparam type="header" name="Date" value="#dateTimeString#">
         <cfhttpparam type="header" name="Authorization" value="AWS #variables.accessKeyId#:#signature#">
      </cfhttp>

      <cfreturn cfhttp.header>
   </cffunction>

Any suggestions welcome,

Andrew.

# Posted By Andrew Grosset | 12/2/07 11:16 PM

I should have added that I can delete the bucket using "S3Fox" (firefox addition) and "S3Backup" so its not vital that I use the S3 Rest Wrapper - but it bugs me that I cant!

# Posted By Andrew Grosset | 12/4/07 1:21 AM

Here you can take HMAC SHA1 function (quite simple) to be used with S3 library:
http://www.coldfusiondeveloper.com.au/go/blog/2008...

and then:
<cfset signature = ToBase64(HMAC_SHA1(variables.secretAccessKey,fixedData)) />

# Posted By Dmitry Yakhnov | 2/10/08 6:16 AM

I've been hacking away on the CFC and I have fixed numerous bugs and added support
for compression, text uploads, folders, logging, using a proxies (aka fidler2) and a lot of other little things here and there

I also changed the behaviour to return the cfhttp result so it can be properly inspected
for the correct status codes and a dev level response status code checker

# Posted By Zac Spitzer | 2/26/09 12:18 AM

[Add Comment]