I've implemented Amazon S3 in the past without a problem. Documentation is a breeze to get through, making implementation super-easy. However, having recently moved from AWS to SoftLayer for hosting, I decided to switch Object Storage from S3 to Softlayer Object Storage.
Softlayer makes it incredibly easy to set up Object Storage on a pay-as-you-go system, just like S3. And you get a reduced rate bandwidth to Object Storage from within a SoftLayer Hosted instance. Cool.
Now to the implementation. When you start poking around on the Object Storage documentation at SoftLayers website, you quickly find yourself at the OpenStack documentation for OpenStack v1.0. No Problem here, but make sure you note that SoftLayer is still using the v1.0 OpenStack API. Now, Like most people might, I wanted to get right to it and upload a file to Softlayer system. In my case, I'm running a web app with a NodeJS server. As such, I find the SDK's page on the OpenStack Documentation and am pointed to a sweet Github project pkgcloud.
Pkgcloud makes it really easy to upload a file from a multi-part form post request in. It goes something like this:
In the above code, username, password, authUrl are all provided under the "Account Credentials" area on SoftLayer's website. I am simply pulling them from a config file. However, you should note the "version: 1" and "useServiceCatalog: false". Those are required to tell pkgcloud that you want to use OpenStack v1.0 API (required by SoftLayer). They currently default to v2.0, and don't officially support v1.0 at all right now. Therefore, you will want to use my forked copy of pkgcloud, until this pull request, #330, is merged into the master branch. (EDIT 10/7/2014 - Pull Request #330 was merged into master, you can now reference pkgcloud Github repo, but NPM has no yet been updated. Look for milestone 1.0 to be released on NPM before installing)
In Addition, the "Stream" variable is simply my "request" object passed from my RESTful API Endpoint. Container is the Object Storage container I want to place the file into, and the resourceKey is the full populated path to the file. E.g. "path/to/my/file.txt". Voila. Once this is done, file.txt is now at /container/path/to/my/.
That is easy (thanks pkgcloud). But now we get to the meat of the issue. Your web-based client wants to now retrieve the file that was just uploaded. To do this, you use secure temporary URL's. These are URL's that must be generated on the server and include some nice features to ensure your files aren't exposed to the public. Lots of systems do this in a similar fashion, like S3.
With OpenStack Swift though, it's not that easy. First you need to set some temporary URL Keys on your OpenStack account, before you can generate the Temporary URL, and to do this, you need some info that SoftLayer doesn't expose through their website. Specifically, you need your X-Auth-Token and your X-Storage-Url. Luckily, these are easy to retrieve, something like:
In the above, I make a GET request to the Softlayer Authentication URL provided by SoftLayer in the Account Credentials for the storage account, it would be something like https://dal05.objectstorage.softlayer.net/auth/v1.0/. I set the 'X-Auth-User' and 'X-Auth-Key' headers to the credentials provided by Softlayer.
If all goes well, I get a 200 response and can retrieve by X-Auth-Token and X-Storage-Url from the response headers, as shown above.
Now that I have these, I can move forward setting the Temporary URL Keys. These are completely arbitrary string values you can make up and set to anything you like, I believe of any length you like. I use a random series of 30 alphanumeric charaters. As well, there are two of them, allowing for key rotation over time, whereby one key is always valid while the other key is getting changed.
To set these keys, you would do:
Softlayer makes it incredibly easy to set up Object Storage on a pay-as-you-go system, just like S3. And you get a reduced rate bandwidth to Object Storage from within a SoftLayer Hosted instance. Cool.
Now to the implementation. When you start poking around on the Object Storage documentation at SoftLayers website, you quickly find yourself at the OpenStack documentation for OpenStack v1.0. No Problem here, but make sure you note that SoftLayer is still using the v1.0 OpenStack API. Now, Like most people might, I wanted to get right to it and upload a file to Softlayer system. In my case, I'm running a web app with a NodeJS server. As such, I find the SDK's page on the OpenStack Documentation and am pointed to a sweet Github project pkgcloud.
Pkgcloud makes it really easy to upload a file from a multi-part form post request in. It goes something like this:
Loading ....
In the above code, username, password, authUrl are all provided under the "Account Credentials" area on SoftLayer's website. I am simply pulling them from a config file. However, you should note the "version: 1" and "useServiceCatalog: false". Those are required to tell pkgcloud that you want to use OpenStack v1.0 API (required by SoftLayer). They currently default to v2.0, and don't officially support v1.0 at all right now. Therefore, you will want to use my forked copy of pkgcloud, until this pull request, #330, is merged into the master branch. (EDIT 10/7/2014 - Pull Request #330 was merged into master, you can now reference pkgcloud Github repo, but NPM has no yet been updated. Look for milestone 1.0 to be released on NPM before installing)
In Addition, the "Stream" variable is simply my "request" object passed from my RESTful API Endpoint. Container is the Object Storage container I want to place the file into, and the resourceKey is the full populated path to the file. E.g. "path/to/my/file.txt". Voila. Once this is done, file.txt is now at /container/path/to/my/.
That is easy (thanks pkgcloud). But now we get to the meat of the issue. Your web-based client wants to now retrieve the file that was just uploaded. To do this, you use secure temporary URL's. These are URL's that must be generated on the server and include some nice features to ensure your files aren't exposed to the public. Lots of systems do this in a similar fashion, like S3.
With OpenStack Swift though, it's not that easy. First you need to set some temporary URL Keys on your OpenStack account, before you can generate the Temporary URL, and to do this, you need some info that SoftLayer doesn't expose through their website. Specifically, you need your X-Auth-Token and your X-Storage-Url. Luckily, these are easy to retrieve, something like:
Loading ....
In the above, I make a GET request to the Softlayer Authentication URL provided by SoftLayer in the Account Credentials for the storage account, it would be something like https://dal05.objectstorage.softlayer.net/auth/v1.0/. I set the 'X-Auth-User' and 'X-Auth-Key' headers to the credentials provided by Softlayer.
If all goes well, I get a 200 response and can retrieve by X-Auth-Token and X-Storage-Url from the response headers, as shown above.
Now that I have these, I can move forward setting the Temporary URL Keys. These are completely arbitrary string values you can make up and set to anything you like, I believe of any length you like. I use a random series of 30 alphanumeric charaters. As well, there are two of them, allowing for key rotation over time, whereby one key is always valid while the other key is getting changed.
To set these keys, you would do:
Loading ....
If you get a 204 statusCode, you're good to go.
Now that we've set our keys, we can finally create our temporary URL for the file in the container and at the path (resourceKey) you need. Here is an example method:
Loading ...
The above code is a little more complex, because I'm checking to determine if I already have my Storage URL required to make the call. If I don't, I request it from SoftLayer and then request the signed URL again.
Hopefully that helps anyone who is new to SoftLayer object storage, whether using NodeJS or any other language.
Oh, Finally a good "getting started" tutorial!
ReplyDeleteThank you for this tutorial. Why don't you use the download-functionality of pkgcloud?
ReplyDeleteHi Simon,
ReplyDeleteIt's been a while since I wrote this and I have since moved back to AWS. However, the issue I was having in the past was generation of temporary (expiring) URLs, such that a user could not share the download URL with someone else and have it available long term. In this case, SoftLayer actually required the setting of this this header property at the account level: 'X-Account-Meta-Temp-Url-Key', and only then would Softlayer allow the generation of temporary URLs using that temp key in the HMAC hash: var hash = crypto.createHmac('sha1', tempUrlKey).update(body).digest('hex');
Hope that helps to clarify things. There may certainly be a better option available now!
Thanks.
Chris
I've been trying to use this method, but have had issues when trying to get the signature with object names that have non-ascii characters.
Deletevar hash = crypto.createHmac('sha1', tempUrlKey).update(body).digest('hex');
Do you have any experience with this?