Eucalyptus Walrus/Amazon S3 SOAP signature is failing

Question

I have been learning how to use Amazon S3 API by using the open source Eucalyptus. So far I have been able to successfully use REST, but now I would also like to use SOAP. I seem to be having trouble generating the correct signature for my request. The service is giving me a 403 Forbidden error:

Traceback (most recent call last):
  File "soap.py", line 31, in <module>
    r = w.download_file('mybucket', 'test.txt')
  File "soap.py", line 27, in download_file
    r = self.client.service.ListAllMyBuckets(self.access_key, timestamp, signature)
  File "/usr/lib/python2.6/site-packages/suds/client.py", line 521, in __call__
    return client.invoke(args, kwargs)
  File "/usr/lib/python2.6/site-packages/suds/client.py", line 581, in invoke
    result = self.send(soapenv)
  File "/usr/lib/python2.6/site-packages/suds/client.py", line 619, in send
    description=tostr(e), original_soapenv=original_soapenv)
  File "/usr/lib/python2.6/site-packages/suds/client.py", line 677, in process_reply
    raise Exception((status, description))
Exception: (403, u'Forbidden')

My code is in Python 2 and uses the SUDS-Jurko library for sending SOAP requests:

from suds.client import Client

class WalrusSoap:
    wsdl_url = 'https://s3.amazonaws.com/doc/2006-03-01/AmazonS3.wsdl'
    server_url = 'https://localhost:8773/services/Walrus'

    def __init__(self, access_key, secret_key):
        self.access_key = access_key
        self.secret_key = secret_key
        self.client = Client(self.wsdl_url)
        self.client.wsdl.services[0].setlocation(self.server_url)
        #print self.client

    def create_signature(self, operation, timestamp):
        import base64, hmac, hashlib
        h = hashlib.sha1(self.secret_key)
        h.update("AmazonS3" + operation + timestamp)
        #h = hmac.new(key=self.secret_key, msg="AmazonS3" + operation + timestamp, digestmod=hashlib.sha1)
        return base64.encodestring(h.digest()).strip()

    def download_file(self, bucket, filename):
        from time import gmtime, strftime
        timestamp = strftime('%Y-%m-%dT%H:%M:%S.001Z', gmtime())
        print(timestamp)
        signature = self.create_signature('ListAllMyBuckets', timestamp)
        print(signature)
        r = self.client.service.ListAllMyBuckets(self.access_key, timestamp, signature)
        return r

w = WalrusSoap(access_key='MOBSE7FNS6OC5NYC75PG8', secret_key='yxYZmSLCg5Xw6rQVgoIuVLMAx3hZRlxDc0VOJqox')
r = w.download_file('mybucket', 'test.txt')
print(r)

I changed the server endpoint, because otherwise the WSDL points to the regular S3 servers at Amazon. I also have two different ways of creating the signature in my create_signature function. I was swapping between one and the other by simply commenting out the second one. Neither of the two seem to work. My question is what am I doing wrong?

SOAP Authentication: http://docs.aws.amazon.com/AmazonS3/latest/dev/SOAPAuthentication.html

SUDS-Jurko Documentation: https://bitbucket.org/jurko/suds/overview

Edit: I realized I forgot to include an example of what timestamp and signature is printed for debugging purposes.

2014-12-05T00:27:41.001Z
0h8vxE2+k10tetXZQJxXNnNUjjw=

Edit 2: Also, I know that the download_file function does not download a file :) I am still in testing/debug phase

Edit 3: I am aware that REST is better to use, at least according to Amazon. (Personally I think REST is better also.) I am also already aware that SOAP is deprecated by Amazon. However I would like to go down this path anyways, so please do me a favor and do not waste my time with links to the deprecation. I assure you that while writing this SOAP code, I was already well aware of the deprecation. In fact one of the links I posted has the deprecation notice printed at the top of its page. However, if you have evidence showing that Walrus completely ditches SOAP or that they stopped working on the SOAP portion, I would like to see something like that. But please do not tell me Amazon has deprecated SOAP.

Answer 1

Eucalyptus does support SOAP (see ZachH's comment about deprecation): https://www.eucalyptus.com/docs/eucalyptus/4.0.2/schemas/aws-apis/s3/index.html

I decided to scrap using python, and I produced some working code with C# instead. Turns out C# has better SOAP libraries.

Answer 2

Eucalyptus does not support SOAP for S3 as of Eucalyptus version 4.0.0.

If you are using an older version of Eucalyptus (pre 4.0), then SOAP should work. Note, however, that the wsdl provided by S3 is not necessarily current or accurate for even their own service. S3 is notorious for changing the API without version or wsdl bumps, particularly since they stopped updating the SOAP API. So, there are likely responses from Walrus that do not conform to the published WSDL because our XML was updated based on the responses we see from S3 (via REST) and the SOAP and REST responses diverged. The signatures should be compatible, however, so worst case you would see 500 or 400 errors, but not 403 responses.

My recommendation is if you really want to learn the S3 SOAP API, you'll have to use S3 proper. The S3 SOAP support in Eucalyptus is there pre-4.0 but may not be compliant with the current state of S3 SOAP--we stopped testing against the S3 SOAP API directly when the AWS SDKs stopped using SOAP in favor of better REST support since that was the API moving forward.

Answer 3

The S3 SOAP API does not support "new" features so the REST API should be used where possible:

http://docs.aws.amazon.com/AmazonS3/latest/dev/SOAPAPI3.html

https://forums.aws.amazon.com/message.jspa?messageID=77821

IIRC recent versions of Eucalyptus do not support SOAP with S3.

That said, the signature looks good to me so I would check if the client/service hosts have the correct time, if there is a difference of more than 15 minutes authentication would fail.

You could also check the cloud-error.log on the Walrus service host as there may be more details on the failure there.