World Line

AWS Key Server

with 15 comments

It’s the last day of 2010, and as an AWS fan there’s a lot to reflect on from the last year. There’s been several new SDKs, a couple of new services, more than twice as many consoles as there were this time last year, and so many major improvements to existing services that if I linked them all, every word in this post would be blue. It’s been really exciting, and nobody could have predicted half of this stuff in 2009.

With that in mind, I’m still going to go ahead and make a bold prediction: I believe, when all is said and done, that the most important release of 2011 will end up being the AWS Identity and Access Management (IAM) service, which is currently still in “Preview Beta”. I don’t know what else 2011 will bring, but I’d be surprised to see anything top this.

This needs justification, of course, since the beta has been proceeding with extremely little fanfare and very few of the AWS community have been talking about it. I think this is because it hasn’t been interpreted right yet; people (including Amazon’s own documentation[2]) still only refer to its ability to split up developers’ access to the account, so that testers can only access test instances, database maintainers can only access RDS operations, etc. This is all well and good, but by itself is not the reason I’m excited about IAM.

To explain, I have to bring up the pair of Mobile SDKs (Android and iOS) released earlier this month. They have an inherent critical flaw: credential management. If you want to write a mobile application that uses AWS services (but not on behalf of the user), you need to, at some point, have your credentials in memory on the device. You can try to obfuscate them, or minimize the amount of time it happens using various tricks like downloading them to SQLCipher-encrypted SQLite blocks, but at the end of the day, the request has to be signed on the device. This presents a problem because a skilled and determined attacker can run your APK in a simulator and read out your credentials no matter how well you’ve obfuscated them. You can try proxying all your requests through a server that does the signing, but that defeats the purpose of using AWS services in the first place — now you have another hop which can be a point of failure, latency, maintenance, which needs to be scaled, etc. I know this is a problem people are actually concerned about, because I’ve spent time answering questions about it in several places.

IAM solves this problem beautifully. It allows you to programatically create low-privileged user accounts on demand for every user of your mobile application, thereby resolving the problem of storing credentials by making the credentials worthless! The advantages are many:

  • Prevents abusing the account in any way other than what could have been done from the application anyway.
  • Allows you to control access through registration, paywalls, etc.
  • Allows you to individually disable abusive accounts.
  • Immediately provides fine-grained permissions control (like only allowing DeleteObject requests on S3 objects created by the same sub-account — the same functionality on a single account would require complicated bookkeeping somewhere else other than S3)
  • Provides a crude form of analytics. You’ll at least have an upper bound on the number of unique users, and roughly how heavily each one is using it.

To demonstrate how neatly IAM works, I’ve created a proof-of-concept awskeyserver project on GitHub. It’s a Google App Engine service that creates on-demand IAM accounts through a RESTful interface. By default it gives them out to everyone, so hitting http://awskeyserver.appspot.com/create_user?group=Users will return something like AKIAIFY7K3N4Y6OE2DLQ:dGV6MHs3pMjRkT4RZmwnOZndOJG75FyOjpeQYVFA, which are the access credentials for an account in the Users group. In this case, the Users group has precisely no permissions, so it’s no big deal that I just posted those on my public blog (a few months ago that would’ve been a code-red disaster!). A mobile app can cache that and use it for the lifetime of the installation.

But that’s not all! If you’re noticing way too many requests for accounts, you can just edit permissions.py and add a CaptchaValidator() policy handler to the Users group. Once you do that, the previous request will instead return something like 03AHJ_VuueXyjZt-oM2Bf1K3c8rfsb_NHnjRQPLKDL0Vc1GhYs4LFmcuEYdBTfpGfYJbtRVwNL-OXeuAyApuHqrZ_J90i3qLJDHKepDFGIGTGQR8f1sRQpIchSDowHQHZczdbeLEdJPmR5Paiq_XjwzcJMMrZ4d1UHXQ, which is a reCAPTCHA challenge id. If you look that URL up in reCAPTCHA it’ll provide you with a captcha image, whose response has to be filled in (along with the challenge id) to the recaptcha_challenge_field and recaptcha_response_field query parameters to awskeyserver. Only then will it cough up the credentials, giving you a way to stem the flow. Captcha’s are one simple way of doing that, but I’ll be adding more context-aware validators to awskeyserver as time goes on.

To show how this works in practice, I’ve taken the sample AwsDemo application that comes with the AWS SDK for Android and modified it[1] to work without any local credentials at all. The modification only lives in one file, AwsDemo.java, which I’ve Gisted. It can handle both plain and captcha policies, and the returned credentials are limited by whatever permissions you set on the server side. In the example below, the group has the following policy file:

{
  "Statement":[{
    "Effect":"Allow",
    "Action":["sdb:ListDomains"],
    "Resource":"*"
  }]
}

and awskeyserver has the policy:

policy = { "Users" : CaptchaValidator() }

So as you’ll see in the screenshots, listing domains works without a problem, but trying to access them results in an error:

Captcha-enabled key server challenge

Once the credentials are there, they're for listing SDB domains only

Any other operation results in an exception.


This sort of behavior was literally impossible to achieve safely before IAM, and now it’s an afternoon’s worth of work. That’s the reason I think IAM will be so huge — up until now, consumers of AWS have always had to hide behind a server, performing requests on the behalf of clients. Amazon always warned you never to give out your secret key (under any circumstances!) but people still did to services like s3fm or Ylastic, because there was simply no other way for them to work. IAM for the first time opens up the possibility for letting clients make their own requests, directly, without having to go against their own AWS accounts to do so. I imagine Dropbox‘s backend architecture would look quite different if it had been designed in a post-IAM world. They could have saved so much by pushing some of the logic their servers perform to the client side and letting it upload to S3 directly.

So yeah. I think 2011 will be the year of client-side AWS applications, and it will be IAM that allows this to happen. If I’m wrong, that means something even cooler comes out of there next year, and I can’t wait to see what it is.


[1]: This is not how I think production code should look like, by the way. It’s very proof-of-concept.
[2]: Well, with one exception. The bottom 10% of this.

Written by Adrian Petrescu

December 31, 2010 at 5:52 am

Posted in Computer Science, Development

Tagged with ,

15 Responses

Subscribe to comments with RSS.

  1. First off, thanks for the Github project. It’s simply awesome! I’m dying to try this out for the backend of an iOS/Mac app I’m writing.

    The major hurdle I’m bumping into right now is that SimpleDB only lets you set policies for domains. This is a problem because I assume most developers, myself included, would like to store login credentials and other private user information in a single SimpleDB domain. As it is right now, you cannot set a group or user policy that will only affect a specific item in a domain. So if one item represents one application user, there is no possible way to prevent other IAM users from retrieving the data for all other application users.

    It’d be great to hear your thoughts on this, and if you have devised any kind of workaround or solution for this.

    Aaron Tait

    March 1, 2011 at 1:20 am

    • Hey Aaron, Thanks for the warm feedback🙂

      That’s a good use case you bring up; I haven’t thought about it until now, but my first instinct would be to encrypt each row with the owning user’s accessKey/secretKey pair before sending to SimpleDB. Since there’s no reason for the secret key to ever leave the user’s device, this should be secure enough; even if a malicious user did use their IAM credentials to dump a whole SDB domain, only their own rows would be legible.

      But I haven’t fully thought this out, so maybe there’s something obvious I’m missing.

      Cheers!
      -Adrian

      Adrian Petrescu

      March 1, 2011 at 1:31 am

      • This is a pretty good solution Adrian. All iOS devices have hardware support for AES encryption/decryption functions so this could be implemented on the app pretty easily and efficiently.

        However, the main use case that I’m developing for is that a user will want to be able to use their account across multiple devices (their iPhone, iPad, and Mac). This could be achievable if I required a user to enter their IAM credentials to authenticate on a new device, but it seems pretty counterintuitive to the standard username and password paradigm.

        The one workaround I can think of is having a backend (perhaps part of awskeyserver) that associates one IAS user to one application user who’s username and password is stored in SimpleDB. When an authorization request comes in (i.e. a user submits a username and password and we need to verify it’s integrity) the backend would lookup the username in IAS to find the private secret, get the corresponding user item from the SimpleDB domain, decrypt the item/row using the private secret, and then check the two passwords. If they match, the backend could return the IAS credentials to the application running on the device.

        So the next problem for me is that I don’t know Python and and adding this on top of awskeyserver is rather intimidating for a newbie😀

        Aaron Tait

        March 1, 2011 at 2:23 am

      • Just realized I made a few typos. IAS=IAM

        Aaron Tait

        March 1, 2011 at 2:33 am

  2. Adrian,

    Wanted to say project is excellent and solved a big problem for me. I reference your work on my stackoverflow answer:

    http://stackoverflow.com/questions/5037425/how-can-you-protect-info-contained-in-an-apk

    The only thing I can’t find out is how to remove the recaptcha option. Am I supposed to do something in permissions.py or somewhere else?

    Cheers, Ryan

    Ryan Cutter

    March 6, 2011 at 9:01 pm

    • Thanks for the comment, Ryan🙂

      Yes, the validators are set in permissions.py. Just set it to None if you don’t want the Captcha validation on.

      Adrian Petrescu

      March 6, 2011 at 9:09 pm

  3. Hi Adrian,

    Thanks for the information in this article. Couple questions – I read in AWS docs and here that keys should not be stored on mobile device. I certainly understand this point, but it seems this is highly application dependent. I mean if I am hosting my public web site on S3 and am using resources/feeds from that site(all public) to populate my mobile app…well I see no reason not to use an IAM read-only user for this data with key embedded with the app. Am I missing something here ?

    Question about the solution you present above is IAM gives you initial max of 5000 users and you can ask for more. If I am hoping my mobile app will grow to 100k or 1M users – am I then at the mercy of Amazon to increase my limit ?? Seems not a good factoid about this solution to report to senior management…Thanks again !

    Steve Abramson

    May 11, 2011 at 8:16 am

    • Hi Steve,

      The mandate not to store credentials on the device is precisely the reason why awskeyserver exists in the first place. If the credentials you’re storing are low-privilege IAM accounts (rather than your main account), revealing them is not a big deal. This is the whole point🙂

      As for the 5,000 user limit, take a look at my reply to Kevin’s comment below for some suggestions there.

      Thanks!
      -Adrian

      Adrian Petrescu

      June 25, 2011 at 2:21 am

  4. Hi Adrian,

    This code is great and thanks for putting this on github. How would I go about updating it so that I can assign a unique key and secret for each user via a rest call so that it shows up when I view this via IAM console. Thanks for any help you can provide.

    Vince

    May 11, 2011 at 7:52 pm

  5. Hello!

    After doing something similar for iOS, I realised I have the same problem than Steve Abramson.

    I asked in Amazon forums and still waiting…
    https://forums.aws.amazon.com/thread.jspa?threadID=67545

    Ricardo

    May 17, 2011 at 3:06 am

  6. Hey Adrian,

    Great stuff, thanks for the project. Correct me if I’m wrong but this would work for up to 5000 users, correct? I know Amazon says that you can “request more accounts” but that’s pretty vague to me, do you know what’s involved with requesting more than 5000 user accounts and how many more you can request?

    Thanks,
    Kevin

    Kevin

    June 25, 2011 at 2:12 am

    • Hey Kevin,

      Thanks🙂 Yeah, IAM has a built-in limit of 5,000. I haven’t had any experience with requesting more accounts, but I have requested other AWS resources (SES quotas, 20+ EC2 instances, etc) which you have to apply for directly, and never had a problem getting them in a timely manner. You might try that first, since I know for a fact that this type of usage of IAM is considered legitimate by the IAM team.

      On the other hand, as a technical solution, you can consider “round-robin”-ing it. When the number of accounts that awskeyserver has given out approaches 5,000 (say, at the 4,500 mark), you can start either deleting or re-using the keys at the beginning (re-using obviously has some minor security implications, but this may be okay depending on what you’re doing). As long as you have some logic in the client application to re-acquire a new key when the old one starts failing, you’ve essentially raised the limit to 5,000 concurrent users, rather than 5,000 total users — quite a difference!

      Enough people have brought this up that I think I will actually implement this in awskeyserver sometime soon. It will be quite time-consuming to test, though…

      Adrian Petrescu

      June 25, 2011 at 2:18 am

      • Hey, thanks for response. Yeah, I thought of that too but then as soon as you do that, you can no longer store user-specific data using this mechanism since the 5001st user will have access to the same objects as the 1st user which means that a service like Dropbox wouldn’t be able to rely on IAM when using a round-robin approach.

        Also, reading your Stackoverflow post from the link above, you mentioned IAM potentially getting into the mobile SDKs after it comes out of Beta but I can’t quite get my head around that – wouldn’t direct SDK access to IAM itself require some admin-level AWS permission/keys essentially reopening the same can of worms all over again? How would limited-access users be programmatically created directly from the mobile SDK without embedding the main AWS account keys and without a remote obfuscation service like awskeyserver?

        Cheers,
        Kevin

        Kevin

        June 25, 2011 at 11:44 pm

      • First of all thanks for a great post : it was the first sign of light when I started looking for such a solution.
        With respect to giving users temporary privileges, amazon now has published a solution in the form of Token vending machine:
        http://aws.typepad.com/aws/2011/09/updated-mobile-sdks-for-aws-improved-credential-management.html
        and here:
        http://aws.amazon.com/code/7351543942956566

        It is not as polished with respect to reCaptcha etc., it would be nice:
        1. Update the amazon code with all the nice features in your project
        2. Update the current key server above with the temporary token granting technique in there: I still would prefer GAE for scaling and also for being free(one needs to choose the size of ec2 instance running the TVM in case of amazon).

        Cheers!

        rajan.

        rajan

        September 23, 2011 at 12:49 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: