Data Portability

The Authosphere is a buzz this morning with XAuth. My first instinct when I started seeing headlines about it was, yet another OAuth competitor?

It turns out it is nothing of the sort and it could turn out to be a very useful service. “Service” you say, isn’t it a standard? It really is a service and not a new authentication standard. But I’ll get to that.

What is it?

The way you can look at XAuth is that it is a cross domain cookie containing a list of services that you use.

What does it solve?

The problem that this aims to fix is really a UI one, not a security one. If you know the user already uses Google accounts and Meebo, there is no need to over clutter your user interface with login buttons to 40 different social networks.

It could work really well hand in hand with both OpenID and OAuth to minimize the user interface for connecting to them. OpenTransact also has potential uses for XAuth as it could be used on a payment form to filter the list of payment types a user uses.

So how does it work?

XAuth essentially defines 3 different parties to the flow:

• Extenders are web services that a user is logged into that present some public API. These often have the same role as the OpenID providers and OAuth Providers.
• Retrievers are web services that want to discover and consume one or more of the Extenders. In OpenID terms these are relying parties and in OAuth terms these are Consumers.
• XAuth.org is the final party. All communication of XAuth happens through an iframe and javascript hosted here. This is just static hosting, all data is stored in the users browser.

How do I use it?

As a developer it is pretty easy to do. Most of the work is in the Javascript layer and there really is very little that needs to be done on the server backend side. As I understand more about the process I will update this page.

XAuth is implemented as a javascript library you import in your header:

<script type="text/javascript" src="http://xauth.org/xauth.js"></script>

This script creates a hidden iframe on your page with the following page http://xauth.org/server.html.

Extender implementation

If you are an Extender, you extend your service by calling the XAuth.extend() function with a token. I would expect this to be done once a user logs in to the Extender service.

XAuth.extend({
  token: "bb87aef641b8f29023a8c207f",
  expire: 1271773157402,
  extend: ["www.meebo.com", "www.youface.com", "*"],
  callback: extendCallback
});

The token is a XAuth specific token, which I don’t completely understand yet. The demo code just generates a random token. Here is what the spec says:

this is the XAuth token, created specifically for use with XAuth and used by an auth extender to request additional services from the extender. This token will be stored with the canonical domain as the key.

My guess is that it should say:

this is the XAuth token, created specifically for use with XAuth and used by an auth Retriever to request additional services from the extender. This token will be stored with the canonical domain as the key.

The spec doesn’t mention more here but I imagine this could be used together with OAuth somehow for a really low permission OAuthToken. But lets see what implementers do with it.

The extend allows you to limit the token to only certain trusted domains. Use “*” to allow anyone access.

The optional callback method is called with the following data:

{
  "cmd": "xauth::extend",
  "id": 0
}

Once this has been called a record is created in the HTML5 Local Storage for the http://xauth.org/server.html page:

If you are unfamiliar with HTML5 Local Storage, think of it as a more evolved form of cookies. Some people have called in browser based NoSQL.

As part of your logout code you should also call XAuth.expire(), which removes it from the list of tokens.

Retriever implementation

The retriever is also pretty simple to implement. You make the call to the XAuth.retrieve() function:

XAuth.retrieve({
  retrieve: ["www.meebo.com", "www.youface.com", "xauth.org"],
  callback: retrieveCallback
});

You must list all the Extender domains you are interested in and the callback function is called with tokens for each of the available Extenders.

{
  "cmd": "xauth::retrieve",
  "id": 4,
  "tokens": {
    "xauth.org": {
      "token": "bb87aef641b8f29023a8c207f",
      "expire": 1271773157402
    }
  }
}

Now remember XAuth doesn’t say what to do with this information. Everything from now follows the regular API’s you already use. I expect that this is initially going to be used to hide various openid login widgets in the beginning.

For an interactive spec/testing console go to the XAuth Specs page.

User privacy issues

So as a user how do you control this? Do I really want these sites to know what I use?

At the XAuth home page you will find this control panel:

It allows you to switch XAuth off completely and to see and manage the tokens Extenders are publishing for you.

Service or Standard

I guess it really doesn’t matter, but strictly speaking this is a service not a standard. It does rely on the XAuth server to provide js and the iframe page.

This is Discovery not Auth

I find probably the biggest issue with XAuth is the choice of name. XAuth is not an authentication protocol. I would call it a discovery protocol. It has more in common with WebFinger (again in a complimentary way) than with authentication protocols like OpenID and OAuth.

Having spent more than 2 years explaining to people that OAuth and OpenID are not the same, this just adds more confusion. Names are really important.

I also think the choice of Extenders and Retrievers is confusing. I think using Provider and Client would be clearer. Lets not invent more terminology if it isn’t necessary.

Conclusion

I think this could be really useful. It looks very easy to implement for both Extenders and Retrievers. The usage of HTML5 functions also hints at a bunch of new services that are now possible.

Of course with any of these kinds of services until there are Extenders out there, very few Retrievers are going to implement this. I understand that both Meebo and Google will support it.

As it is so easy to implement, it might not hurt at all for smaller startups to implement Extenders. It could be a way to promote your API.

2 news stories on the same day are quite interesting in their contrast.

Pin and Chip is broken

The first one has the collective might and minds of the European banking system and their suppliers who overlooked a slight issue in their authentication protocol for authenticating Chip cards with a pin number. In Europe most Visa/MC cards are smart cards and have to be authenticated with a pin. This in theory allows for an authenticated payment message.

Only problem was that, well one very important bit of the message was not authenticated leaving a gaping hole. I won’t go into the details as well as Ross Anderson does. He is one of the security researchers who discovered the flaw. Unfortunately it sounds like carders discovered it before them.

Now what to do with these supposedly safe authenticated transactions? There is no way of knowing which ones were fake. You can’t mass revoke all european cards. Some one is in a bit of a bind right now.

Grader’s security screw up

The second story was about HubSpot a Cambridge, Mass. based startup who self admittedly screwed up and let a malicious user comprise the security of their Grader service a rating service for twitter users.

Granted we are not talking about a system that handles the majority of Europe’s electronic point of sales transactions here. But they know they screwed up. However due to the fact that Grader used OAuth they were able to mitigate any damage pretty quickly by asking Twitter to revoke their Consumer credentials and any tokens they had issued to it.

Revokability

The difference is that while both Chip and Pin and OAuth are ways of doing delegated authentication, the only token to revoke in the Chip and Pin scheme is in the card itself. The standards behind Chip and Pin assumes that it’s technology is perfect and through their rule books that all parties involved along the long chain from the card to the issuing bank can automatically be trusted.

This is basically the exact issue I described in The sorry state of Payment Standards.

OAuth does not define how a user authenticates himself to either of the services involved, rather it is focused on the delegation.

The delegation is done in the form of an authorized token that can be equipped with limits and can at any time be revoked. It is under the control of the user. In this case Grader themselves request the revocation as they knew that all of their credentials were compromised. Where do the European bankers even start to clean up this mess?

I think OAuth a simple (as authentication standards go) standard developed on a mailing list by a small group of developers has incredible potential in payments applications. This is of course why we picked it as one of the fundamental building blocks for OpenTransact.

Is OAuth perfect? Probably not. Nothing is 100% secure. It has had one serious security flaw which was fixed. But by design it is revokable. You can do something about it if something goes wrong. There is now an IETF OAuth Working Group working on making it an official internet standard.

One of the great new standards made possible by OAuth is PortableContacts. Joseph Smarr from Plaxo is the main force behind it together with Chris Messina and others.

Portable Contacts as the name suggests is a standard for allowing one application access to your contact data on another application. The standard is a nice example of a very clean API which supports both JSON and XML.

I was looking around and I couldn’t find a Ruby library for it, so I wrote this simple Portable Contacts Ruby Gem to allow me and others to use it.

It requires OAuth and the Ruby OAuth Gem. If you need to use it with a Rails application use the OAuth Plugin if using straight Ruby use the Ruby Gem. All you need is an AccessToken object and a Portable Contacts URL to get started.

@access_token = ... # instantiate access token
@client = PortableContacts::Client.new "http://www-opensocial.googleusercontent.com/api/people", @access_token

# Get users profile

@profile = @client.me

puts @profile.display_name
=> "Bob Sample"

# Get users contacts
@contacts = @client.all

If you are using the plugin it is very easy to get started accessing Google.

Just install it and configure it. Once you have a GoogleToken for your user:

@google_token=GoogleToken.find_by_user_id @user.id
@client = @google_token.portable_contacts

# Get users profile

@profile = @client.me

puts @profile.display_name
=> "Bob Sample"

# Get users contacts
@contacts = @client.all

As this is the very first version there are likely to be issues, please do submit patches. I would also love to include a PlaxoToken etc for the OAuth Plugin if someone wants to submit one.

Plans

Eventually there will be some sort of auto discovery so you don’t need to provide a Portable Contacts url.

I would also like to add a server component to the gem, making it easy for you to provide portable contacts for your own application.

Yahoo implemented OAuth sometime last year. So now you use the same OAuth libraries that you use to connect to Google, Twitter, MySpace etc to access your users Yahoo address books. This is the Yahoo OAuth Documentation.

The only problem is that Yahoo quietly pushed through an extension to OAuth called the OAuth Session Extension. This was all done publicly but it involves some major changes to OAuth token management, which is the exact thing that has been hardest for people to understand about OAuth in the first place.

Right now I’m feeling a bit like Arthur Dent in the Hitchhikers Guide to the Galaxy lying in front of the bulldozer trying to demolish my home:

“But Mr. Dent, the plans have been available in the local planning office for the last nine months.”

“Oh yes, well, as soon as I heard I went straight round to see them, yesterday afternoon. You hadn’t exactly gone out of your way to call attention to them, had you? I mean, like actually telling anybody or anything.”

The thing is Authentication standards are hard, they are by necessity complex. Therefore it is vital that they are predictable. It isn’t the job of an developer trying to connect to their users Yahoo address book to understand all the different complexities of the standard. This is the job of the library writers.

I think we have done a great job in the Ruby community with the OAuth libraries. The “Ruby Gem” of which I was one of the co-authors of does a great job of handling the complexity, but still caused several problems with new users as it still required you to understand the OAuth flow. My new OAuth Consumer Rails Plugin attempts to hide that to make it extremely easy for developers to write applications that access Twitter, Google, FireEagle etc.

The Token Refresh Step

While I hadn’t personally been looking at the special requirements for communicating with Yahoo as I haven’t had the need until now. I had noticed the large amounts of problems people have had with Yahoo. I knew there were a couple of extra fields to pass along throughout the flow, which isn’t a problem. I also noticed that the tokens were quite large at around 800 bytes. Those are all things you can work around as they don’t change the flow. The big problem is the completely new Token Refresh step.

Basically when you get an AccessToken from Yahoo it is only valid for an hour. So you need to refresh it by requesting a new access token before attempting to use it again. This means that we as consumers of Yahoo web services need to monitor expiration of Yahoo tokens, create an additional HTTP requests before we can connect for whatever it is we are using it for.

No, this is not insurmountable, they just add a lot more complexity to our code.

We are asked to pay for Yahoo’s scalability issues

However when you dig down and find out the reason for this it is quite interesting. Apparently the reason for this is due to Yahoo’s security architecture.

Allen Tom from Yahoo writes in the original proposition#

Very large service providers may have a distributed architecture in
which the service endpoints are able to cryptographically verify the
authentication credentials without calling back to a central database.
Permission revocation is implemented by issuing consumers a relatively
short lived session credential, and requiring consumers to
periodically obtain new session credentials from a central
authorization service. The authentication service performs the
necessary checks before issuing new session credentials.

Although it is technically possible for SPs to build an OAuth
translation layer on top of their existing services, the cost may
involve additional latency and decreased reliability if OAuth
verification required a query to a database located in a very distant
datacenter.

This is also mentioned in a tiny paragraph at the end of the OAuth Session Extension Spec

Allowing Access Tokens to be revoked before they expire requires Service Providers to perform a database lookup before serving a Protected Resource. For performance reasons, Service Providers may want to issue Access Tokens that can be validated without a database lookup, provided that the Access Token lifetime is less than then the Service Provider’s allowable latency for Access Token revocation.

So Yahoo proposed this for all intents and purposes to avoid a database call on every request. I can understand they want to avoid this, but there are lots of other solutions.

This all explains the Yahoo token size. Basically Yahoo’s token contains expiry information, probably a user id and some sort of consumer id. This is all encrypted and signed similar to the Rails’ cookie based sessions. This is very clever and I can appreciate the design. This allows the various servers in Yahoo’s cloud to verify OAuth tokens without accessing a central OAuth token database.

The expiry and refresh section is so users can revoke an applications token. By the time my application attempts to refresh the token from Yahoo’s token servers my user might have removed the permissions and the token refresh is declined.

A solution to Yahoo’s problem

Rather than us the developers having to refresh the token, I think it would make a lot more sense to leave expiry out of the token completely. The following would happen within Yahoo and be completely hidden from external application:

1. Application requests contact info from Yahoo
2. Yahoo’s Contacts API server looks in local token cache for applications token
3. Token is not found in cache
4. Contacts API server asks internal OAuth server to verify Token and returns secret
5. Token and token secret is stored in cache with 1 hour expiry

This allows Yahoo to manage their internal OAuth token expiry in a very scalable way without adding any additional complexity for external developers. And yes Yahoo I will happily sign an IPR for you if you need it to implement this.

Just say no to extra vendor tax

What Yahoo’s OAuth implementation is, is a de facto tax on people developing towards their API’s. It adds additional overhead and complexity to libraries, extra unnecessary http requests for developers and breaks what is supposed to be a standard.

I have decided to stand firm and not add support for Yahoo to the OAuth Plugin. I started work on it but I have removed it from the current release and won’t add it back until they follow the standard OAuth flow. I encourage other library authors to do the same.

Why is this so important

OAuth was designed as lower level modern authentication standard that other higher level applications can be built upon. An extension on top of it is the OAuth discovery proposal which in theory would allow you to connect to any provider by just giving it a single top level URL.

So lets say I wanted to connect to Yahoo. I could just do something like:

@yahoo = OAuth::Consumer.new :site=>"http://yahoo.com"

Everything afterwards would be automatic. No specifying of request token urls, authorization urls etc.

What makes this even more interesting is when you start adding higher level API’s on top of OAuth, such as Portable Contacts. This would allow you with a single library with no extra code access your users address book, whether it is on Yahoo, Google, Plaxo or even your own personal address book server. Having a straightforward discoverable protocol without extra overhead makes it simple for developers to access. It removes all the proprietary code exceptions we have to write today.

Having proprietary extensions on something as fundamental as the OAuth token request cycle breaks the beauty and simplicity of PortableContacts and other future standards.

Should standards be dictated by large corporations?

My point about all of this is that I don’t think it is right that large companies try and push through changes to community created standards solely because it will make their life a little bit easier. OAuth came out of small startups and individuals who saw a common problem and wanted to create a solution. We should not just blindly accept every suggestion that comes out of large companies just so we can announce with much excitement that we have the support of Google, Yahoo or Microsoft. I think it’s great that they join in the process, but we should not bend our principles, just for the sake of them joining.

The debate on the mailing list

I have been debating this on the OAuth mailing list and while my initial tone was definitely negative, it was never targeted at any individuals. Unfortunately the discussion has now become more of a personal fight, which I don’t think is particularly useful.

While this particular discussion is about Yahoo I would be equally disturbed by it if any other major provider tried to do the same thing. Yahoo is famous in OAuth circles for their corporate culture and infrastructure leaking into the standards as wast last seen during the OAuth IPR debate last year.

Yahoo has done great work in a whole lot of instances. FireEagle in particular is still one of the best OAuth implementations out there. I just wish they had done a better job of thinking through their corporate OAuth strategy.

It has been fairly easy to provide OAuth services in your web application (see How to turn your Rails Site into an OAuth Provider), but to actually consume Twitter, FireEagle, my own Agree2 and other OAuth services has been a fairly manual affair.

There are great gems out there that wrap up the process for the above mentioned services. So it hasn’t been too hard to support one of them. But what to do if you want to support 5 different services today and more in the future?

I knew there should be some generic approach to handle the OAuth authorization process, but had not spent too much time thinking about it until we actually needed to consume external web services for Agree2.

Well I think I’ve cracked it in a a nice Dont Repeat Yourself fashion.