XBIZ recently sat down with Dowland to talk about the basics (and sometimes the not-so-basics) of managing password security in a high-volume content distribution environment.
XBIZ: Everybody knows password protection is important, but how much impact can an adult site password breach really have? Is it largely a matter of wanting to prevent spikes in bandwidth, or are there potentially more severe consequences from such a breach?
Dowland: User password breaches can definitely cause an impact in bandwidth, but overall that's a minor concern. There are plenty of programs available that can monitor for these sorts of breaches and shut them down before they become a big concern. One we've used for years with success is ProxyPass.
Another, possibly more serious problem is the impact to customers that have their password breached without their knowledge. There are a lot of customers who insist on using easily guessable passwords, or who leak their password in some other way. It's an inconvenience to those customers to have their account reset with a new password when they just want to use the system. Unfortunately, there's nothing you can do besides reset their password and try to make it as easy as possible for them to get their new password, or to reset it again to a new password that they choose.
XBIZ: Does your company use a proprietary or off-the-shelf solution for password management?
Dowland: It's a completely proprietary system based on a highly distributed database system using custom replication methods. We used to use flat text files, but had problems with file integrity, slow propagation and lack of audit trails. Our new system is highly available, very robust in design using a transaction model, is very fast at propagating passwords to all cluster boxes (less than a second on average) and provides a very easily audited system with a lot of safeguards built in to prevent problems. We've been running on that system for several years now, and tend to forget it's even there because it just works.
XBIZ: Is it harder to secure something like PlugInFeeds, which supplies content to a large number of third-party sites, than it is to maintain password security for a network of your own sites, or is it basically the same kind of challenge?
Dowland: PlugInFeeds is a completely different challenge for security than a traditional paysite. The main problem is that the PlugInFeeds servers have no access to a client's password database. We can't verify if a user is a valid user to them once the user is on our servers. Instead, we use a custom Apache module to allow us to pass around tokens for verification. So the tokens have a given TTL (Time To Live), and they also contain all of the info we need to track where any given user came from, so we know which client to bill for the bandwidth, etc. We rely on those tokens for our front line of security, but we do also offer referrer verification as an extra layer if a customer desires, but at no point do we rely on referrer verification alone.
The main security challenge of PlugInFeeds came when figuring out how to pass our authentication token from the client to the server on each request. We had a few options that we had used before: Cookie, basic auth (https://username:password@domain.com/), or parameters (?token=KDS87...). There were serious shortcomings with all of those methods, though.
The cookie method didn't work with the media players reliably. Certain media players would not pass the cookie that the browser had. So when we embedded the media player in the page, it would request the file without the cookie, and the server would have to reject it. The basic auth method worked with the media players, but it required us to maintain a current list of allowed "tokens" on all of the media servers. This required expensive database connections in the back-end, and was prone to inconsistency in the face of transient errors. We have media servers in several different data centers around the U.S., so it was doubly problematic for those.
The parameters solution was not desirable because then we would have to have server-side scripts process all requests, including simple media requests. It's not very efficient to have file transfers tunneled through a script. It's much better to just let Apache handle that, since that is what it does well.
Instead, we hit on a novel method of putting the token directly in the URI of the request. So our token pretends to be the top-level directory of the site — i.e., https://pluginfeeds.com/_uk_TOKEN123/file.jpg (_uk_ stands for URI Key).
As long as all of the links on a page are relative, the browser happily sends the token with the request. If needed, we can also construct absolute URLs in the page that include the token as well.
This made passing around the token much easier, and it also meant that there didn't have to be any back-end communication between servers. Each media server can verify a token completely locally without having to communicate with any databases, or anything. You can see these tokens whenever you're browsing PlugIn Feeds pages.
There was still one challenge, though, with all of the possible methods, and that was link-sharing. In our solution, the token is in the URL, so it can be easily shared with other people.
The last major piece of our custom Apache module is a mechanism for maintaining the "state" of a token that is cluster-aware. All of the media machines constantly chat with each other in the background, exchanging information about what users they've seen, from where, and who they're blocking. They have a lot of logic built in to come to consensus about how to deal with "naughty" users. Once the system determines that a token has been shared, it will shut it down on all media servers immediately. It also has logic built in to detect shared proxies, however, like the infamous AOL dial-up proxies, so that users on those kinds of systems don't get shut down inadvertently.
Overall it's a system that has worked amazingly well, and is another one that we tend to forget about because it simply works. Since implementation, there has been no breach of this system. It has proven itself to be quite resilient to the undesirables of the Internet.
Implementation of this system was no easy feat, however, and required months and months of development time.
So yes, I would say that securing PlugInFeeds was a challenge many orders of magnitude more difficult than a traditional adult paysite.
XBIZ: It sounds like your company has developed a robust solution to user authentication — thanks for giving us an inside look.
