Log in

No account? Create an account

Previous Entry | Next Entry

Why hasn't Amazon fixed things overnight?

Well, if you have read Amazon CTO Werner Vogel's blog or seen any of his recent presentations, you'll definitely be (like me) inclined to "cock-up over conspiracy" as the explanation for the current shambles.

So why have things gone this desperately wrong this quickly?

The simple answer is Amazon's architecture. It's highly distributed, and there's no operations team. Each component (and over 200 go into a single page) is run by its development team, of four to five people. They are responsible for its features, its development - and for making sure it runs effectively. The result should be a company that can move quickly in response to outside events.

At least that's the theory.

I'm afraid the real world doesn't work like that. I've been a developer and I've managed developers and I can tell you that what really happens is something like this:

Someone comes up with a neat idea that they evangelise among the other developers, and it gets added to the platform. The developers become wedded to their idea, and they keep adding features. Something from the outside occurs that affects the data managed by the service, and they don't notice. After all, it's their design and it's perfect. The problem gets worse, and a few external symptoms are noted and passed on to the developers. They're too busy to pay much attention to them, and so they ignore them. Then suddenly, BANG, and everything breaks.

Oh, and it's a holiday weekend and there's no one there to actually handle the problem as the whole team's gone off on a skiing trip.

Now I can't guarantee that's what has happened with the deletion of GLBT content from the Amazon ratings system, but I suspect it's more likely than not.

So here's where my conjecture comes in:

Someone probably had the idea of reducing Amazon's exposure to bad publicity without increasing the site's legal liability. Manual censorship of the rankings would certainly make the service more liable, so the idea was probably a tool that would let the site's users do the work for it. After all, if the community doesn't like it, then, well, US community standards laws apply and you're safe. A group of developers coded it up, and it worked well - for a while.

Either a parameter wasn't quite right, or someone released a new version of a keyword file without testing - and, well, suddenly the GLBT books were off the list. Maybe someone gamed the system, too - it's impossible to tell from outside.

A separate test and operations team would have been likely to spot the underlying flaw before it got released - or at least spotted the first wave of complaints and started to triage them effectively, with a more productive response than "It's a glitch".

So now Amazon has to unwind data that's spread across its distributed application platform, which may be stored in any or all of three different kinds of database, and in at least three different geographies and many more data centres.


That's going to take a while to deal with.

Meanwhile their Seattle-based PR team is just about to start a very long day - and a group of developers are going to be desperately trying to explain just went wrong.

[ETA 23/4/2012. After three years of this post being targeted heavily by spammers, I have locked commenting.]


Apr. 13th, 2009 05:46 pm (UTC)
This stuff started in August 2008 for erotic fiction writers; Mark Probst's book apparently got delisted in February.

I'm not buying "glitch". Otherwise why were pro-gay books delisted but anti-gay material still available?
Apr. 13th, 2009 06:08 pm (UTC)
shoddy metatags and exclusion filters. This is why Judy Blume's Wifey didn't get deranked, and one version of E.M. Foster's Maurice is deranked, but not the Penguin Classics version.

The secret is fail in the automated code. They probably had a filter for 'romance' but exclusions based on other tags, certain authors, and publishers.
Apr. 13th, 2009 06:10 pm (UTC)
I'm still suspicious. "Glitches" don't usually involve warning publishers first, and some writers are reporting just that.
Apr. 13th, 2009 06:18 pm (UTC)
Oh, I'm sure that the hiding "adult" stuff policy is a new/expanded policy, I'm just pretty sure that this particular implementation and the search results (that is to say, the hideousness that shows up as #1 search result) fiasco is both unintentional and a result of untested, unfortunate code.

I'm pretty sure that if they had anyone testing this before they released it, this would have been caught... either the people doing the implementing ignored feedback, or pre-implementation testing of search results wasn't done at all.
Apr. 13th, 2009 06:22 pm (UTC)
All of that seems valid & likely. But as a consumer, my trust has been damaged and I'm more likely to seek out other sources for my online ordering in the future because of this, particularly because Amazon has not handled this well...private statements to GLAAD but nothing public other than "GLITCH NOTHING TO SEE HERE"?

I only used Amazon to purchase new books from indie authors who needed the ranks; other new & used purchases go through my local brick & mortar independent, BetterWorld or Half.com. If they're going to fuck with the rankings system in a non-transparent way, I'm sure my authorly friends are happy to sell new copies of my referrals through Powell's or the brick & mortar major chains.
Apr. 13th, 2009 07:19 pm (UTC)
Not t'all saying it shouldn't be. This kind of stuff happens all the time in MMOS for example... it's what we call Game Breaking Oversight. What this should do is spread out over the land as a banner of example of why companies that use software should fricking /test/ their stuff before they implement, for pete's sake.

No matter how non-malicious one is, the perception of indifference is likely to make one look proper malicious. And... well, consequences are consequences, the road to hell, et al.

Or, tl;dr (directed at software devs, pas at you, popelizbet)--Testing is your friend! Untested live code is NOT your friend! Check before you implement!

Or better yet, help the economy! Open a QA dept today!
Apr. 13th, 2009 07:23 pm (UTC)
the perception of indifference is likely to make one look proper malicious

Precisely this...and while I'll grant them it was a holiday weekend when all this went down? When the top result for "homosexuality" shows up as an anti-gay book designed to "prevent" homosexuality, it sure looks like someone's pushing an agenda, and it's that image that is going to stay with a lot of people, along with the high ranking still proudly displayed by 30 Years of Centerfolds while poor Annie Proulx's National Book Award winner sat stripped & Heather's two mommies disappeared from view.