?

Log in

No account? Create an account

Previous Entry | Next Entry

Why hasn't Amazon fixed things overnight?

Well, if you have read Amazon CTO Werner Vogel's blog or seen any of his recent presentations, you'll definitely be (like me) inclined to "cock-up over conspiracy" as the explanation for the current shambles.

So why have things gone this desperately wrong this quickly?

The simple answer is Amazon's architecture. It's highly distributed, and there's no operations team. Each component (and over 200 go into a single page) is run by its development team, of four to five people. They are responsible for its features, its development - and for making sure it runs effectively. The result should be a company that can move quickly in response to outside events.

At least that's the theory.

I'm afraid the real world doesn't work like that. I've been a developer and I've managed developers and I can tell you that what really happens is something like this:

Someone comes up with a neat idea that they evangelise among the other developers, and it gets added to the platform. The developers become wedded to their idea, and they keep adding features. Something from the outside occurs that affects the data managed by the service, and they don't notice. After all, it's their design and it's perfect. The problem gets worse, and a few external symptoms are noted and passed on to the developers. They're too busy to pay much attention to them, and so they ignore them. Then suddenly, BANG, and everything breaks.

Oh, and it's a holiday weekend and there's no one there to actually handle the problem as the whole team's gone off on a skiing trip.

Now I can't guarantee that's what has happened with the deletion of GLBT content from the Amazon ratings system, but I suspect it's more likely than not.

So here's where my conjecture comes in:

Someone probably had the idea of reducing Amazon's exposure to bad publicity without increasing the site's legal liability. Manual censorship of the rankings would certainly make the service more liable, so the idea was probably a tool that would let the site's users do the work for it. After all, if the community doesn't like it, then, well, US community standards laws apply and you're safe. A group of developers coded it up, and it worked well - for a while.

Either a parameter wasn't quite right, or someone released a new version of a keyword file without testing - and, well, suddenly the GLBT books were off the list. Maybe someone gamed the system, too - it's impossible to tell from outside.

A separate test and operations team would have been likely to spot the underlying flaw before it got released - or at least spotted the first wave of complaints and started to triage them effectively, with a more productive response than "It's a glitch".

So now Amazon has to unwind data that's spread across its distributed application platform, which may be stored in any or all of three different kinds of database, and in at least three different geographies and many more data centres.

Ooops.

That's going to take a while to deal with.

Meanwhile their Seattle-based PR team is just about to start a very long day - and a group of developers are going to be desperately trying to explain just went wrong.

[ETA 23/4/2012. After three years of this post being targeted heavily by spammers, I have locked commenting.]

Comments

( 74 comments )
Page 1 of 3
<<[1] [2] [3] >>
debgeisler
Apr. 13th, 2009 01:40 pm (UTC)
All of this is going to make a fascinating set of case studies in PR crisis management.
sbisson
Apr. 13th, 2009 01:48 pm (UTC)
Oh indeed.
(no subject) - therealdrhyde - Apr. 13th, 2009 02:38 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 13th, 2009 03:16 pm (UTC) - Expand
(no subject) - qviri - Apr. 13th, 2009 10:50 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 13th, 2009 11:18 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 13th, 2009 11:22 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 13th, 2009 11:31 pm (UTC) - Expand
(no subject) - qviri - Apr. 14th, 2009 12:38 am (UTC) - Expand
(no subject) - hal_obrien - Apr. 14th, 2009 08:22 pm (UTC) - Expand
(no subject) - qviri - Apr. 14th, 2009 10:17 pm (UTC) - Expand
(no subject) - qviri - Apr. 14th, 2009 10:28 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 14th, 2009 10:52 pm (UTC) - Expand
(no subject) - qviri - Apr. 14th, 2009 11:58 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 15th, 2009 12:11 am (UTC) - Expand
(no subject) - therealdrhyde - Apr. 25th, 2009 02:28 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 25th, 2009 06:34 pm (UTC) - Expand
(no subject) - therealdrhyde - Apr. 25th, 2009 02:25 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 25th, 2009 06:25 pm (UTC) - Expand
(no subject) - debgeisler - Apr. 13th, 2009 05:38 pm (UTC) - Expand
(no subject) - dampscribbler - Apr. 13th, 2009 06:53 pm (UTC) - Expand
(no subject) - palmer_kun - Apr. 13th, 2009 07:13 pm (UTC) - Expand
(no subject) - ksol1460 - Apr. 13th, 2009 08:39 pm (UTC) - Expand
(no subject) - estrelladesax - Apr. 13th, 2009 05:25 pm (UTC) - Expand
#amazonfail - (Anonymous) - Apr. 13th, 2009 08:22 pm (UTC) - Expand
tigerbright
Apr. 13th, 2009 01:43 pm (UTC)
I've already linked to you in replies to others. :)
kenmarable
Apr. 13th, 2009 02:24 pm (UTC)
Just goes to show you, never release code or data changes over a holiday weekend.

"But fewer users will be using the system..."

*sigh*


popelizbet
Apr. 13th, 2009 05:46 pm (UTC)
This stuff started in August 2008 for erotic fiction writers; Mark Probst's book apparently got delisted in February.

I'm not buying "glitch". Otherwise why were pro-gay books delisted but anti-gay material still available?
(no subject) - caudelac - Apr. 13th, 2009 06:08 pm (UTC) - Expand
(no subject) - popelizbet - Apr. 13th, 2009 06:10 pm (UTC) - Expand
(no subject) - caudelac - Apr. 13th, 2009 06:18 pm (UTC) - Expand
(no subject) - popelizbet - Apr. 13th, 2009 06:22 pm (UTC) - Expand
(no subject) - caudelac - Apr. 13th, 2009 07:19 pm (UTC) - Expand
(no subject) - popelizbet - Apr. 13th, 2009 07:23 pm (UTC) - Expand
johnpalmer
Apr. 13th, 2009 02:44 pm (UTC)
Well, if you have read Amazon CTO Werner Vogel's blog or seen any of his recent presentations, you'll definitely be (like me) inclined to "cock-up over conspiracy"

It's Monday morning, and I spent way too long trying to figure out how one "cock(s) up over conpsiracy" before I realized what you'd meant.
rabidsamfan
Apr. 13th, 2009 02:44 pm (UTC)
A possible explanation, sure, but it wouldn't have worked if some management level policy setter hadn't agreed that it would be okay to a) block rankings on some material and b) block unranked items from the front page search.

That takes it past glitch and into the arena of fail. If Amazon wants the statistic "Amazon Rank" to mean something, it cannot be applied to only a bowdlerized portion of the database. And blocking people from being able to search in the easiest fashion for material you want to sell is just plain stupid.
kshandra
Apr. 13th, 2009 02:52 pm (UTC)
In comments on a post putting forward a trolling theory that I don't quite buy, eldestmuse said something that ties into this, IMO:

Even if they aren't deliberately targeting the LGBT audience, the fact remains that the philosophy behind putting the feature to remove the sales ranks of bestselling authors, thus hurting their livelihood based on morality instead of popularity, based on being "adult" is fundamentally objectionable.
(no subject) - rabidsamfan - Apr. 13th, 2009 02:58 pm (UTC) - Expand
jane_dark
Apr. 13th, 2009 02:48 pm (UTC)
What you're saying makes sense, though I find it pretty troubling that so little of the hetero-adult/erotic content has been targeted, and that makes it a bit harder for me to accept that this is a well-intentioned glitch. Bestiality is okay, John Barrowman is not?

At best, a few individuals in the system of developer checks really screwed up.

I'm waiting with interest to see Amazon's official response -- with names attached, and on their website, as opposed to reported by WaPo or Publishers Weekly.
(Anonymous)
Apr. 13th, 2009 04:13 pm (UTC)
So, you think it is okay for books with erotic content (including great works of English literature like _Lady Chatterley's Lover_ and _Little Birds_) to be "hidden" from view just because they are about (eek!) sex? Please!
(no subject) - jane_dark - Apr. 13th, 2009 04:29 pm (UTC) - Expand
pingback_bot
Apr. 13th, 2009 03:14 pm (UTC)
#amazonfail 2
User briarwood referenced to your post from #amazonfail 2 saying: [...] the story and Amazon told them the same steaming pile: it's "a glitch". Meanwhile posted about how this happened from a technical standpoint none of which does anything to make the investigations done by Dear Authorany less damning. ... [...]
(Anonymous)
Apr. 13th, 2009 03:15 pm (UTC)
ironic
ironically, this is what happens when information processing and distribution is managed by corporate boobs.
masgramondou
Apr. 13th, 2009 03:36 pm (UTC)
I think that, as I wrote in this post you are basically correct. The problem is that I think the whole thing is in fact symptomatic of some major underlying problems regarding scalability and (now that I've thought about it some more) some fundamental assumptions about (meta)data which have turned out to be flat out wrong.
(Anonymous)
Apr. 13th, 2009 03:40 pm (UTC)
Very recognizable and plausible, but...
I'm also a developer, and I've also managed developers (still am), and I distinctly recall having a name for those kind of scenarios:

Incompetence.

Which by the way still doesn't excuse Amazon's lack of response (a simple "this is clearly not our intent, please hold on while we look into it" would have sufficed).

But most of all, the fact that the culture within Amazon is such that it is possible to implement an automated system that quietly, without informing either the public or the authors and publishers, makes books disappear from searches and rankings is A Very Bad Thing. Whether it was a management decision or a developers decision, whether the current result was intentional or not doesn't really matter.

It's not the fact that Amazon hasn't rolled back these changes yet that is worrying. It's that they've shown no intent whatsoever to do that, or even acknowledge that it might be wrong.
conuly
Apr. 13th, 2009 07:09 pm (UTC)
Re: Very recognizable and plausible, but...
a simple "this is clearly not our intent, please hold on while we look into it" would have sufficed

Exactly. It might not have made any of us happy, but at least we would not have all felt insulted by that.

It's not the fact that Amazon hasn't rolled back these changes yet that is worrying. It's that they've shown no intent whatsoever to do that, or even acknowledge that it might be wrong.

Agreed.
(Anonymous)
Apr. 13th, 2009 03:43 pm (UTC)
Disability books affected also
It's not only GLBT books being affected. Some disability books also have been affected. Details at:

http://textualfury.wordpress.com/2009/04/12/amazonfail/
http://lisybabe.blogspot.com/2009/04/amazonfail.html


I've now posted the following text as a tweet. Others may please feel free to copy/paste this text as their own tweets also. It links to the above two blog posts on the disability angle of this mess. The hashtags are to help ensure that your tweet will show up more easily under certain keyword searches in twitter.com.

#amazonfail on #disability too. Please RT both http://is.gd/s5I6 and http://is.gd/s5Xr #glitchmyass #glbt #disabled

damsel_ophelia
Apr. 13th, 2009 04:22 pm (UTC)
Re: Disability books affected also
Rebroadcasted. Not getting as much press, but just as important - and sickening.
Re: Disability books affected also - essaying - Apr. 13th, 2009 08:44 pm (UTC) - Expand
pingback_bot
Apr. 13th, 2009 03:44 pm (UTC)
User boji referenced to your post from saying: [...] as discovered by who did indeed phone. Oh and more on how such a tech-clusterfuck can happen: h e r e. Best places to watch this are still on twitter at #amazonfail & #glitchmyass. From there I learnt that Heather has two Mommies has ... [...]
lyorn
Apr. 13th, 2009 04:48 pm (UTC)
(here via makinglight)

I have worked in projects like that, and yes, that is the way shit happens, and can't be stopped from rolling downhill quickly.

One item still boggles me: It seems likely that at the beginning of all this was the attitude that it's OK to fudge statistics without informing people who might use them. Of course, amazon sales rank might be more voodoo than statistics anyway...
pingback_bot
Apr. 13th, 2009 04:58 pm (UTC)
Amazonfail: Day Two
User afterthree referenced to your post from Amazonfail: Day Two saying: [...] study. And if you think this is a quick fix and are wondering why everything's still de-ranked, explains why it isn't. [...]
amberdine
Apr. 13th, 2009 05:19 pm (UTC)
Thank you, yes. First sensible theory I've heard. Pretty much was my guess, too, though I wasn't sure how the developers worked.

I have some dealings with Amazon as a reviewer and there are loads of completely innocuous things their database screws up regularly.
Page 1 of 3
<<[1] [2] [3] >>
( 74 comments )