Previous Entry | Next Entry

Why hasn't Amazon fixed things overnight?

Self Portrait
Well, if you have read Amazon CTO Werner Vogel's blog or seen any of his recent presentations, you'll definitely be (like me) inclined to "cock-up over conspiracy" as the explanation for the current shambles.

So why have things gone this desperately wrong this quickly?

The simple answer is Amazon's architecture. It's highly distributed, and there's no operations team. Each component (and over 200 go into a single page) is run by its development team, of four to five people. They are responsible for its features, its development - and for making sure it runs effectively. The result should be a company that can move quickly in response to outside events.

At least that's the theory.

I'm afraid the real world doesn't work like that. I've been a developer and I've managed developers and I can tell you that what really happens is something like this:

Someone comes up with a neat idea that they evangelise among the other developers, and it gets added to the platform. The developers become wedded to their idea, and they keep adding features. Something from the outside occurs that affects the data managed by the service, and they don't notice. After all, it's their design and it's perfect. The problem gets worse, and a few external symptoms are noted and passed on to the developers. They're too busy to pay much attention to them, and so they ignore them. Then suddenly, BANG, and everything breaks.

Oh, and it's a holiday weekend and there's no one there to actually handle the problem as the whole team's gone off on a skiing trip.

Now I can't guarantee that's what has happened with the deletion of GLBT content from the Amazon ratings system, but I suspect it's more likely than not.

So here's where my conjecture comes in:

Someone probably had the idea of reducing Amazon's exposure to bad publicity without increasing the site's legal liability. Manual censorship of the rankings would certainly make the service more liable, so the idea was probably a tool that would let the site's users do the work for it. After all, if the community doesn't like it, then, well, US community standards laws apply and you're safe. A group of developers coded it up, and it worked well - for a while.

Either a parameter wasn't quite right, or someone released a new version of a keyword file without testing - and, well, suddenly the GLBT books were off the list. Maybe someone gamed the system, too - it's impossible to tell from outside.

A separate test and operations team would have been likely to spot the underlying flaw before it got released - or at least spotted the first wave of complaints and started to triage them effectively, with a more productive response than "It's a glitch".

So now Amazon has to unwind data that's spread across its distributed application platform, which may be stored in any or all of three different kinds of database, and in at least three different geographies and many more data centres.

Ooops.

That's going to take a while to deal with.

Meanwhile their Seattle-based PR team is just about to start a very long day - and a group of developers are going to be desperately trying to explain just went wrong.

[ETA 23/4/2012. After three years of this post being targeted heavily by spammers, I have locked commenting.]

Comments

( 75 comments )
Page 1 of 2
<<[1] [2] >>
debgeisler
Apr. 13th, 2009 01:40 pm (UTC)
All of this is going to make a fascinating set of case studies in PR crisis management.
sbisson
Apr. 13th, 2009 01:48 pm (UTC)
Oh indeed.
(no subject) - therealdrhyde - Apr. 13th, 2009 02:38 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 13th, 2009 03:16 pm (UTC) - Expand
(no subject) - qviri - Apr. 13th, 2009 10:50 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 13th, 2009 11:18 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 13th, 2009 11:22 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 13th, 2009 11:31 pm (UTC) - Expand
(no subject) - qviri - Apr. 14th, 2009 12:38 am (UTC) - Expand
(no subject) - hal_obrien - Apr. 14th, 2009 08:22 pm (UTC) - Expand
(no subject) - qviri - Apr. 14th, 2009 10:17 pm (UTC) - Expand
(no subject) - qviri - Apr. 14th, 2009 10:28 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 14th, 2009 10:52 pm (UTC) - Expand
(no subject) - qviri - Apr. 14th, 2009 11:58 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 15th, 2009 12:11 am (UTC) - Expand
(no subject) - therealdrhyde - Apr. 25th, 2009 02:28 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 25th, 2009 06:34 pm (UTC) - Expand
(no subject) - therealdrhyde - Apr. 25th, 2009 02:25 pm (UTC) - Expand
(no subject) - hal_obrien - Apr. 25th, 2009 06:25 pm (UTC) - Expand
(no subject) - debgeisler - Apr. 13th, 2009 05:38 pm (UTC) - Expand
(no subject) - dampscribbler - Apr. 13th, 2009 06:53 pm (UTC) - Expand
(no subject) - palmer_kun - Apr. 13th, 2009 07:13 pm (UTC) - Expand
(no subject) - ksol1460 - Apr. 13th, 2009 08:39 pm (UTC) - Expand
(no subject) - estrelladesax - Apr. 13th, 2009 05:25 pm (UTC) - Expand
#amazonfail - (Anonymous) - Apr. 13th, 2009 08:22 pm (UTC) - Expand
tigerbright
Apr. 13th, 2009 01:43 pm (UTC)
I've already linked to you in replies to others. :)
kenmarable
Apr. 13th, 2009 02:24 pm (UTC)
Just goes to show you, never release code or data changes over a holiday weekend.

"But fewer users will be using the system..."

*sigh*


popelizbet
Apr. 13th, 2009 05:46 pm (UTC)
This stuff started in August 2008 for erotic fiction writers; Mark Probst's book apparently got delisted in February.

I'm not buying "glitch". Otherwise why were pro-gay books delisted but anti-gay material still available?
(no subject) - caudelac - Apr. 13th, 2009 06:08 pm (UTC) - Expand
(no subject) - popelizbet - Apr. 13th, 2009 06:10 pm (UTC) - Expand
(no subject) - caudelac - Apr. 13th, 2009 06:18 pm (UTC) - Expand
(no subject) - popelizbet - Apr. 13th, 2009 06:22 pm (UTC) - Expand
(no subject) - caudelac - Apr. 13th, 2009 07:19 pm (UTC) - Expand
(no subject) - popelizbet - Apr. 13th, 2009 07:23 pm (UTC) - Expand
johnpalmer
Apr. 13th, 2009 02:44 pm (UTC)
Well, if you have read Amazon CTO Werner Vogel's blog or seen any of his recent presentations, you'll definitely be (like me) inclined to "cock-up over conspiracy"

It's Monday morning, and I spent way too long trying to figure out how one "cock(s) up over conpsiracy" before I realized what you'd meant.
rabidsamfan
Apr. 13th, 2009 02:44 pm (UTC)
A possible explanation, sure, but it wouldn't have worked if some management level policy setter hadn't agreed that it would be okay to a) block rankings on some material and b) block unranked items from the front page search.

That takes it past glitch and into the arena of fail. If Amazon wants the statistic "Amazon Rank" to mean something, it cannot be applied to only a bowdlerized portion of the database. And blocking people from being able to search in the easiest fashion for material you want to sell is just plain stupid.
kshandra
Apr. 13th, 2009 02:52 pm (UTC)
In comments on a post putting forward a trolling theory that I don't quite buy, eldestmuse said something that ties into this, IMO:

Even if they aren't deliberately targeting the LGBT audience, the fact remains that the philosophy behind putting the feature to remove the sales ranks of bestselling authors, thus hurting their livelihood based on morality instead of popularity, based on being "adult" is fundamentally objectionable.
(no subject) - rabidsamfan - Apr. 13th, 2009 02:58 pm (UTC) - Expand
jane_dark
Apr. 13th, 2009 02:48 pm (UTC)
What you're saying makes sense, though I find it pretty troubling that so little of the hetero-adult/erotic content has been targeted, and that makes it a bit harder for me to accept that this is a well-intentioned glitch. Bestiality is okay, John Barrowman is not?

At best, a few individuals in the system of developer checks really screwed up.

I'm waiting with interest to see Amazon's official response -- with names attached, and on their website, as opposed to reported by WaPo or Publishers Weekly.
(Anonymous)
Apr. 13th, 2009 04:13 pm (UTC)
So, you think it is okay for books with erotic content (including great works of English literature like _Lady Chatterley's Lover_ and _Little Birds_) to be "hidden" from view just because they are about (eek!) sex? Please!
(no subject) - jane_dark - Apr. 13th, 2009 04:29 pm (UTC) - Expand
pingback_bot
Apr. 13th, 2009 03:14 pm (UTC)
#amazonfail 2
User briarwood referenced to your post from #amazonfail 2 saying: [...] the story and Amazon told them the same steaming pile: it's "a glitch". Meanwhile posted about how this happened from a technical standpoint none of which does anything to make the investigations done by Dear Authorany less damning. ... [...]
(Anonymous)
Apr. 13th, 2009 03:15 pm (UTC)
ironic
ironically, this is what happens when information processing and distribution is managed by corporate boobs.
masgramondou
Apr. 13th, 2009 03:36 pm (UTC)
I think that, as I wrote in this post you are basically correct. The problem is that I think the whole thing is in fact symptomatic of some major underlying problems regarding scalability and (now that I've thought about it some more) some fundamental assumptions about (meta)data which have turned out to be flat out wrong.
(Anonymous)
Apr. 13th, 2009 03:40 pm (UTC)
Very recognizable and plausible, but...
I'm also a developer, and I've also managed developers (still am), and I distinctly recall having a name for those kind of scenarios:

Incompetence.

Which by the way still doesn't excuse Amazon's lack of response (a simple "this is clearly not our intent, please hold on while we look into it" would have sufficed).

But most of all, the fact that the culture within Amazon is such that it is possible to implement an automated system that quietly, without informing either the public or the authors and publishers, makes books disappear from searches and rankings is A Very Bad Thing. Whether it was a management decision or a developers decision, whether the current result was intentional or not doesn't really matter.

It's not the fact that Amazon hasn't rolled back these changes yet that is worrying. It's that they've shown no intent whatsoever to do that, or even acknowledge that it might be wrong.
conuly
Apr. 13th, 2009 07:09 pm (UTC)
Re: Very recognizable and plausible, but...
a simple "this is clearly not our intent, please hold on while we look into it" would have sufficed

Exactly. It might not have made any of us happy, but at least we would not have all felt insulted by that.

It's not the fact that Amazon hasn't rolled back these changes yet that is worrying. It's that they've shown no intent whatsoever to do that, or even acknowledge that it might be wrong.

Agreed.
(Anonymous)
Apr. 13th, 2009 03:43 pm (UTC)
Disability books affected also
It's not only GLBT books being affected. Some disability books also have been affected. Details at:

http://textualfury.wordpress.com/2009/04/12/amazonfail/
http://lisybabe.blogspot.com/2009/04/amazonfail.html


I've now posted the following text as a tweet. Others may please feel free to copy/paste this text as their own tweets also. It links to the above two blog posts on the disability angle of this mess. The hashtags are to help ensure that your tweet will show up more easily under certain keyword searches in twitter.com.

#amazonfail on #disability too. Please RT both http://is.gd/s5I6 and http://is.gd/s5Xr #glitchmyass #glbt #disabled

damsel_ophelia
Apr. 13th, 2009 04:22 pm (UTC)
Re: Disability books affected also
Rebroadcasted. Not getting as much press, but just as important - and sickening.
Re: Disability books affected also - essaying - Apr. 13th, 2009 08:44 pm (UTC) - Expand
pingback_bot
Apr. 13th, 2009 03:44 pm (UTC)
User boji referenced to your post from saying: [...] as discovered by who did indeed phone. Oh and more on how such a tech-clusterfuck can happen: h e r e. Best places to watch this are still on twitter at #amazonfail & #glitchmyass. From there I learnt that Heather has two Mommies has ... [...]
lyorn
Apr. 13th, 2009 04:48 pm (UTC)
(here via makinglight)

I have worked in projects like that, and yes, that is the way shit happens, and can't be stopped from rolling downhill quickly.

One item still boggles me: It seems likely that at the beginning of all this was the attitude that it's OK to fudge statistics without informing people who might use them. Of course, amazon sales rank might be more voodoo than statistics anyway...
pingback_bot
Apr. 13th, 2009 04:58 pm (UTC)
Amazonfail: Day Two
User afterthree referenced to your post from Amazonfail: Day Two saying: [...] study. And if you think this is a quick fix and are wondering why everything's still de-ranked, explains why it isn't. [...]
amberdine
Apr. 13th, 2009 05:19 pm (UTC)
Thank you, yes. First sensible theory I've heard. Pretty much was my guess, too, though I wasn't sure how the developers worked.

I have some dealings with Amazon as a reviewer and there are loads of completely innocuous things their database screws up regularly.
(Anonymous)
Apr. 13th, 2009 05:52 pm (UTC)
not just erotica, adult content or GLBTQ material being deranked
Having pored over the lists of deranked items and having tested out different points of access for page rankings myself, I think that this idea, while plausible as an explanation for "designers = numbnuts" doesn't explain why material that is not adult, not glbt related, and not erotic/porn has also been delisted: material about sex education for teens, histories, feminist theory, award winning novelists whose work is made into oscar-winning films...
Either there is deliberate internal censorship of some kind, or a caving-in to the book-burners because there is too much patently offensive and adult material that is still ranked: dog-fighting, Playboy centerfields, Mein Kampf....
If there were a tag logic to what has been deranked other than "standard right-wing fundamentalist agenda," it could be understood as a "glitch" of the kind you describe. Unless some mid-level designers are trying to game the rankings, from within? But that does not explain the Customer Service responses going back to February.
Sorry, your explanation does not hold up to more extensive research on the actual items being deranked.
davidkevin
Apr. 13th, 2009 06:37 pm (UTC)

The most likely explanation, it seems to me: after some testing going back to February, Amazon.com's content rating system was deliberately, massively gamed over western Christianity Easter weekend by one or more anti-gay rights groups.

Amazon had no reason to suddenly want to censor gay-written or gay-content books, but like script-kiddies and Russian mafia, these creeps found a software weakness to exploit.

ksol1460
Apr. 13th, 2009 08:41 pm (UTC)
But if this were true, why amazon didn't simply release a statement saying "sorry, we were hacked", fix it, and go back to normal?
(no subject) - ravan - Apr. 13th, 2009 11:33 pm (UTC) - Expand
this just in - ksol1460 - Apr. 13th, 2009 09:04 pm (UTC) - Expand
emeraldsedai
Apr. 13th, 2009 06:39 pm (UTC)
Neil Gaiman just referenced this post on his blog, so standby for traffic.

Still waiting for Amazon to SAY SOMETHING. I don't need a complete or even technical explanation. But some kind of, "Wow, we're aware of the problem and are working to fix it as fast as we can because we value all our writers, publishers and customers a whole lot" would be good.
duck_or_rabbit
Apr. 13th, 2009 06:39 pm (UTC)
You were cited on Neil Gaiman's blog, which is how I found you. Interesting to see how things shake out and thanks for this.

If it is technology, not people behind this deranking, the least I'd like to see is an apology. Can't restore trust without that. Ought to come fast, too.
conuly
Apr. 13th, 2009 07:47 pm (UTC)
Ought to have already come, back when they made that glitch comment.
azdaja_dafema
Apr. 13th, 2009 07:12 pm (UTC)
You're right - whether by accident or design I've seen sites get destroyed by code changes with unforeseen consequences, I've /made/ code change errors that have taken me hours to fix - heaven forbid what would happen to a site that complicated. The timing is awkward for them and they'll have management breathing down their backs, so I'm assuming they'll get it sorted within a week. Whether the PR ever comes back is debatable though.
wyrd_sane
Apr. 13th, 2009 07:19 pm (UTC)
I say we all head over to the Amazon PR phone lines and help their day be as busy as possible.

--
Furry cows moo and decompress.
pingback_bot
Apr. 13th, 2009 07:44 pm (UTC)
More posts on the subject
User conuly referenced to your post from More posts on the subject saying: [...] Here's a coherent explanation of why things haven't been FIXED yet. Not sure how accurate it is, but I'm trying to post interesting links, anyway. This link is telling how customer service messed up and why it's totally not their fault.... [...]
loonyemi
Apr. 13th, 2009 07:44 pm (UTC)
Thank you.

What a nightmare. I feel so sorry for those PR guys. The Internet was the worst thing ever to happen to that arm of corporate employ.
mouseworks
Apr. 13th, 2009 07:49 pm (UTC)
The comment-bombing theory is bogus
My publisher confirmed that she only categorized my book, Centuries Ago and Very Fast, as sf and glbt, so all the suggested tags that Amazon came up with were generated by Amazon staff or contract workers (one of them strongly implies that someone at Amazon at least skimmed the book because there's nothing in the promotional material to suggest a "gay cop" lurks inside the book.

Amazon.co.jp has an age check page in English and Japanese if a person searches for a glbt item from their main pages (English is available).

My guess is that Amazon did do some automation without checking to see how many books would be caught in delisting glbt and erotica books.

CA&VF hadn't been included with my other books when searching on my name until this weekend after complaints from me (and probably also from my publisher. This was the case for over a month.
(Anonymous)
Apr. 14th, 2009 07:27 pm (UTC)
Re: The comment-bombing theory is bogus
My guess is that Amazon did do some automation without checking to see how many books would be caught in delisting glbt and erotica books.

Which would suggest that someone at Amazon hasn't yet caught up with the idea that homo-/bi-/trans- sexuality pertains to more than copulation.
pingback_bot
Apr. 13th, 2009 07:58 pm (UTC)
Update on the #AmazonFail
User sheherazahde referenced to your post from Update on the #AmazonFail saying: [...] a few meetings, and come up with a solution. The problem isn't likely to be fixed overnight. Why hasn't Amazon fixed things overnight? by Dear Jeff Bezos: Let's be Adults, Shall We? by Susie Bright Amazon 'Glitch' Removes Sales ... [...]
Page 1 of 2
<<[1] [2] >>
( 75 comments )

Latest Month

September 2014
S M T W T F S
 123456
78910111213
14151617181920
21222324252627
282930    
Powered by LiveJournal.com
Designed by Tiffany Chow