?

Log in

No account? Create an account

Previous Entry | Next Entry

Why hasn't Amazon fixed things overnight?

Well, if you have read Amazon CTO Werner Vogel's blog or seen any of his recent presentations, you'll definitely be (like me) inclined to "cock-up over conspiracy" as the explanation for the current shambles.

So why have things gone this desperately wrong this quickly?

The simple answer is Amazon's architecture. It's highly distributed, and there's no operations team. Each component (and over 200 go into a single page) is run by its development team, of four to five people. They are responsible for its features, its development - and for making sure it runs effectively. The result should be a company that can move quickly in response to outside events.

At least that's the theory.

I'm afraid the real world doesn't work like that. I've been a developer and I've managed developers and I can tell you that what really happens is something like this:

Someone comes up with a neat idea that they evangelise among the other developers, and it gets added to the platform. The developers become wedded to their idea, and they keep adding features. Something from the outside occurs that affects the data managed by the service, and they don't notice. After all, it's their design and it's perfect. The problem gets worse, and a few external symptoms are noted and passed on to the developers. They're too busy to pay much attention to them, and so they ignore them. Then suddenly, BANG, and everything breaks.

Oh, and it's a holiday weekend and there's no one there to actually handle the problem as the whole team's gone off on a skiing trip.

Now I can't guarantee that's what has happened with the deletion of GLBT content from the Amazon ratings system, but I suspect it's more likely than not.

So here's where my conjecture comes in:

Someone probably had the idea of reducing Amazon's exposure to bad publicity without increasing the site's legal liability. Manual censorship of the rankings would certainly make the service more liable, so the idea was probably a tool that would let the site's users do the work for it. After all, if the community doesn't like it, then, well, US community standards laws apply and you're safe. A group of developers coded it up, and it worked well - for a while.

Either a parameter wasn't quite right, or someone released a new version of a keyword file without testing - and, well, suddenly the GLBT books were off the list. Maybe someone gamed the system, too - it's impossible to tell from outside.

A separate test and operations team would have been likely to spot the underlying flaw before it got released - or at least spotted the first wave of complaints and started to triage them effectively, with a more productive response than "It's a glitch".

So now Amazon has to unwind data that's spread across its distributed application platform, which may be stored in any or all of three different kinds of database, and in at least three different geographies and many more data centres.

Ooops.

That's going to take a while to deal with.

Meanwhile their Seattle-based PR team is just about to start a very long day - and a group of developers are going to be desperately trying to explain just went wrong.

[ETA 23/4/2012. After three years of this post being targeted heavily by spammers, I have locked commenting.]

Comments

( 74 comments )
Page 2 of 3
<<[1] [2] [3] >>
(Anonymous)
Apr. 13th, 2009 05:52 pm (UTC)
not just erotica, adult content or GLBTQ material being deranked
Having pored over the lists of deranked items and having tested out different points of access for page rankings myself, I think that this idea, while plausible as an explanation for "designers = numbnuts" doesn't explain why material that is not adult, not glbt related, and not erotic/porn has also been delisted: material about sex education for teens, histories, feminist theory, award winning novelists whose work is made into oscar-winning films...
Either there is deliberate internal censorship of some kind, or a caving-in to the book-burners because there is too much patently offensive and adult material that is still ranked: dog-fighting, Playboy centerfields, Mein Kampf....
If there were a tag logic to what has been deranked other than "standard right-wing fundamentalist agenda," it could be understood as a "glitch" of the kind you describe. Unless some mid-level designers are trying to game the rankings, from within? But that does not explain the Customer Service responses going back to February.
Sorry, your explanation does not hold up to more extensive research on the actual items being deranked.
davidkevin
Apr. 13th, 2009 06:37 pm (UTC)

The most likely explanation, it seems to me: after some testing going back to February, Amazon.com's content rating system was deliberately, massively gamed over western Christianity Easter weekend by one or more anti-gay rights groups.

Amazon had no reason to suddenly want to censor gay-written or gay-content books, but like script-kiddies and Russian mafia, these creeps found a software weakness to exploit.

ksol1460
Apr. 13th, 2009 08:41 pm (UTC)
But if this were true, why amazon didn't simply release a statement saying "sorry, we were hacked", fix it, and go back to normal?
(no subject) - ravan - Apr. 13th, 2009 11:33 pm (UTC) - Expand
this just in - ksol1460 - Apr. 13th, 2009 09:04 pm (UTC) - Expand
emeraldsedai
Apr. 13th, 2009 06:39 pm (UTC)
Neil Gaiman just referenced this post on his blog, so standby for traffic.

Still waiting for Amazon to SAY SOMETHING. I don't need a complete or even technical explanation. But some kind of, "Wow, we're aware of the problem and are working to fix it as fast as we can because we value all our writers, publishers and customers a whole lot" would be good.
duck_or_rabbit
Apr. 13th, 2009 06:39 pm (UTC)
You were cited on Neil Gaiman's blog, which is how I found you. Interesting to see how things shake out and thanks for this.

If it is technology, not people behind this deranking, the least I'd like to see is an apology. Can't restore trust without that. Ought to come fast, too.
conuly
Apr. 13th, 2009 07:47 pm (UTC)
Ought to have already come, back when they made that glitch comment.
azdaja_dafema
Apr. 13th, 2009 07:12 pm (UTC)
You're right - whether by accident or design I've seen sites get destroyed by code changes with unforeseen consequences, I've /made/ code change errors that have taken me hours to fix - heaven forbid what would happen to a site that complicated. The timing is awkward for them and they'll have management breathing down their backs, so I'm assuming they'll get it sorted within a week. Whether the PR ever comes back is debatable though.
wyrd_sane
Apr. 13th, 2009 07:19 pm (UTC)
I say we all head over to the Amazon PR phone lines and help their day be as busy as possible.

--
Furry cows moo and decompress.
pingback_bot
Apr. 13th, 2009 07:44 pm (UTC)
More posts on the subject
User conuly referenced to your post from More posts on the subject saying: [...] Here's a coherent explanation of why things haven't been FIXED yet. Not sure how accurate it is, but I'm trying to post interesting links, anyway. This link is telling how customer service messed up and why it's totally not their fault.... [...]
loonyemi
Apr. 13th, 2009 07:44 pm (UTC)
Thank you.

What a nightmare. I feel so sorry for those PR guys. The Internet was the worst thing ever to happen to that arm of corporate employ.
(Deleted comment)
(Anonymous)
Apr. 14th, 2009 07:27 pm (UTC)
Re: The comment-bombing theory is bogus
My guess is that Amazon did do some automation without checking to see how many books would be caught in delisting glbt and erotica books.

Which would suggest that someone at Amazon hasn't yet caught up with the idea that homo-/bi-/trans- sexuality pertains to more than copulation.
pingback_bot
Apr. 13th, 2009 07:58 pm (UTC)
Update on the #AmazonFail
User sheherazahde referenced to your post from Update on the #AmazonFail saying: [...] a few meetings, and come up with a solution. The problem isn't likely to be fixed overnight. Why hasn't Amazon fixed things overnight? by Dear Jeff Bezos: Let's be Adults, Shall We? by Susie Bright Amazon 'Glitch' Removes Sales ... [...]
pingback_bot
Apr. 13th, 2009 08:59 pm (UTC)
AmazonFail Continued
User ladyelleth referenced to your post from AmazonFail Continued saying: [...] hack-fu and don't understand it. describes the sequence of events that's currently happening. on why this hasn't been fixed overnight. offers another well-written post on the issue. blogs about AmazonFail. Feministing has word ... [...]
alexandriabrown
Apr. 13th, 2009 10:15 pm (UTC)
Thank you for this explanation from the technical side. I was inclined to believe the cock up version when I started to do my on investigation and realized that the results were a. not consistently replicable and b. completely random. By that I mean works that were both queer and *clearly* adult were showing up with rankings/were searchable/were on the rankings lists.

That's not to say that Amazon didn't mess up, it did, but that it was a different mess up than what was being reported.
silrini
Apr. 13th, 2009 11:39 pm (UTC)
THANK YOU. I love it when people actually explain what the problem might be and that it might not be what everyone is screaming about. 'Course, that said, there probably is something vaguely censor-like going on there(I don't like to commit myself until I really know what's going on), and that needs to be dealt with. Thank you for this calm analysis of the probable situation. ^__^
slyfoot
Apr. 14th, 2009 07:34 am (UTC)
As a former database developer, I could definitely see something like this happening with a large Oracle database, only with Oracle a rollback wasn't all that difficult. I am inclined to think this is a case of Hanlon's Razor. I find it a bit too much of a stretch to believe that Amazon has something against literature involving sex with disabled people. I wish everybody who actually *knows* that this sort of cockup is possible would also point out that the main catalyst for the whole #amazonfail hashtag doesn't think there was any malicious intent.
pingback_bot
Apr. 15th, 2009 03:28 pm (UTC)
Another take on the whole GLBTQ Amazon fiasco
User natf referenced to your post from Another take on the whole GLBTQ Amazon fiasco saying: [...] 's take on the saga: Technology, Books and Other Neat Stuff - Why hasn't Amazon fixed things overnight? I think he may be right. As an ex-developer / ex-programmer myself, I can see that his scenario is totally possible - even probable! ... [...]
Page 2 of 3
<<[1] [2] [3] >>
( 74 comments )