Algorithm and Blues

Image copyright Nick Gentry ©

On the eve of the election, two things I have read this week have combined in my head and I have not been able to stop thinking about them. The first thing is the excellent comment that Dave Freer left on my post earlier this week. The second is this video by music critic Chris Weingarten. The subject of these two influences – or at least the tenuous connection I have built between them – is the conflict between the benefits of technology and the tyranny of numbers.

OK, so even to me that sounds a bit dramatic. But it is true. I’ve touched on this topic before in a previous post, and I came to the conclusion that optimisation of artistic expression by algorithm may well be possible, and even useful, but it’s really bloody depressing. I still feel this way. I was at first skeptical of Dave’s explanation of how mathematical modelling of book acquisition could be possible, but he convinced me. Snip:

At the moment, you have your gut feel and the bookscan figures to decide what you buy. If you had better quality data (ie. laydown, returns, normal sales of that sub-genre and laydown within each geographic area … you could say which … would make your company more money, which had the lower risk, what was actually a reasonable ask for the books in question. It could also tell the retailer which were good bets for their area, and publisher where to push distribution. It doesn’t over-ride judgement, it just adds a tool which, when margins are thin, can make the world of difference.

I am forced to agree with Dave that if such a tool were available it would be of great use to publishers to help decide what to buy, and in a great many instances, what books would sell (if you still don’t believe it, I recommend reading the whole thing). Nonetheless, it fills me with despair. As Weingarten says in the video I linked to, most of us who got into the world of writing did so because we suck at maths. But it’s not just that. There’s a kind of ethical issue at stake here too. The availability of a tool like this would make publishers lazy. I once heard the use of test audiences for TV pilots and films described as being more about ass covering than actually predicting the success or failure of a film. And I have to say the same thought occurs to me about the statistical modelling of book acquisition.

This is not to say the information wouldn’t be useful, but it would mean that when a book that tested well in the model bombed, publishers could throw their hands up in the air and say, “Well, it tested well.” It would be a tool that sales directors and corporate executives would use to dampen creativity in publishing. Presumably (though correct me if I’m wrong, Dave!) the sales of statistical outliers that don’t easily fit into a pre-existing genre or sub-genre would not be easily predicted under this model. And there are a lot of books that don’t fit into genres. I’ve heard it said that when it comes to books there are almost as many genres as there are books. Does that mean that publishers would just use their own judgement? Or would they be even more unlikely than they already are to take on books that aren’t safe bets?

Of course, Dave will probably tell us that this amazing statistical model would only be a tool. It wouldn’t ‘override judgement’ as he says in the quote above. But humans like to rely on machines and numbers – especially when it comes to difficult decisions. Sometimes that comes at the cost of something difficult to quantify. And perhaps on this day, when the leaders of our country are trying to win an election based as much upon the statistically predicted thoughts of a few key voters in a few key marginal seats as any true leadership, beliefs, policies or moral character, I fear that ceding our decision-making to an algorithm has the potential to take away far more than it gives us. What do you think?

The Kind of the Rest – My New Novel

In my last post I wrote about a nightmarish scenario in which books we read are created automatically by a software algorithm. I’ve had time to think about it since then, and to use the wonderful TweetWriter, a promotional tool for the Melbourne Writers’ Festival. TweetWriter takes the body of writing that makes up your Twitter feed and creates a customised book based on your style of expressing yourself, which, as everyone knows, is usually at its finest in tweet form. Here’s my automagically customised back cover blurb:

When asked to described the real Joel Blacklock, Joel would often say “I am the walrus”, but 127 loyal fans will always know Joel best as the prolific writer who published over 1643 works, all written from Joel’s secluded playboy style mansion in Sydney, Australia.

In this astonishing latest work, ‘THE NAME OF THE KIND’, Blacklock’s writing is both mesmeric in tone and labrynthine in structure, and is surely destined to achieve the status of a contemporary classic.

Hear that? ‘Mesmeric in tone and labrynthine in structure’. One hundred and twenty seven fans can’t be wrong!

Let’s just say the software algorithm that will write a true bestseller is some time away. Quite aside from the fact that the back cover blurb is littered with errors and the title of the book is different on the front and back (I tend to think The Name of the Kind is better than The Kind of the Rest – thoughts?) I can’t imagine a single person would even pick this book up off the shelves.

Nonetheless, for the curious and the lazy, here are a few of Twitter’s finest minds at work.

The always adorable @stephenfry:

The redoubtable @johnbirmingham, erm I mean John Birmingha:

The terrifying @tara_moss:

What about our potential prime ministers? Here’s Red:

And the Mad Monk?

Check out your own book titles and link to them in the comments.

The Melbourne Writers’ Festival runs from 27 August to 5 September 2010.

Why Amazon Would Make a Bad Dinner Party Guest

Have you ever met one of those people at a party who within minutes seems to know your life history, sexual proclivities and history of insanity? They ask a lot of questions while at the same time manage to reveal nothing about themselves. Data miners are a scourge of the modern social gathering, and they make a lot of people uncomfortable, and for good reason – information is power. Most people aren’t comfortable with the idea that someone they barely know suddenly knows what colour underwear they are wearing. In the era of Facebook, Amazon, Apple and Google, however, the data miners have started selling us back the benefits of collecting our secrets. Amazon and Apple can recommend products to us based on what we already like, Google and Facebook read our emails and messages and serve up ads based on what we’re talking about. We take these recommendations and automatic tailoring of services for granted. It isn’t a person, after all, prying into our buying habits and personal data in order to create a profile based on our likes and dislikes. It’s an algorithm. A piece of software. No biggie, right?

Right. This is 2010! Those of us who engage in online shopping and social media have obviously at some point decided that the benefits outweigh the invasion of privacy. Maybe we don’t like to think about it very much, turn a blind eye to it to some extent, but we still want what these companies offer us. Nonetheless, we should never forget how incredibly valuable this information is. When it comes to the buying habits of readers, this data has traditionally been very difficult to collect. A book publisher once told me that the only way to afford market research in publishing is in fact to publish books. Publishers try things out by instinct. If the public likes it and the company makes money, they stick with it. If not, they discard the author or the genre as easily as they came across it. All things considered, publishers would still prefer to publish fewer books that make more money. Thankfully for the reading public, it has ordinarily been difficult to know what sells and what doesn’t. Publishers aren’t constrained by absolutes – although they might have to run their books past the gauntlet of previous sales figures, the reason many books make it out of the slush pile is that the publisher has a ‘feeling’.

The point is, in the digital age the information about what people like to read and who they are can be collected more easily than ever before. If you buy books from Amazon, Amazon knows your age, your gender, where you live, what kind of job you have, how much money you generally spend on books, what books and authors you like – they may even know why. For a multinational conglomerate, it is not that much of a stretch to collate this data and see what kind of books are working in the market right now. This, ultimately, is why publishers are terrified of companies like Amazon publishing books. Although Amazon lacks the traditional technical expertise of a publishing house, they possess this new kind of information that publishers have never had access to. What might they do with it?

Can you imagine a world in which each book is dissected based on the plot line (three acts? four?) the number of words (people nowadays really prefer only books under 100,000 words), the number of female characters and so on and so forth until publishing books for particular markets turns into a paint-by-numbers drawing. It’s not like there aren’t enough authors out there who would write anything in order to get published. Previously, it has been impossible to imagine as complex a thing as a book being understood as a collection of data. But with the data now available to retailers like Amazon and Apple on the internet, what’s to stop them using this information for more than just recommendations? And would this be a bad thing? Perhaps in this future I am imagining we will shrug off the sterilisation of our entertainment by algorithms in the same way we have shrugged off the sterilisation of our social interactions in the same way. Perhaps these new improved algorithm-based books will be so much more entertaining than regular old books that we will turn a blind eye to the process? What do you think? Is this a ridiculous sci-fi nightmare? Or is this something you can imagine playing out for real?