What Book Discovery Is Missing

17th October 2013

I've just read 20 books about vampires (no, not really, but let's run with this). Guess what I'd really like to read next? If Amazon and most other prominent book discovery engines are to be believed, what I'm really after is yet another book about vampires, ideally by the same authors I've already read. Because reading almost the exact same book over and over is what everybody wants, isn't it?

There is some safety in recommending books based on similar characteristics and authors, because of course these elements correlate somewhat with how much a book is likely to be enjoyed. But for me, that's not the point of discovery. If I'm already familiar with an author it's easy enough to find their other books, or if I'm reading a franchise series it's not difficult to find other works in it.

I've posted before about the problems of surfacing the same kind of content all the time. But there's something more insidious going on than a narrowing of our culture. What we're experiencing is a decimation of the midlist, and a growing gap between midlist authors and a small group of bestselling authors. It's winner takes all.

Juliet E. McKenna has blogged about this in relation to Waterstones' recent decision to sell toys, further reducing the shelf space for books. Juliet writes: "I'm an author who's seeing her income eroded year on year by changes in the industry I can do nothing about. "

And in the comments: "What figures on booksales holding up don't show is that the gap between the comparatively few authors who sell shedloads and the rest, who sell less and less, is getting wider and wider. Which is a problem for the industry in the longer term, because best-sellers always came out of the midlist, thanks to reader word of mouth and booksellers handselling. That's how JK Rowling got started with Harry Potter - and any number of other examples besides."

Fewer places to experiment

At the same time as high street booksellers are contracting, libraries are under unprecedented attack. I've been campaigning for Save Lincolnshire Libraries, because in Lincolnshire we're facing cuts to two thirds of the static libraries, not to mention many of the mobile services. Library closures like these are threatened around the country because of the Tory insistence on stripping local authorities of funding. That's another avenue of discovery closing, because libraries are one of those places that readers can take a gamble on new authors. And one thing that's interesting about libraries is that their annual lists of most-borrowed books are often very different from the bestseller charts.

So we have a situation where there are fewer outlets for authors, and these are heavily influenced by what's selling well on Amazon. Even when you factor in alternative discovery engines like Bookish and Goodreads, these are also skewed in favour of recommending books that are already popular. With Goodreads, that's because like Amazon it depends on the number of star ratings, and with Bookish it's because similarity, bestsellerdom and who the author is form a part of its recommendation algorithm.

What we need is new discovery tools that take no account of popularity or sales, and yet improve on the recommendation algorithms that already exist.

How to fix discovery

My last round of submitting a manuscript to various literary agents was very instructive. Whilst the majority had guidelines that specified which genres they wanted to read and which they didn't, a significant minority didn't. These eclectic readers said things like, "I don't know what I want, but I'll know it if I see it". Book suggestions based on similar themes and genres are no use for this kind of genre-agnostic reader.

Similarly, aggregate star ratings aren't all that useful, I've found, and again they tend to result in the kind of suggestions that everyone has already heard of anyway.

What's needed is a way to make suggestions that take account of a reader's need for novelty, and progression. Our reading tastes evolve throughout our lives, and even as adults our reading levels will tend to subtly increase the more we read. One approach could be to classify all books, including adult books, by reading age, and to suggest books of slightly higher reading levels than those a reader has enjoyed. I'm talking here about a search engine that would examine the full text of a book and figure out the reading level. I'm aware that some ways of doing this can be inaccurate, but there's scope for refining this process.

Readers can also be categorised into different types that have nothing to do with genre: those who like long or short books, fast or slow paced ones, or graphic-heavy, or books with no swearing. It's possible to dig out a lot of information by using a computer programme to analyse these aspects of a novel.

This close examination of text is similar to what Booklamp are doing with their Book Genome Project. It's worth a look, because it's different from any other discovery system I've come across so far.

Booklamp are barking up the wrong tree, however, because their project is heavily focused on looking for similarities between books, particularly when it comes to themes. I don't think these are the best indicators of how enjoyable a book is going to be, although it may come in handy for anyone researching a certain topic in non-fiction. But I feel they are edging closer to the right answer. What's missing is a detailed study of which factors actually make a difference to people's reading enjoyment from amongst the many different aspects of writing style. And, crucially, although we know one size doesn't fit all when it comes to reading, can we usefully sort readers into different types other than genre (or as well as), and use that to accurately predict which books they will love? For instance, some people skim read whilst others prefer to take their time with a story. Will one group like short, densely-written texts with a higher reading age and less repetition, compared with the other group?

I'm hopeful of a future where most books are getting four or five stars, thanks to improvements in the way they are recommended to people. But then, I'm a signed-up, card-carrying daydreamer.

© Ros Jackson