Digg announced several new features today, the most interesting being a new category of podcasts. As Digg say on their blog:

Now you can Digg your favorite podcast series and individual podcast episodes. Not only can you see a list of the most popular podcasts by section, you can also dive into any individual podcasts to see the most Dugg individual episodes. And don’t forget - every time you Digg a podcast or podcast episode that is bookmarked in your profile and shared with your friends.

What this tries to solve

This may fill a niche; in my opinion there are few, if any, places on the internet at the moment where discoverability of good podcasts which are outside the realms of tens of thousands of listeners is possible. Odeo tries to solve this problem, with their ‘recently starred’ section on their homepage. However, that’s too random to be of any use. Pluggd is trying, too, but a quick glance through the upper echelons of their directory merely reveals podcasts I have already heard of. As far as I can tell the benefit Digg brings to this, which could potentially solve the problem, is the time-critical nature of its social bookmarking. In other words, the top of the directory will hopefully be different every time I look at it, because of the site’s basis in social news. This increases the chance that there will be something new which I haven’t seen before, but which the community has judged is interesting, presented to me.

However, I fear that whilst this may be the most effective attempt at solving this problem so far, it still doesn’t take an approach which stands a real chance of helping me find that gem of content which only appeals to a very few people. Read on after the jump for why, and what I think the solution might be…

Getting down the tail

The phenomenon which Chris Anderson called the Long Tail (in the context of media, essentially that anyone can now create and publish their own content, resulting in a far greater quantity of content with each item potentially able to reach some sort of audience) has meant that there is likely to be the perfect podcast or blog out there, just ‘made’ for me, covering exactly the subjects that I am interested in. However, it may only appeal to a very small number of people, perhaps less than 100.

Digg fails

An example: A plumber starts a podcast about domestic plumbing; he talks about the latest tips, new products, funny anecdotes. A really well put together podcast, of great interest to all plumbers but of no interest to anyone not a plumber. Our podcasting plumber submits his podcast to Digg to promote it. Just by chance, 10 of the Digg members who are plumbers, and the podcasting plumber’s friend, who isn’t a plumber, find the podcast in the depths of the site and digg it, delighted that they have found such an interesting podcast.

So those 10 plumbers have found the new podcast. Great. But the podcast didn’t make it onto the front page, because 10 diggs isn’t enough to do that. That means that the rest of the plumbers who watch Digg, but don’t have time to look any further than the front page, didn’t find the podcast, which means that that podcast has only reached a tiny fraction of it’s (tiny) potential audience.

We can see that a new system is needed - the Digg model doesn’t work.

One possibility

There are two options, then, which could have worked. If there were such a site as ‘Plugg’ (Digg for Plumbers - get it!), to which a reasonable percentage of the potential audience of this plumbing podcast were subscribed. So, to widen this out from the example, someone within every ‘microcommunity’ (in this case plumbers) could set up a Digg-like site for the community to recommend content to itself. The problem with this model is that the audience could potentially be too small to effectively prevent gaming. The podcasting plumber would only have to pay 5 friends to get his podcast to the front page of ‘Plugg’ if he wanted to cheat, rather than 100 or so for Digg.

My preferred option

The other option is a robust recommendation algorithm. We’ve seen this sort of thing for websites with Stumbleupon, and for music with Pandora. To use my example, the podcasting plumber submits his podcast to this service, which has the same broad audience as Digg. Those 10 plumbers find the podcast in the depths of the system, and recommend it, because they find it interesting. The algorithm then first sees what those 10 have in common, and notices that they all liked a plumbing blog that was submitted a while ago. So it works out that this podcast probably appeals to everyone that that plumbing blog appealed to, so it automatically suggests it to everyone who liked that blog. Whilst it would obviously be more complicated, you get the sort of approach I am suggesting. If this were technologically possible, and it could gain traction (remember, you have to have those 10 plumbers who find the podcast without it being recommended to them, and you also have to keep people patient whilst the recommendation engine works out what they like) then it would seem that this would be the superior solution.

Digg is excellent, but it takes the one-size-fits-all approach. It is valid, and it is genuinely useful. However, it can’t and doesn’t solve all our content discovery problems. I think there is room for another company doing something similar to what I outline above, and I hope we see one.

Apologies for the obscenely long post. I’ve stuck this post in the future challenges series, as I was planning on writing a post about this problem in that series anyway, but it made sense to do it in the context of Digg. The series talks about some of the problems which we need to solve as the web, and new media, go forward. We’ll know we’ve reached Web 3.0 when we’ve solved them!

PS: Warm welcome to Sam Davyson.