I am taking a break before writing Part 3 of The Fundamentals of base calling where I will be discussing on what technically needs to be addressed to improve the raw accuracy of the Ion Torrent. Stay tuned especially if you want the C/C++ example source code. This is the first post of a 2 part series on the Life Grand Challenge. The first part involves criticism because there must be something they are doing wrong, you would think that every nerd and his imaginary girlfriend would be working on this problem given the 1 million dollar prize. The second part are suggestions and discussions using the successful NetFlix crowd sourcing model. Before I start with the hater stuff, I would like to say that engaging the talent of the scientific community is a good thing. It recognizes that there exists smart and innovative people outside the Broad, Sanger, BGI, etc. and their opinions are important. Something Illumina and Roche must learn! Being in these large sequencing centers breeds laziness and unaccountability, while labs on a shoe string budget must be very resourceful and innovative in their thinking and savvy in their spending to compete with the big boys for high impact publications. Anyways enough ranting and on to the topic.
This idea of crowd sourcing (not to be confused with crowd surfing) is the best thing since incorporating laws in nature that allow nuclear fusion to occur, allowing a sun that sustains life OR the best thing since sliced bread. This business model allows corporations to hire a limitless supply of talent from around the world without paying them unless they achieve the goal. This is even WORSE idea than outsourcing to India and China as it uses greed to take advantage of the WHOLE world!
The Life Grand Challenge is a crowd sourcing initiative launched by Life Technologies just before the Easter holidays. It is an innovative way of engaging the scientific community in solving at present, 3 Ion Torrent improvements:
The 4th challenge relates to the SoLiD sequencing platform. Not quite 7 at the moment so there are reporters out there that really need to use their fingers to count. These challenges are hosted on the Innocentive Pavillion (i.e. website). I will now concentrate all discussion on the Accuracy challenge which is purely computational or that’s what it seems. The way it works is that for every quarter a new internal benchmark is released and challengers aim to better that benchmark by first demonstrating it on the data set provided. The data set is an E. coli sequencing run, surprise surprise ! The performance benchmark is achieved when a challenger achieves the same amount of 100Q23 reads as that of 100Q20 reads by the ion-Analysis program.
- First Quarter benchmark – 366,880 100Q23 reads (from total 753,210 reads)
- Second Quarter benchmark – 632,468 100Q23 reads (from total 874,282 reads)
In the first quarter of the competition (mid April – mid June), despite the number of “Innocentive Active Solvers”, there were no submissions to Innocentive (to my knowledge) even ones that did not reach the benchmark. We are currently in the second quarter of the challenge, which ends September 15th then Life Technologies will release a new benchmark target at the start of Third quarter of the competition. There are currently no submissions that appear on the Leaderboard. Either the technology and chemistry has improved in one quarter OR the problem just got twice as hard as the number of total reads did not increase that much.
How many ACTIVE accuracy challengers are there?
There are two important factors in all competitions, difficulty and number and quality of your opposition? I have outlined above how difficult the problem is. Currently there are 541 “active solvers” according to Innocentive. This is not a measure of how many active accuracy challengers out there but merely the number of people who have clicked on the terms and conditions so they can view the competition details. For example, I am an active manuscript writer. Every day I intend to start my manuscript but never get around to it 🙂 Also in a recent press release, the claim was that there were 500 Life Grand Challengers. Many of these would be Accuracy challengers as the other two challenges require a PGM. Again it has not distinguished between active and the Innocentive count.
To be active in the grand challenge, the challenger must have done the following
- Join the Ion Community – TorrentDev space – 54 active out of 2694 possible people. Less than a handful have discussed Challenge specific topics.
- Read the technical specs on the Raw Image Acquistion File Format (DAT file) – less than 60 views on this document and most was actually me returning to it!
- Downloaded the latest source code (optional) – 35 views for latest. Earlier version has 500+ views.
You can be the judge based on the numbers above, how many people are actively competing, however it would be IMPOSSIBLE to do without at least joining the Ion Community to get either the file format specification or the source code. Although previous versions of the source code (i.e. Torrent Suite v1.3) had a high number views, I don’ t think many did much after viewing or even downloading. If they did you would expect the script kiddies to complain that they can’t get the code to compile, mainly because there were bits missing at each initial release. I have been the only one to correct them on this because yes I have downloaded and unpacked AND COMPILED the source code! In the two not so important source packages there are still bits missing.
From the point of view of Life Technologies/Ion Torrent
Normally a business would hire or allocate a team of people to achieve a goal and needs to pay a salary regardless of if they achieve the goal or not. In other words, if you are working on this problem, you are better off going for a job with Ion Torrent. Also these team of people take sick days, annual leave, go through divorces have psychological problems and usually some don’t get along with each other. In other words everyone is a potential HR nightmare.
In addition, R & D staff on Ion Torrent have at least a 2 years head start on everyone so having a moving target that resets every 3 months… come on the only people that are going to beat Ion Torrent are crazy Russian geniuses…. who are most probably doing crazy things instead. Not to mention staff probably get a bonus every quarter every time they make the target that much harder to achieve by the community, which is what they are employed to do.
From the point of view of the Challengers
First and foremost you are already behind the eight ball as you are competing with people with a two year head start and unlimited access to massive data sets and the latest planned hardware and chemistry improvement. Having access to code is great… code with limited comments and stuff there for historical reasons doesn’t really help with understanding it fast. Also there is a lot of code to decipher. Some may say why bother looking at the code? My response you need to know what paths have been taken first before you dedicate weeks of your time developing something else.. because you may find you have developed something that was already in the code and worse still it is a crappier version.Secondly, you probably have a day job and family/friends so not much free time. Again you are competing with people in Ion Torrent who do this during the day as they are employed to do this. Lastly, you try a new technique that takes 2 months to implement but it doesn’t work. What are you left with? A pissed off wife, that you didn’t spend time with the family in your free time OR if you are a student, time you could have spent impressing girls on Dance Dance Revolution! For those that are thinking of taking up the challenge. I can tell you from my experience it doesn’t impress girls and doesn’t make you more attractive to them which is a shame because in theory it should LOLZ. On a serious note, the optimal play I have analyzed is to be employed by Ion Torrent because at least if your ideas don’t work out…. you still have money in your bank account.
Now that I’m done with the harsh criticism, the next blog post (Part 2) will discuss what the Life Grand Challenge initiative can learn from the successful NetFlix crowd sourcing model and including my 2.2 cents (GST inc) worth.
Disclaimer: For the good of all mankind! This is purely my opinion and interpretations. I have tried my best to keep all analyses correct. I am also off the angry pills.