Rapid improvements may lead to comparison pitfalls

In a previous blog post, I showed the rapid improvements Ion Torrent has made over a short 3 month period. I assume that both the GS Junior and the Illumina MiSeq have experienced improvements also over this time. These desktop sequencers require less financial commitment and usually requires only one grant to fund. In contrast, the big toys (eg. HiSeq, SoLiD 5500) require a massive financial commitment to purchase and operate. The profit margin may be much higher for the big toys BUT not much volume will be sold. In Sydney, one hand is needed to count how many big toys out there. By selling small toys the profit margins can be reduced as larger volumes are expected to sell. This creates a much larger user community instead of 3 people in each city who have observed a “sequencer in the wild”. The best analogy would be the personal computer (PC) revolution largely replacing the need for mainframes. I believe that the affordable costs of all three technologies will push the biotech industry to be more innovative and cost effective. No matter what sequencer you buy all customers will enjoy the benefits of this healthy competition.

The strength of computational analysis (including bioinformatics) is the relative ease of reproducibility given a data set and method. This is made even more easier, if the method involves using software with all options/setting specified. No other science offers the same warm fuzzy feeling of reproducibility. The greatest thing that has come out of the “Sequencing Wars” is the public release of raw data sets to accompany the application notes. This is the first time EVER I have seen a biotech company say “You don’t believe us? Here’s the raw data analyze it yourself!”. This is extremely bold move that has been taken first by Ion Torrent then shortly followed by Illumina MiSeq. For once, this empowers the customers to make their own conclusions! Many kudos to Ion Torrent and Illumina for making their raw data available. I am yet to find any public raw data for the GS Junior. In fact, Roche appears to be behind Life Technologies and Illumina when it comes to embracing the information age. Come on guys, pick up your act and get rid of them dinosaurs in upper management!

There is a long list of the technicalities and nuances with comparing the competing desktop sequencers. This problem is not unique when comparing different technologies. The unique issue comes from the pitfalls resulting from the rapid improvements. This is not like comparing two different car models or laptop models where their performance is rather static before and after purchase. In contrast, sequencing technology purchased can further improve through the consumables themselves and software versions for data analysis. The point of my previous post was to highlight this fact. Therefore, it is extremely important to assess when a data set was analyzed as more rapidly evolving technologies may be extremely disadvantaged. Showing the performance of a sequencing technology from 2011 Q2 (i.e. 3 months ago) is not an accurate representation of the technologies current performance!

I will illustrate my point using the rapid evolution of the most talented music artist of all time, Britney Spears. Britney spears is a singer, dancer, actor, perfume model, fallen angel and mother all in one.

Britney the Singer

Britney the Dancer

Britney the Actor

Britney the Perfume Model

Britney the Fallen Angel

Britney the Mother

Similarly, depending on when you look at Britney Spears’ career you would be underestimating her amazing talents.

Disclaimer: For the good of all mankind! This is purely my opinion and interpretations. People should just give Britney a break, she’s only human. 

2 responses to “Rapid improvements may lead to comparison pitfalls

  1. We were a bit late to buying cellphones the first time – it was 1999 or 2000 – and I noticed that, at that time, all the major plans were somewhat different. There were a variety of ways they differentiated (e.g. minutes per month, versus price, versus cost of overtime, versus cost of add-on phone, etc) so that none had a head to head comparison with another for price *and* service. In something as different as nextgen sequencing, the same paradigm is found again.

  2. Bruce, thanks for the insight and interesting parallel with mobile phone plans. 1999-2000 is a bit late, I was on to my second phone by then 🙂 Oh many hours wasted playing that Nokia snake game during lectures!

    The mobile phone comparison would have been much better than Britney… but I like Britney. I like Celine Dion more though 😛

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s