Thursday, April 22, 2010

Chris Dag from BioTeam - Trends from the Trenches

Chris Dagdigian from BioTeam chaired the Track 1 Infrastructure - Hardware talks, and got some additional time due to the unfortunate loss of Phil Butcher from Sanger due to the volcano. Chris ran through the latest Trends From the Trenches presentation, and had some interesting updates.
  • He had expected blades to win the HPC hardware battle, but has not seen that come to pass, it's still a split field
  • Intel is currently the chip of choice, but AMD might be back in the game
  • BioTeam has done more Sun Grid Engine consulting in the first quarter of 2010 than all of 2009; he's not concerned about SGE's future following the Oracle acquisition of Sun
  • He got a laugh from the crown with "private clouds - still stupid in 2010". He notes that this is just marketing speak and doesn't really mean anything.
  • Public clouds, on the other hand, are very real, and close to being mainstream... he's a strong supporter of their use in the right situations.
  • DIY cluster/parallel filesystems have a higher risk of implementation failure rate due to lack of pre-sales planning and design, especially in smaller shops. He also recommended commercial solutions with formal support programs.
  • Clusters are increasingly utilizing fat nodes (32 core, 128 GB+ memory)
  • Petascale storage is no longer risky, and single namespace solutions are recommended
  • He expressed concern about the downstream analysis of data (such as sequencing) eating up storage capacity - while the HTPS pipeline is relatively easy to model, secondary analysis is much more difficult.
He had an interesting observation regarding communication between IT and scientists. He gave the example that scientists will often ask for 100% uptime and full data protection, but don't realize that the difference between five 9s and four 9s of uptime is several million dollars. He emphasized the need for bettter communication between IT and research, and that IT needs accurate accounting of IT costs so that it can explain the costs of services and facility

He argues that the DNA data deluge will get better, mostly because the sequencing vendors are becoming more efficient in delivering data from the instruments. I would agree that the per run sizes will stabilize, but as the costs continue to plummet for sequencing, it will drive much greater demand, thus continuing the pressure on storage and compute infrastructures.

He also talked about an issue that Jackson has had recent experience with, and that is the challenge of high-speed networking. He noted that moving large data around requires more than just big pipes and bandwidth. His experience is the number of hops between locations can have a huge impact on performance, as well as the tools and protocols utilized.

No comments: