This wiki service has now been shut down and archived

DIR General ideas and suggestions

From ESIWiki

Jump to: navigation, search

Stratis remarked on Friday that "the definition of what makes a problem (or a computation, or a system) data-intensive has not emerged". However, I believe that we can see a distinction between problems that are "data-intensive" versus those that are "compute-intensive". With compute-intensive problems, the challenge can be merely a lack of CPU cycles, and more processors, faster cores or parallel algorithms can offer a route towards a solution. Perhaps a data-intensive problem could be defined as one where the barrier to progress is not a shortage of compute-power, but is due to some aspect of data scale or complexity. During the week we saw many applications where throwing more CPU cycles at the problem wouldn't solve it.

We saw examples of different ways in which data can bite you. It would be good to have a catalogue of the kinds of problems that arise in data-intensive research, and to know what danger signs to look out for. These problems can arise at various points in the data life cycle, from data capture, through curation, analysis, visualisation and curation. Even if computer scientists don't have answers to a particular data-oriented problem encountered by scientists in other areas, it should be possible to help scientists to anticipate problems ahead, so that they can plan their research accordingly.

In reporting back from a breakout session on Thursday, Shantenu made a comment along the lines that if we can't scale up our solutions, scientists will have to scale down their ambitions. I like that comment, and it highlights the importance of taking the design constraints imposed by data scale and complexity into account when planning a research study.

GrahamKemp 13:29, 1 April 2010 (UTC)

This is an archived website, preserved and hosted by the School of Physics and Astronomy at the University of Edinburgh. The School of Physics and Astronomy takes no responsibility for the content, accuracy or freshness of this website. Please email webmaster [at] ph [dot] ed [dot] ac [dot] uk for enquiries about this archive.