Researchers need to establish what the dataset represents, appreciate how the data can best be pre-processed and analysed and determine how much of the data is required to answer the question. Appropriate sampling methods can quickly transform a big dataset into something more manageable.
According to an article in The Register, published 25 March 2015 by John Nicholson on “Big data wizards: LEARN from CERN, not the F500,” the Large Hadron Collider provides “One of the best hong kong rcs data examples of the management of big data.” collisions per second but has around 100 collisions of interest per second that it wants to review — CERN (the European Organisation for Nuclear Research) filters the data and disregards around 99.99% of the sensor stream data produced. As Nicholson puts it, while the CERN team “may not “know what it’s looking for, it knows what it has already seen”.
If having broken the data down, it is still too large to run on a PC, you don’t necessarily need a Hadoop-type framework (Hadoop is an open-source java-based programming framework that supports the processing of large datasets in a distributed computing environment). Other options include building a big data dashboard using Google BigQuery, adding visualisations through Google Chart tool, or using some of the excellent if lesser known open source software.