Reply to post: It would help if the author knew what he was talking about.

BIG DATA wizards: LEARN from CERN, not the F500

Ian Michael Gumby

It would help if the author knew what he was talking about.

As someone who's been involved in using 'Big Data' tech in the F500 companies for over 5 years, and as someone who's helped set the strategy with a couple of the F500 companies, I can say that the author doesn't know jack.

You are talking apples to oranges when you try to talk about sensor data in the same way you talk about transactional data. Big difference. In terms of sensor data. Is it discrete or continuous data? If its continuous data then you may have long periods of no change in readings that get stored and this data could be tossed and then focus on the discrete data which occurs when there is a change in the sensor's input.

The reason vendors are saying to store everything is that many who are new to Big Data don't know what data is relevant and what data is not, or what data may become relevant when you combine it with other data sets.

Currently the F500 tend to store things in a relational model and when items don't fit, they get dropped. By going to a semi-structured or unstructured system, you can retain more attributes which may hold value. In terms of purging data or moving to cold storage... there are other factors like regulatory and business use cases that determine what to do.

In terms of the vendors, they are not going to tell a business what to do or how to do it. (Of course they'll spin reality to let them sell their version of Hadoop/Big Data and what tool is best.) They are going to say, when in doubt, save everything. Hardware is relatively cheap and getting cheaper in terms of cost per TB.

Sorry, but the author should look inwards and think more about the problem than trying to base a recommendation on something he read on the internet.

Not all problems are equal, so why should their solutions be the same?

I guess Mr. Nicholson should add more tools to his tool belt. His hammer vision makes everything look like nails and even with nails, you will want to think about different types of hammers. ;-P

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2022