Big Data or Big Data Failure?

understand your data

You’ve seen the success stories; the ones you hear about companies who quadruple sales in 3 months or cut costs in half overnight by analysing their data.  But for every success story there’s a dozen Big Data Failure stories.

None of the big wins we hear about is easy to come by, but they are possible.  It requires a structured approach, a small team and a plan (and perhaps some late nights and plenty of Red Bull!)

Big Data or Big Data Failure

Finding real value within your Big Data is a challenge most organisations face and the task usually falls in the lap of one or two people who work on it in parallel with their main job.  These gals or guys tinker away with any data they can get hold of and explore relationships, hoping to find something good. 

After a few months, the excitement fizzles and the data gets cast aside as people focus on getting their work done.  They never find those game changing insights.  Big Data becomes a Big Data Failure as it all gets too hard.

But it shouldn’t be that way.  Here’s our view on the six-step process to successful DIY Data Analytics:

  1. Hypothesise
  2. Organise
  3. Visualise
  4. Analyse
  5. Crystalise
  6. Socialise

Step 1: Hypothesise; work out what are you looking for

The team approach is so essential, mainly from the group think perspective.  Collectively think about all the issues you’d like to tackle that you think the data can answer.

Don’t think about data, just focus on what you’d like to explore.  Think:

  • Business pain points
  • Operational bottlenecks
  • Data credibility

Socialise these problems and get others to collaborate and contribute.  Getting more people involved during this process will generate greater momentum when it comes time to publish your results. 

When you have a shortlist, identify a few that you think will be plausible and valuable to explore.

Step 2: Organise your data

Make an inventory of each source including where it’s located, how often it’s updated, who the gatekeeper is and so on.  The more you know about the source of each piece of data the stronger your analysis will be.

Map out how the various data sources are connected and identify the key pieces of data that bind things.  Look for the common thread like a client number, project number, invoice and company name.

Using an ETL like Alteryx can connect all your data sources and bring them into a structure that you can work with.

Step 3: Visualise your data

This is the fun part.  It’s time to eyeball your data and look for patterns, trends and correlations.

Seeing your data visually should open your eyes to a different perspective, particularly when you are graphing blended data.  Trends and patterns are easier to detect and correlations can be tested in even the most basic BI tools. 

Don’t fall into the spurious correlation trap!

This step should give you some idea of where to dig deeper and how to find the Gold in those Big Data hills.

Step 4: Analyse your data

You don’t need a PhD in mathematics to analyse your data.  OK, it helps, but your core strength lies in your understanding of your business and context is more important than anything at this stage.

Delve deeper into your data and explore the correlations to see if they represent causality.  Focus on what is directly connected to valuable outcomes that will have a positive impact on efficiencies or resolve a recurring problem the business is facing.

Isolate the findings down to one or two key measures and thoroughly test them.

Step 5: Crystalise the message onto a single page

Very little time is usually devoted to delivering results effectively, but you really should make a big deal of it.  Tell your story and create a clear, concise and engaging narrative.

Don’t assume your results will be immediately intuitive to everyone.  You’ve spent a lot of time getting to your answer and have been elbows deep in the data for weeks if not months.  Take your audience on the journey you’ve just been on and draw them into the reasons why your message is really important.

Oh, and don’t use a pie chart.

Step 6: Socialise your findings

The ideal goal of any visualisation is to get the “aha!” moment when the audience first sees your results.  While the medium will dictate the method it won’t alter your message.

Consider your audience and the presentation options you have available:

  • Presentation deck
  • PDF Report
  • Interactive dashboard
  • Q&A sessions

Give people a takeaway, or something they can interact with and explore on their own.  This is a powerful lure.

Finally, get feedback.  Whether it’s good or bad, it will be valuable.  Use it to work on improving and expanding your findings.


The most effective approach to successful DIY Data Analytics is to work as a team, start small and be goal-oriented.  Follow these steps and you’ll stand every chance of avoiding Big Data Failure.

  1. Hypothesise
  2. Organise
  3. Visualise
  4. Analyse
  5. Crystalise
  6. Socialise

 And don’t forget the Red Bull.