Blog posts on Data Science, Machine Learning, Data Mining, Artificial Intelligence, Spark Machine Learning

Monday, February 3, 2014

Data Analysis Steps

After going through the overview of tools & technologies needed to become a Data scientist in my previous blog post, in this post, we shall understand how to tackle a data analysis problem.
Any data analysis project starts with identifying a business problem where historical data exists. A business problem can be anything which can include prediction problems, analyzing customer behavior, identifying new patterns from past events, building recommendation engines etc.

The steps for solving a data analysis problem can be shown as below:

Identify Business Problem:
 “Define Problem statement”
This is the first step of analysis. Business identifies a problem and a problem statement with desired outcome is defined. In this stage, a Data Scientist should understand the problem statement, the domain knowledge of the problem. After thorough understanding of the problem statement, a Hypothesis will be proposed.

 Data Acquisition:
“Identify data sources”
As a second step, all the data sources related to the problem statement will be identified and pulled into a central repository. The data sources can vary from SQL databases to text files to csv files to online data. If the data size is large we may use Hadoop to pull, store & pre-process the data.

Process/Clean Data:
 “The accuracy of the results of analysis depends on the quality of data” 
Data Clean step is considered to be one of the very important phases in Data analysis. The accuracy of the analysis depends on the quality of data.
Few approaches:
  • Formatting the data as per the data analytical tools we use.
  • Missing data handling
  • Data Transformations like normalizing the data Identifying outliers & handling etc.
Exploratory Analysis:
 “Embrace the data visually before diving further”
The objective of this step is to understand the main characteristics of the data. This analysis is generally done using visualizing tools. Performing an Exploratory analysis helps us:
  • to understand causes of an observed event
  • to understand the nature of the data we are dealing with
  • assess assumptions on which our analysis will be based
  • to identify the key features in the data needed for the analysis
Graphical Techniques:Scatter plots, box plots, histograms
Quantitative techniques: Mean, median, Mode, Standard deviation
Model Generation & Validation:
“Select-Train-Evaluate” 
This step involves extracting features from the data and feeding them into the machine learning algorithms to build a model. Model is the solution proposed for the problem statement. This step involves: Model selection, model training and model evaluation.
Model selection: Based on the type of business problem we are dealing, a model will be built. For example,if the objective of the analysis is to predict a future event, we need to build a Regression model for prediction.
Model Training: After selecting the Model for the analysis, the entire dataset is divided into 2 parts – Training data & Test Data. 3/4th of the entire data will be fed as input to the Model Algorithms.
Model Evaluation: Once the model is built. The next step is to test the model & validate it. The data used for testing the model is the remaining 1/3rd of the dataset in the previous step.
Visualize Results:
 "Show the results visually" 
This is the final step of Data analysis where the results of the model & problem solved will be presented generally in visual plots/graphs.
Few visualizing tools: d3.js, ggplot2, tableau.

Please go through the tools/technologies , skill set required to learn Data Analysis here

31 comments:

  1. With the base of endeavors however limit of conceptualizing, the reality of the business is changed. It goes with the assessment of the on-going tasks and profitability.data science course in pune

    ReplyDelete
  2. Nice blog Thank you very much for the information you shared.
    data science

    ReplyDelete
  3. I was blown out after viewing the article which you have shared over here. So I just wanted to express my opinion on Data Analytics, as this is best trending medium to promote or to circulate the updates, happenings, knowledge sharing.. Aspirants & professionals are keeping a close eye on Data Analytics Course in Mumbaito equip it as their primary skill.

    ReplyDelete
  4. Such a very useful Blog. Very interesting to read this article. I have learn some new information.thanks for sharing. know more about

    ReplyDelete
  5. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.
    Click here

    ReplyDelete
  6. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!
    ExcelR data analytics

    ReplyDelete
  7. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.


    Invisalign specialist

    ReplyDelete
  8. The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
    Artificial Inteligence course in Patna
    AI Course in Patna

    ReplyDelete
  9. Join ExcelR and get data science certification to get your dream data science job. data science course syllabus

    ReplyDelete
  10. Very nice blog and articles. I am realy very happy to visit your blog. Now I am found which I actually want. I check your blog everyday and try to learn something from your blog. Thank you and waiting for your new post.

    data science course in India

    ReplyDelete
  11. Your article acts as a valuable resource for both beginners and individuals looking to improve their data analysis skills. The integration of practical considerations, like the tools and technologies used throughout the process, adds a tangible dimension to your guidance. Thank you.
    Data Analytics Courses in Pune

    ReplyDelete
  12. "Great breakdown of the data analysis steps! This clear and concise guide is a valuable resource for anyone starting in the field of data science. Thanks for sharing!"
    Data Analytics Courses In Bangalore

    ReplyDelete
  13. This blog post provides a comprehensive and structured guide to the essential steps of data analysis. It's a valuable resource for aspiring data scientists, offering clear explanations and practical insights. Well-done.
    Data Analytics Courses In Dubai

    ReplyDelete
  14. "Your data analysis blog is a goldmine for anyone seeking insights and clarity in the world of data. The way you break down complex concepts and provide real-world applications is incredibly helpful. It's like having a data expert at my fingertips. Keep the insights coming!"
    Data Analytics Courses in Zurich

    ReplyDelete
  15. Anyone looking for understanding and clarity in the data world will find your blog on data analysis to be a treasure trove. Your ability to simplify difficult ideas and offer practical applications is quite beneficial. It's like I always have access to a data specialist. Keep the insights coming!"
    Data Analytics Courses in Agra

    ReplyDelete
  16. It's great to see so many resources and tools available for this purpose. Keep up the good work.
    Data Analytics Courses In Chennai

    ReplyDelete
  17. Understanding these steps is like having a roadmap to uncover valuable insights from data.
    Data Analytics courses IN UK

    ReplyDelete
  18. This article is a goldmine of information. Thanks for the insights!

    ReplyDelete
  19. I learned so much from this post. It's like a mini-education in the subject matter.

    ReplyDelete
  20. The guide on how to tackle a data analysis problem is really informative and insightful thanks for sharing.
    data analyst courses in limerick

    ReplyDelete
  21. the steps of data analysis are really well explained. Looking forward to more quality blogs like this.
    financial modelling course in melbourne

    ReplyDelete
  22. Thank you for sharing valuable and in depth tutorial on Data Analysis Steps.
    Digital Marketing Courses In Bhutan

    ReplyDelete
  23. I appreciate how the post acknowledges the iterative nature of data analysis, emphasizing the need to revisit and refine steps as insights unfold. It reflects a realistic and adaptable approach to the dynamic nature of data. Digital Marketing Courses In Norwich

    ReplyDelete
  24. This post delves into the foundational steps of data analysis, emphasizing problem identification within historical datasets. It highlights the diverse range of business problems that data analysis addresses, from predictive tasks to customer behavior analysis and pattern recognition. Understanding these steps sets the groundwork for effective problem-solving and insights extraction within data science projects.
    Data Analytics courses in new york

    ReplyDelete
  25. Your insightful breakdown of the essential steps in data analysis provides a comprehensive guide for tackling business problems. The clear explanations and structured approach make it a valuable resource for both beginners and experienced professionals. Thank you for sharing this informative post!

    Investment banking courses syllabus

    ReplyDelete