Year: 2018
Type: article
Abstract: Conventional Econometrics and Big Data• Conventional econometric techniques (regression) often work well, but there are issues unique to big datasets that may require different tools (Varian, 2014).• Why? Size of the data and p value More potential predictors (variable selection) Non linear relationship Machine Learning• Divide the data into training, testing and validation.• Machine learning finds particular function (s) that provide a good prediction of y as a function of x.• Out of sample performance (overfitting problem) minimizing sum of square residuals.• Historically, machine learning deals cross-section data.• The data may be "fat," (lots of predictors relative to the number of observations) or "tall" (lots of observations relative to the number of predictors). Big Data Variety Structured Data• Automated Multiple-Pass Method (AMPM), developed by the Agricultural Research Service of the United States Department of Agriculture (USDA)• ABS follow AMPM approach for Australian National Nutrition and Physical Activity Survey.• Face to face and telephone interview
Authors: Sayan Chakrabarty, Arpita Chakraborty
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2891769987