Primary Source Material is Crucial for Facts & Research
Monday, November 8, 2021
Back in college we all had access to those often bulky, hard to use research databases, that sometimes worked, but often steered us in the wrong direction. We had to teach ourselves Boolean operators to properly navigate them. There was a reason other than torture for utilizing those; to help us all find primary source materials to write our research papers.
As is often the case, that's about the only time we used academia, raw data and studies to conduct our analysis on any given topic. Unfortunately, many of us negated those skills in our everyday lives. We only read tweets and not the accompanying story bylines and don't question it. We turn on cable news for answers. As soon as we rely on others to conduct the analysis, we lose control of what is fact and what is not.
For the purposes of this post, I'll be looking at the continued importance of primary source for conducting our own research for whatever we desire. I'll point out what to search for, how to do so, and how to read misleading studies and research.
Stated before, this should be a refresher from college or even high school, yet we forget such things in the era of social media and cable news. Bias is at a premium, and this should be your first factor when looking for a source to research a claim.
We will use the example of historical context of the following: A database may contain a personal letter from John F. Kennedy or Richard Nixon urging to sway the constituent their way to vote. While this is a primary source and is unique and certainly has its place in history, it is quite biased and should not be used to factcheck, unless the piece is a part of a larger historical research project, per say.
Let's take the example of a major economic number; the monthly non-farm payroll report from the Bureau of Labor Statistics. This is the primary source for all data relating to U.S. employment, unemployment, wages, including a break down where the jobs were gained, lost, and why. The method for collection is survey. Since COVID-19, the BLS has also factored in the ways in which the survey takers communicate their situation.
The response rate for the household survey was 75 percent in September 2021. While the rate was lower than the average before the pandemic of 83 percent for the 12 months ending in February 2020, it was considerably higher than the low of 65 percent in June 2020.
Bureau of Labor Statistics
We must keep in mind as the response rate returns more to normal levels, that there still may be some slack in respondents, creating a larger margin of error (MOE) in responses. As sample sizes decrease, the chance for skew increases. Though this is a primary source, keep in mind any data deterioration that may arise as the survey was collected, in this case for the month of October 2021.
Continuing on the document, the BLS talks about the misclassification issue. Surveys are meant as a point-in-time Continuing through the document, the BLS talks about the misclassification issue. Surveys are meant as a point-in-time and simply not capable of handling entire population sizes. Technically, if an employee is "on leave" due to COVID, they are not considered unemployed, thus, a misclassification has taken place.
If the misclassified workers who were recorded as employed but not at work for the entire survey reference week had been classified as “unemployed on temporary layoff,” the unemployment rate would have been higher than reported.
Bureau of Labor Statistics
Given that COVID-19 was a once-in-a-generation situation, statistical measures can be improved upon moving forward if any other possible disruptive events occur. Like all Data Scientists, hypothesis must be carefully created, methodologies are more important on a national level such as the BLS, and the data requires further refinement and consultation as to what "voluntary leave" or "furloughed" means if these become larger data points in surveys going forward.
An economic number may not be what it seems on a headline or in an article posted in CNBC, Bloomberg, NYT, or Fox Business, for example. Their job is to get clicks and engagement (positive and negative). It's how these sites and companies boost ad revenue in a world where Facebook and Google dominate the online ad market. It's your job to question where these claims came from, to consider what the bias may be, and to retrace the steps to obtain a deeper understanding of what the numbers are "really" telling.
Though it may seem a bit absurd, we all must be capable of basic data science when it comes to understanding the headline. False claims and skewed articles run ramped in the age of social media. Older publications have unfortunately fallen into the same category as they race for clicks and their own share of the ad market. Leave your own biases aside when considering what to think after reading questionable content. Do your homework, as it were. The true comprehension of the story will come through and you can inform others why these pieces may have gotten the story wrong.