Finding Information for General Chemistry Laboratory Work: Evaluating Information

It is tempting to think that all scientific information is accurate, particularly in fields like chemistry that rely on quantitative data to support conclusions; however, even the best journals issue retractions (pulling papers from circulations) when articles are discovered, after publication, to be fraudulent, inaccurate, or based on erroneous premises. 

It is also tempting to think that chemical information is unbiased.  After all, reactions either work or they don't, and the "data don't lie."  In point of fact every piece of scientific information is biased, for two reasons.

1. Bias of experience: Scientists use the word "interpret" to describe their analysis of data.  This means that, while there is no surety that their premises are correct, they are attempting, to the best of their abilities, to explain the data in terms of things that they know or believe to be true.  Experience comes into play here greatly; a scientist with more experience in the field or who has read more widely on the subject will have more information at her fingertips and will be able to bring this experience into the interpretation of the data.  A scientist with less experience will have a much more narrow perspective and will not be able to bring as many diverse findings to the interpretation.  Unfortunately, the only cure for the bias of experience is age.
2. Bias of expectation:  This is also known as "confirmation bias," and it begins at the time of project selection.  A scientist will only elect to do a project if it has a good chance of success... and if he can secure the funding for it!  Once he begins the work, he develops a working hypothesis, which he alters as the project progresses.  It is very tempting to "see what you want to see," and he must be careful not to let his hypotheses or his expectations color the interpretation of his results.

One can apply the same questions to evaluating a piece of research that one can apply to dissecting a story or a piece of literature: who, what, where, when, why, and how.  When evaluating research, however, these questions take on a slightly different meaning.  You should ask yourself all six questions whenever looking at scientific information in both familiar and unfamiliar areas.

The National Institute of Standards and Technology asks the following three questions of all pieces of data.

1. How well is the data generation described?
2. How do the data follow the known physical laws?
3. How do the data compare to other measurements or calculations of the same phenomena?

Source: Reproduced from https://srdata.nist.gov/CeramicDataPortal/pds (Accessed June 4, 2017).

In addition, NIST offers some definitions of various types of data that can be applied to pieces of data located on a substance or phenomenon.  They are presented in the order of their "authority."

1. Certified: Standard reference values, specific to known production batches
2. Validated: Confirmed via correlations and models
3. Qualified: Basic acceptance criteria satisfied
4. Commercial: Manufacturer's data for specific commercial materials
5. Typical: Derived from surveys of nominally similar materials
6. Research: Preliminary values from work in progress
7. Unevaluated: All other data

Source: Reproduced from Munro, Ronald G.  Data Evaluation Theory and Practice for Materials Properties, NIST Special Publication 960-11.  Washington, DC: National Institute for Standards and Technology, 2003.