Thinking critically about the validity and reliability of a research resource generally involves asking yourself a series of questions about the quality of both the item and the content of that item.
While research that has gone through a peer-review process is generally considered more high quality, or credible, many academic/scholarly resources have also been criticized for gate keeping and perpetuating heteronormative, western, white, or male-centric voices.
When evaluating sources, you may want to ask yourself: are there any perspectives or a voices missing?
If the answer is yes, how could you layer or expand your research to search resources that are more non-traditional in addition to the scholarly resources?
Some questions to ask when critically evaluating sources:
Author
What are the author's credentials, such as, institutional affiliation [where he or she works], educational background, past writings, or experience? Is the book or article written on a topic in the author's area of expertise? Has your instructor mentioned this author? Have you seen the author's name cited in other sources or bibliographies? Is the author associated with a reputable institution or organization? What are the basic values or goals of that organization or institution?
Date of Publication
When was the source published? Is the source current or out-of-date for your topic?
Edition or Revision
Is this a first edition of this publication or not? Further editions usually indicate a source has been revised and updated to reflect changes in knowledge, to include prior omissions, and to better harmonize the contents with the intended needs of its readers.
Publisher
Note the publisher. If the source is published by a university press, it is likely to be scholarly. Although the fact that a publisher is reputable does not necessarily guarantee quality, it does show that the publisher has a high regard for the source being published [their reputation as an academic publisher relies on it.]
Title of Journal
Is this a scholarly or a popular journal? This distinction is important because it indicates different levels of complexity in conveying ideas and the intended readership.
Intended Audience
What type of audience is the author addressing? Is the publication aimed at a specialized or a general audience? Is this source too elementary, too technical, too advanced, or just right for your needs?
Objectivity
Is the information covered considered to be fact, opinion, or propaganda? It is not always easy to separate fact from opinion. Facts can usually be verified; opinions, though they may be based on factual information, evolve from the interpretation of facts. Does the information appear to be valid and well-researched, or is it questionable and unsupported by evidence? Note errors or omissions. Are the ideas and arguments advanced more or less in line with other works you have read on the same topic?
Coverage
Does the work update or clarify prior knowledge, substantiate other materials you have read, or add new information? Does it extensively or only marginally cover your topic? Does it provide a balanced perspective? If the item in question does not meet this criteria, you should review enough sources to obtain a variety of viewpoints.
Evaluative Reviews
In the case of books, locate critical reviews of the work. Is the review positive? Is the book under review considered a valuable contribution to the field? Do reviewers agree on the value or attributes of the book or are there strong differences of opinion? Does the reviewer mention other books that might be better? If so, locate these sources for more information on your topic..
Increasingly, fraudulent manuscripts that resemble legitimate research articles have made their way through the peer review process and have been published in reputable journals. “Paper Mills”, organizations that produce and sell fraudulent manuscripts are at the center of this problem. This news article in Nature describes how more than 10,000 research articles have been retracted in 2023 due to integrity issues and the systematic manipulation of the publishing process.
The impact of fraudulent research being published with the stamp of authority of a peer-reviewed journal is far-reaching. Not only is it damaging to the trust researchers place in the publication system, but the fraudulent research may be used to build more research, wasting money and time for a researcher.
Scholarly journals have become increasingly aware of paper mill articles and are working to develop methods to screen for them. For researchers, it can be difficult to detect these published articles. In some cases, AI is exacerbating the problem making it easier to fabricate research.
The retraction of an article is the removal of an already published article from a journal. This decision may be made by the journal's editor and/or editorial board. Retractions do not happen because of small editorial errors. An article would be corrected in this case. Retractions reflect more serious issues with an article.
Researchers should become familiar with how retractions are communicated in their discipline and journals. This can help to avoid citing fraudulent research, but of course this only applies to research that has been identified as fraudulent. The Retraction Watch database is a tool that can be used to identify retracted journal articles.
