The posting below gives some good advice on how to critically look at research abstracts. It is from Chapter 3, in the book, Critical Reading and Writing for Postgraduates, by Mike Wallace & Alison Wray. Published by SAGE Publications Ltd,1 Oliver’s Yard ,55 City Road, London EC1Y 1SP. www.sagepublishing.comCopyright © Mike Wallace and Alison Wray 2016. All rights reserved. Reprinted with permission.
UP NEXT: Prop Up Your Presentations
---------- 2,508 words ----------
A First Look: Interrogating Abstracts
Keywords: abstracts; information; insight; questions
We saw in Chapter 1 that critical reading comes quite naturally when we are suspicious of the motives of the author. In academic writing, however, it can feel uncomfortable at first to ask questions. After all, the author is a published researcher and you might feel that he or she has more knowledge and expertise than you. In Chapter 4 we will show you how to get beneath the surface of a published text, to identify potential weaknesses in claims or alternative ways of interpreting evidence. Doing so successfully requires some confidence in asking questions, and this chapter introduces a technique for developing that confidence.
The technique makes use of an important design feature of the abstract typically found at the start of a published paper. We will come to what that design feature is in a moment. Abstracts are invaluable, because one of the most challenging aspects of working with the research literature is just how much of it there is. It would be impossible to read everything fully just to see if it was relevant to your needs. It follows that authors have a responsibility to ensure that the abstract carries the most useful and appropriate information for the reader. For that reason, in Chapters 17 and 18 we will look at how to write a good abstract for your own dissertation, journal article and conference presentation.
It is important to note that although we will focus in this chapter on a way of using the abstract on its own, there is no substitute for the full reading of an article once you have decided that you need to use it. The reason, as we shall see, is that there is not enough information in the abstract for it to give you all the knowledge you need, if you are planning to refer to the paper as a whole.
The aim here is to help you make a habit of ‘active’ reading – that is, thinking critically all the time when you are reading. As a result, you will enjoy your reading more, and find it more engaging. You are less likely to wake up at teatime and have no recollection of what you have been reading all day.
Using the abstract as a resource
The author of an academic paper provides an abstract as a service for the reader. It summarizes the paper, making it easy for the reader to gain an idea of what topics it covers, what sort of approach is taken and what the main claims are. But the abstract has a hidden value for the critical reader, precisely because it is so short.
As will become clear in the later chapters, critical reading, as defined and demonstrated in this book, revolves around asking questions of the text that are relevant to what you need to know for your own research. But it can be difficult, when reading the entire paper, to keep track of whether those questions are being adequately answered or not. Information is not necessarily given in the most convenient places, and so there is always the risk of missing something important. When you are reading a research article you can easily feel overwhelmed, wondering if in fact a certain piece of information is in the text and you just missed it. It would be much easier if you knew, before you started, what exactly you were looking for.
This is where the abstract comes in. When you read the full article, you generally expect that most of the information you need to know will be there somewhere. But when you read the abstract, you know that it probably will not all be there. After all, an abstract is typically no more than 150 words – far from long enough to explain everything. Most importantly, the abstract will typically make claims without any space to back them up.
As a result, if you ask yourself ‘is this a good study?’ (which is a core element of evaluating the usefulness of a paper), the abstract may not give you enough information about how the authors reached their conclusions for you to work out the answer. Even if you are asking the more basic question ‘is this study relevant enough to what I need?’, you may find that you cannot tell, because the abstract misses out some key aspect of the theoretical approach, data source or set of underlying assumptions and beliefs. It is this inherent absence of information that we will be using here to prompt your critical thinking.
Asking questions of an abstract
Even if you would usually feel nervous asking questions about a published paper, when you are dealing with the abstract, you know you are right in believing that they have not told you what you need to know. How could they have, in such a brief text? And, from what they have told you, you now know what else you need them to say. For example, as you read the abstract of a paper that uses statistics to report research, you might be asking:
- Yes, but what did you mean by that term?
- You have told me what sort of informants you used, but how many were they, and why did you choose them?
- You have told me what you wanted to find out, but why was it interesting and important to find it out?
- You have said that a lot of research has been done on this topic, but what are the key studies that I should be checking out?
- You have said that you found a significant difference between your two experimental groups, but what statistical test were you using?
These are the sorts of questions that you might ask someone face-to-face, if they had given you a brief overview about their work. In response, they would no doubt fill in the gaps, and you could probe further till you had all the information you needed. With a written article, the authors are not in the room with you, but you can use their full text as a potential source of answers to the questions raised by reading the abstract. Before you looked at the paper in its entirety, you could work out what answers would satisfy you, and why. So for this paper:
- I will expect a clear definition of that term. I will want to see if they are aware of the problem with defining it that I have read about in other work. If I do not find a definition, or if the definition is inadequate, I will need to work out whether that is just an oversight, or whether they have failed to appreciate the complexity of the phenomenon. If the latter, then it may not be clear what they have really found out here.
- I will expect a clear statement about the identity and selection of the informants. Because other work I have read suggests that age is an important variable relating to the feature under examination, I will be particularly looking for information about the age of the participants. If they are all young, and the authors nevertheless claim that their findings are generalizable across the entire population, I will be able to cite that other literature to cast doubt on the quality of their backing (warranting) for that claim.
- I will expect a clear steer from them about why they think it is interesting to extend the existing research into this new domain. Previous studies in other domains have claimed that the phenomenon is universal. If that is so, their study was not necessary. So I want to know why they thought this domain might be different.
- I will be checking their literature review for two things — relevant studies that I have not read, and that they can convince me I should go and find; and studies that I know about but that they do not mention. Where they have not mentioned a study that I think is important, I will be looking for clues as to why. If I conclude that they were not aware of a particular line of research that would have altered how they see things, I will be able to mention that in my critical commentary – after all, it could undermine the reliability of their claims.
- I will expect them to indicate what statistical test they used, and to justify it. Before I read the paper, I will try to figure out what stats I think they would use here, and then I can see if that is what they did. If they did not, I will try to figure out why. Perhaps I can learn from them. Or perhaps they have made a decision I have good reasons to disagree with.
How the questions help you think
The sharply focused questions that you develop by reading the abstract will, mostly, be answered in the main paper, because it was only lack of space that prevented them occurring in the abstract. However, one or two of the questions might not be answered. And because you are reading the article looking for answers, you are much more likely to notice when that happens. Because the questions are based on your own current knowledge and the nature of your own research project, they will be the very ones of most importance to you. Not finding the answers in the main text will, as a result, be very relevant to your narrative. It will give you the basis for a critical commentary about the paper.
A worked example
The process of critically reading an abstract is best explained through an example. Firstly, note that although we have talked about the abstract, in fact the interrogation begins even higher up the paper – with the title and the authors. You are free to ask whatever you most need to. It is an exercise in focusing on what is important for you in your study, and in developing confidence to think of questions.
Source: Educational Researcher, 2012, vol. 41 (9): 339-351
Title: Are Minority Children Disproportionately Represented in Early Intervention and Early Childhood Special Education
- This is a journal that addresses researchers of education. Why did the authors choose this audience?
- What do they mean by “minority”? Which country are they talking about?
- What age range are they covering?
- How do they define “early intervention” and “special education”? Do their definitions create any potential problems?
Authors: Paul L. Morgan,1 George Farkas,2 Marianne M. Hillemeier,1 Steve Maczuga1
¹Population Research Institute, The Pennsylvania State University, University Park, PA; ² University of California, Irvine, CA
Abstract: We investigated whether and to what extent children who are racial-ethnic minorities are disproportionately represented in early intervention and/or early childhood special education (EI/ECSE).
We did so by analyzing a large sample of 48-month-olds (N = 7,950) participating in the Early Childhood Longitudinal Study-Birth Cohort (ECLS-B), a nationally representative data set of children born in the United States in 2001.
- Why are there four authors? What skills does each bring?
- What is the Population Research Institute?
- Where in UCI does Farkas work?
- What is their main purpose in writing this paper likely to be?
- Why did they think that racial-ethnic minority children might be disproportionately represented? What was their starting assumption?
- Is this sample size big enough for the kind of questions they want to answer? For example, how much variation might there be across different States and/or different minority groups?
- Who says that the ECLS-B is nationally representative? Has anyone challenged that?
Multivariate logistic regression analyses indicate that boys (odds ratio [OR] = 1.66), children born at very low birth weight (OR = 3.98) or with congenital anomalies (OR = 2.17), and children engaging in externalizing problem behaviors (OR = 1.10) are more likely to be represented in EI/ECSE.
Children from low-socioeconomic-status households (OR = 0.48), those displaying greater numeracy or receptive language knowledge (ORs = .96 and .76, respectively), and children being raised in households where a language other than English is primarily spoken (OR = .39) are less likely to be represented in EI/ECSE.
Statistical control for these and an extensive set of additional factors related to cognitive and behavioral functioning indicated that 48-month-old children who are black (OR = .24) or Asian (OR = .32) are disproportionately under-represented in EI/ECSE in the United States.
- What is ‘odds ratio’? Are large values a stronger effect, or weaker? Have they demonstrated cause or just a link?
- Do they define ‘low birth weight’ clearly?
- What sorts of ‘congenital anomalies’ are they talking about, and where are the lines drawn? Similarly for ‘externalizing problem behaviors’. Might the definitions affect the findings (e.g., if certain anomalies and behaviors are less prevalent in some ethnic groups)?
- What languages ‘other than English’ are they talking about?
- Do they take into account large minority communities (e.g., Spanish) versus being the only kid with that language in the neighborhood
- What were the ‘additional factors’?
- Are any reasons given for this disproportionate under-representation?
Welcome to critical thinking
It takes only a few minutes to generate questions by interrogating an abstract. It is something you can easily try out and develop the technique for. All you need to do is print out an abstract and annotate it by hand. Or, if you prefer, copy and paste the text into an electronic document and add comment boxes. There are no right or wrong answers when you create these questions. Simply, write down whatever comes into your head as something that you want to know and the authors have not told you. Prioritize questions that are most relevant to the study you are doing yourself – that is, questions it would be useful to know the answer to, for the sake of your own research. It is fine if you ask about things they do tell you in the main part of the paper – in fact, that is what you should generally expect will happen. Using the questions, when you read the paper you will have a good idea of what you are looking for. And if the authors do not ever answer your question, then that gives you food for thought. For example, if they never define their main terms, you can consider whether that is a problem: are there alternative potential definitions that would impact on how their claims are interpreted? Or, if they do define their terms, are they then consistent in deploying these terms across the paper? (Authors often are not.)
Having given you this quick way into a paper using the abstract, in the next chapter we introduce the first of our main tools for engaging with a critical reading of the main paper. By the end of the chapter you will not only have an overview of how that tool works, but you will also see how it relates to the pre-questioning you do in the abstract.