RESEARCH: Understanding Where Data Come From : SAMPLES

Sampling Terminology

The purpose of this chapter is to provide you, the consumer of research, with an overall understanding about the importance of and the thinking that goes on when choosing a sample. I first provide some initial definitions of terminology, which are essential for understanding the rest of the discussion. These definitions are followed by two segments that discuss the two major sampling paradigms found in research in applied linguistics. The choice of paradigm, as you might suspect by now, is guided by the research question being asked by the researchers. The chapter ends with a discussion of the ethics of using human participants in a research study.

The sample is the source from which data are drawn to answer the research question(s) and/or to test any hypothesis that might be made. The sample consists of one or more cases . In most studies the cases are made up of human beings, referred to as subjects or, more currently, participants . For example, Luo and Liao (2015) used 30 students (20 males, 10 females) in their study investigating the effects of using corpora to correct errors in EFL students’ writing. In other studies, the cases might be inanimate objects from which researchers extract their data. Examples are corpora of verbal discourse such as an accumulation of newspaper articles, or when researchers cull their data from transcriptions of taped dialogs. Rett and Hyams (2014), for instance, used the corpora of 45 American English-speaking children in the CHILDES database in their study of the acquisition of syntactically encoded evidentiality . They extracted 70 perception verb similatives (a type of sentence) out of the database for their data. Interestingly, they put this under the heading of “Subjects.” Although the corpora constituted the verbal output of 45 children, the children were not the participants of the study; the corpora were the source of the verbal data. They narrowed the corpora down to 70 statements, which were their final sample (i.e., the objects of the study).


Sometimes the reader can be confused as to what makes up a sample, as seen in the Rett and Hyams (2014) study. The answer is determined by which data source is used to answer the research question(s). If it is directly from the participants, then the qualities of the participants need to be reported. If it is from objects like corpora, then the qualities of the corpora should be summarized. For reasons outlined next, there are different uses, which demand different combinations of participants/ objects to answer different questions.




Sampling Paradigms


There are a number of ways that a sample is chosen to do research. Table 4.1 provides a list of some of the most commonly used techniques along with the main purposes for using them and brief definitions. The two general paradigms I refer to are representative ( probability ) and purposeful ( non-probability ). The first consists of techniques that try to capture a sample that best represents a defined population-based on probability theory. The second attempts to identify samples that are rich in specific information. Representative sampling (more commonly referred to as probability sampling) has one aim: finding a sample that reflects salient characteristics of a specific population so that the results of the study can be generalized to that population. Purposeful sampling (also referred to as nonprobability or purposive sampling) is more concerned with the unique characteristics of the sample itself aside from any larger population— which is not to say that the results cannot be applied to other situations (more on this later). Before going further, I want to state that purposeful sampling is the paradigm that is most commonly used in applied linguistic research. However, I am presenting the material in the following order because I believe that the section on representative sampling lays down foundational principles for understanding sampling theory.

The Representative Sampling Paradigm


As previously mentioned, in the representative sampling paradigm the goal of the researcher is to generalize the findings and interpretations of the study to a larger population. The sample is a portion of a larger population. The word population usually means everyone in a country or a city. In research, this word has a more technical use; although similar, population means all the members of the group of participants/objects to which researchers want to generalize their research findings. This is referred to as the target population . In other words, the criterion for defining a target population is determined by the group of people to which researchers would like to generalize the interpretations of the study. For example, the population might be all learners of English as a foreign language (EFL) or it might be a more limited group of all learners of EFL who attend an English-medium university. For another study the target population may be entirely different. 

Typically, having access to the entire target population to which researchers want to generalize their findings is impossible. For example, having access to all learners of EFL who attend English-medium universities throughout the world is, in practice, impossible. However, researchers may have access to English-medium universities in their own country. 

Whatever is available for use becomes the experimentally accessible population (Gall, Gall , & Borg, 2008). It is to this population the findings of a study can be directly generalized, not to the entire target population. The only time that researchers could make inferences from the findings of their study to the target population is when they can show that the experimentally accessible population possesses similar characteristics to the larger target population. For the rest of the book, I use the phrase target population with the understanding that I am referring to the experimentally accessible population. 

Selecting a representative sample is important for making use of the findings of a study outside of the confines of the study itself. This is because the degree to which the results of a study can be generalized to a target population is the degree to which the sample adequately represents the larger group— the degree to which a sample represents a population is determined by the degree to which the relevant attributes in the target population are found in the sample.

Figure 4.1 illustrates the relationship between the sample and the population. I have used different graphic symbols to represent different attributes of a population. These attributes could be gender, age, level of education, level of language proficiency, and so on. Notice that the attributes in the sample (A, B, C, D, F) almost match exactly the attributes in the population; however, attribute E is missing in the sample. In this case, the sample is not 100 percent representative of the population, but it is very close. Most likely we could conclude that the population was representative enough to make tentative generalizations. However, there would always remain caution due to the missing attribute E.


The degree to which findings of a study can be generalized to a larger population or transferred to similar situations is referred to as external validity (or transferability: Miles & Huberman, 1994). To achieve this type of validity, researchers must demonstrate that the samples they use represent the groups to which they want to apply their findings. Otherwise, without this important quality, the findings are of little use outside of the study. The more representative the sample is to the population, the greater the external validity. In other terms, the more similar the characteristics of the sample are to other situations, the better the transfer of conclusions. 

Identifying the target population is not always easy. For example, Hong-Nam and Leavell (2007) examined the language learning strategies of bilingual and monolingual students. The authors described the participants as 428 monolingual Korean university students (223 males and 205 females, aged 18– 28) and 420 bilingual Korean– Chinese university students (182 males and 238 females, aged 20– 28). They also provided information about the bilingual participants, which included the number of students using either Korean or Chinese at home or with friends, and overall proficiency (beginning, intermediate, or advanced) in both languages. Then they compared the two groups on gender, English proficiency, years of study, years studying English, test-taking experience, and travel abroad. 

You can see rather quickly that the sample becomes more complex as the authors add more details about the participants. Without a careful read of the article, it can quickly become confusing about who the target population is. In this study, the target population is identified in their research questions as monolingual Korean and bilingual Korean– Chinese university students. All of this additional information is important to the study, but does it add to or distract from clarifying the target population? Is it all bilinguals who use more complex strategies to learn when compared to monolinguals or is it only bilinguals with more extensive English backgrounds? These questions are important to answer if the researchers want to generalize their findings to larger populations than these subgroups represent.

 The problem of researchers not identifying their target populations is not uncommon in published research. However, without this information, the consumer cannot evaluate whether correct generalizations are being made. 

 An additional note is that choosing a representative sample is used not only for quantitative research. Some qualitative studies also seek this quality in their samples. To illustrate, Oz ˙anska-Ponikwia (2016) used 97 Polish L2 adult learners of English in their study of the effects of immersion in the L2 culture on the Polish culture-specific emotion of te ˛ sknota in the qualitative part of their study. They took care in describing their participants in great detail so that readers could transfer their findings to similar groups. However, they also warned in their conclusion section that the sample was still too small to generalize to the target population.




Observational Procedures


The procedures under this heading involve capturing data through visual observation. The use of human observers as data collectors is as old as research itself. It has long been known that the main advantage of human observation of data, over some form of impersonal instrument, is that the former allows researchers flexibility when exploring what new, and sometimes unexpected, phenomena might be uncovered.


On the other hand, some believe that observational procedures suffer from three disadvantages. The first is that they generally take more time than instrumental procedures. Consequently, they are usually more costly. Second, they are more limited in the numbers of participants/objects that are used for data gathering. Third, they allow for varying degrees of subjectivity . That is, the influence of factors such as attitude, temporary emotional and physical states, etc. can distort the perception of the observer. However, others believe that these three weaknesses are, in fact, strengths of this category of procedures. The fact that it takes more time, they argue, means that there is better chance to obtain quality information despite the cost. Using fewer subjects is not a problem if the purpose is to observe information-rich samples. Last, subjectivity is viewed as positive because researchers become personally involved with the data collection. In addition, if multiple observers are used and compared to one another for degree of agreement, subjectivity is controlled. When all is said and done, I believe that most everyone would agree that observational procedures are powerful means for gathering data. 

Observational procedures have many different formats. First, the person(s) doing the observing can vary considerably. Observers can be researchers, people employed to make the observations, or participants observing themselves. Second, observers might be very involved with the purpose of the study or totally oblivious to why they are being asked to observe. Third, observers might be recording information that requires no interpretation of the observations or be required to give their own evaluative judgments as to what their observations mean. Fourth, the observation process may or may not be backed up with recording devices. Researchers often use recording devices (audio or video) to aid in further analysis. 

In the following discussion, I will show how these different formats are used by surveying the most common observational techniques with their strengths and limitations. This section is based on the degree to which the observer is personally involved with who/what is being observed, beginning with the most involved observer, the self. It ends with the interviewer.

If you are researching language speaking, then your findings Should be from the factor that you say it is, and not swayed into something else by other factors like gender, societal roles, age and other unrelated factors.

Could the Quality of the research be improved or enhanced by a re-evaluation of the method in which data is gathered to make the research more credible?

Perry, Jr., Fred L.. Research in Applied Linguistics : Becoming a Discerning Consumer, Taylor & Francis Group, 2017. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/nottingham/detail.action?docID=4825127.
Created from nottingham on 2021-02-18 03:36:55.











Discerning Discussions and Conclusions
Six Needed Ingredients

 Researchers vary in the format they use to wrap up their studies. Some will only have a discussion section. Others will have both discussion and conclusion sections. You might also see additional subheadings, such as “summary” and/or “implications.” Some attach their discussion section to their results section, labeled something like “results and discussion,” followed by a final conclusion. Almost all will have at least a paragraph on the limitations of their study. Regardless of the format they use, they usually include the following six ingredients in the discussion/conclusions section of their paper.


• An overview of the study : The purpose of the study should be restated, the questions under investigation summarized, and any propounded hypotheses reiterated.
• Overview of the findings : The findings should clearly be related to the research question(s) and/or shown how they support or fail to support any hypothesis being proposed.
• Relation of findings : The findings of the study should be related to previous research findings and theoretical thinking.
• Attention to limitations : The researchers should evaluate their own study and point out any weaknesses and/or limitations.
• Possible applications : The conclusions should contain how the results can be applied to practical situations. • Future possibilities : Topics for future research should be proposed.

Seven Questions Every Consumer Should Ask 

When evaluating the discussion/conclusions section of a study, there are questions that the consumer should address: 

1. Do the findings logically answer the research questions or support the research hypothesis? Here is where the consumer must be wary. Many, if not all (except for me, of course—ha!), researchers have their biases and would love to find answers to their questions, or support their hypothesis from the results of their studies. Because this final section gives researchers the right to conjecture about what the findings mean, it is easy to unintentionally (or even intentionally) suggest things that the results do not support. 

2. Does the nature of the study remain consistent from beginning to end? My students and I have noticed that some studies begin as exploratory studies but end up as confirmatory ones. In such cases, the introduction section has one or more research questions, with no specific hypothesis stated. However, in the discussion section we suddenly read “and so our hypothesis is confirmed by the results.” Another variation of this is that some researchers generate hypotheses in the discussion section—which is their right—but then go on to suggest that their results now support the hypotheses. This is circular We cannot use the same data to support hypotheses from which they have been formulated. A new study must be made to test these hypotheses. 

3. Are the findings generalized to the correct population or situations? Most studies, in fact, cannot be generalized to a broadly defined population. The reason is that most samples are not randomly selected, nor are they typically large enough to adequately represent a target population. Consequently, results of such studies are suggestive at most and need to be followed up with a number of replications. If the same findings are repeated using different samples from the target population, then we can have more assurance that we are on the right track. (This is where meta-analysis plays an important role—to be discussed in Appendix A.) A well-written discussion section will be careful to warn readers of this problem. 

4. Are the conclusions consistent with the type of research design used ? The main concern here is whether causation is being inferred from research designs that are not geared to demonstrate this effect. Having an idea of the type of design being used will help the consumer know whether this error is made when reading the discussion and conclusion section. Nonexperimental designs such as descriptive or correlational ones cannot be used to directly show causation. Yet, especially in the latter case, some researchers have slipped into suggesting that their f indings indicate that one variable influences another. When researchers apply their findings, they are often tempted to recommend that people manipulate one variable to cause changes in another. Unless their research design warrants this application, they have made a logical error. 

5. Are the findings and conclusions related to theory or previous research ? To help contribute to the big picture, a well-written discussion/conclusions section should attempt to tie the findings and interpretations to any current theoretical thinking or previous research. This might be done through showing how the findings support what has gone before or by providing evidence to refute some theory or challenge previous research. 

6. Are any limitations of the study made clear ? There are very few, if any, perfect studies in the literature. Regardless of how good a study is, a conscientious researcher will mention what the limitations are to caution the reader from being overly confident about the results. Often they even dedicate a subheading to this. 

7. Is there consistency between the findings and the applications? As previously mentioned, when inferential statistics was discussed, some researchers confuse statistical significance with practical significance. I repeat the warning here. Just because a finding is statistically significant does not mean that it has practical significance. I have seen relatively small correlations, such as r = .30, interpreted as an important finding because it was statistically significant, or the difference of five points between a treatment and a control group given importance for the same reason. Yet, is either of these findings large enough to get excited about? Maybe, but much depends on the cost in time, human resources, and finance to make changes based on that .30 correlation or those extra five points owing to the treatment. The consumer needs to be on alert when a researcher advocates costly changes based on statistical significance. This is where effect size is applied, as mentioned in Chapter 7 and elaborated in Appendix B. 

In the following, the discussion/conclusions sections of two studies have been evaluated using the above set of questions. The criteria I used for choosing these articles were that they were used previously in former chapters and they each represented a different type of research: qualitative and quantitative. The first is an example of a qualitative study using a micro-ethnographic design and verbal data. The second is an example of a causal-comparative study using numerical data.