Abstract
Plagiarism appears to be a common problem among college students, yet there is little evidence on the effectiveness of interventions designed to minimize plagiarism. This study presents the results of a field experiment that evaluated the effects of a web-based educational tutorial in reducing plagiarism. We found that assignment to the treatment group substantially reduced the likelihood of plagiarism, particularly among student with lower SAT scores who had the highest rates of plagiarism. A followup survey suggests that the intervention reduced plagiarism by increasing student knowledge rather than by increasing the perceived probabilities of detection and punishment.
I. Introduction
The individual and public investments targeted at the acquisition of higher education are substantial. For example, in 2008, over 19 million students were enrolled at postsecondary, degree-granting institutions in the United States and the expenditures of these institutions exceeded $430 billion (Snyder and Dillow 2010), roughly 3 percent of gross domestic product (GDP). A distinguishing hallmark of this extensive human-capital investment is the opportunity for students to hone both their capacity for original, critical insights and their ability to express these insights in the written word. However, there is a broad concern that these investments are often compromised by student plagiarism, an illicit behavior thought to have grown increasingly common over the last two decades because of both technological change (for example, electronic access to full-text resources and cut-and-past word processing) and shifting social norms among young adults (Rimer 2003).
There are at least two broad economic motivations for efforts to reduce plagiarism. First, plagiarism may impose negative externalities on others by, for example, lowering grades of students who do not plagiarize and diminishing the signaling value of educational credentials for all students (that is, the “detrimental reliance” argument outlined by Posner 2007). Second, plagiarism may lower the human capital of those who plagiarize. More specifically, copying another’s text may reduce one’s subject-matter knowledge relative to understanding and expressing material in an original manner. More unambiguously, plagiarism harms human-capital acquisition by attenuating a student’s capacity for critical reasoning and original expression, skills that are often characterized as the signature achievements of selective postsecondary schooling. To some extent, these human-capital consequences of plagiarism are internalized by students. However, they will have efficiency consequences as well if students do not understand the personal costs of their behavior and/or the costs of their human-capital investments are often highly subsidized (for example, by parents, endowment spending and government support).
Though there has been little objective measurement of student plagiarism, evidence from college student surveys consistently suggests that plagiarism is quite common (for example, McCabe, Treviño, and Butterfield 2001). The prevalence of this behavior is likely to persist in part because college instructors tend to put little effort into detection and are frequently unwilling to engage formal campus disciplinary procedures, instead choosing to resolve cases of academic misconduct informally and lightly (Schneider 1999).
Some policy commentators have correspondingly recommended combating plagiarism through increased enforcement of its strong statutory prohibitions (Thompson 2006). However, the available descriptive evidence also indicates that college students often do not have a clear understanding of what actually constitutes plagiarism or how it can be avoided (Power 2009, Howard and Davies 2009). Students may endogenously choose to avoid the costly acquisition of such knowledge and skills because the low probabilities of detection and punishment make such efforts unappealing.
This type of “rational ignorance” can occur in other economic settings where acquiring information is costly and the probability that the information will be instrumentally useful is quite small. The seminal example involves low-information voting as a response to the small likelihood of influencing an election (Downs 1957). A similar dynamic may also contribute to tax evasion when audit probabilities are low and other types of poor regulatory compliance when information is costly and enforcement unlikely.
This study presents the results of a field experiment that evaluated the effects of a web-based educational tutorial in reducing plagiarism among college students. Our study makes several important contributions. First, we provide evidence on the prevalence and characteristics of student plagiarism using objective measures, which are based on the analysis of a large number of actual student papers rather than student self-reports as in nearly all of the previous literature. Second, we show that a low-cost, easily scalable intervention can dramatically reduce plagiarism among college students. Third, our results suggest the more general importance of information as a public good that can possibly reduce other illicit behaviors that are poorly understood by otherwise, well-intentioned decision-makers.
The group-randomized trial presented here was conducted in undergraduate social-science and humanities classes at a single, selective postsecondary institution during the fall 2007 semester. Each of the participating classes had a Blackboard web site that provided students with access to course materials. All of the students in the participating courses (573 students who wrote a total of 1,256 papers) were unaware that they were participating in a research study but were required by their instructors to submit their writing assignments electronically through these Blackboard-based course web sites. The classroom-level treatment consisted of requiring students to complete a short but detailed and interactive Blackboard-based tutorial on understanding and avoiding plagiarism. We collected searchable, electronic files of the papers from all of the participating courses and analyzed these papers using proprietary plagiarism-detection software (Turnitin.com).
Our results indicate that plagiarism occurred in 3.3 percent of the papers from courses randomly assigned to the control condition. We find that random assignment to the web tutorial reduced instances of plagiarism by roughly two percentage points overall (that is, a two-thirds reduction) and that this treatment effect was concentrated among students with lower SAT scores. The results of an ex-post survey and quiz completed by participating students suggest that the treatment was effective in large part because it increased student awareness about what constitutes plagiarism and knowledge of effective writing strategies. We find much weaker evidence that the intervention altered student perceptions about the likelihood of detection and/or sanctions associated with detection. But, regardless of whether the relevant mechanisms were knowledge or deterrence, our results provide compelling evidence that a relatively brief, targeted intervention can substantially reduce the prevalence of an important illicit behavior. Furthermore, the potential appeal of this type of policy strategy is further enhanced by the fact the intervention has a relatively low cost and can fairly easily be implemented at scale with high fidelity.
The reminder of the paper proceeds as follows. In Section II, we discuss the prior literature and present a simple economic model of plagiarism that will help us interpret the results of our analysis. In Section III, we describe our field experiment in greater detail. Section IV discusses our methodology, including how we address several key analytical concerns. Section V presents our results, and Section VI concludes.
II. Student Plagiarism
A. Prior Literature
Plagiarism by students is widely thought to be common, particularly with the recent diffusion of word-processing software and Internet access to full-text resources (Rimer 2003). However, we know of no nationally representative data on the prevalence or trends in this form of illicit behavior. Instead, most of the available data on academic misconduct are based on self-reports by students from a relatively small number of selected postsecondary institutions (McCabe, Treviño, and Butterfield 2001). For example, an influential early survey of students from 99 colleges and universities during the spring of 1963 (Bowers 1964) found that 28 percent of students indicated that they had plagiarized from published materials at least once since entering college. More recent surveys of students at selected institutions (McCabe and Treviño 1997; Scanlon and Neumann 2002; McCabe 2005) similarly suggest that plagiarism is common.1
However, other research suggests that students do not appear to have a uniform and accurate understanding of what actually constitutes ethical writing, which casts doubt on all of the self-reported survey studies of plagiarism. For example, a recent study by Baruchson-Arbib and Yaari (2004) found that undergraduates are less likely to view plagiarism from resources that are available online as a form of academic dishonesty. Power (2009) also found that only about 20 percent of the first-year students at small university accurately confirmed that material restated from a source document needed to be cited. Focus groups conducted by Power (2009) also indicated that college students did not understand plagiarism or how to avoid it but were confident that they would not be severely punished. The available data from college instructors affirm that confidence. Several surveys of college faculty (Wright and Kelly 1974; Singhal 1982; Nuss 1984; McCabe 1993) suggest a strong preference for dealing with incidents of academic misconduct informally and relatively lightly rather than following established campus procedures that are viewed as frustrating and time-consuming. In describing how college faculty respond to instances of academic misconduct, one prominent researcher noted that “the number who do very little is very large” (Schneider 1999).
This evidence suggests that student plagiarism may persist as a common illicit behavior because students are not equipped with the knowledge and skills needed to write ethically and because the low probabilities of detection and meaningful punishment do not provide meaningful incentives to improve their understanding or their compliance with statutory regulations. This characterization along with concerns about cost effectiveness and scalability motivated the design of the educational intervention studied here, a web-based tutorial described in detail in the next section.
Cross-sectional and pre/post comparisons suggest that institutional and classroom practices may influence plagiarism (McCabe, Treviño, and Butterfield 2001, Bilic-Zulle, Azman, Frkovic, and Petrovecki 2008, Jackson 2006). Similarly, a lab-experimental study found that student exposure to vignettes about peer attitudes and behavior regarding academic misconduct influenced the likelihood that they or a protagonist would engage in such behavior (Rettinger and Kramer 2009). However, we know of no prior field-experimental or convincingly quasi-experimental evidence on the efficacy of anti-plagiarism strategies.
B. Theoretical Framework
The tutorial described in detail below may reduce plagiarism by providing otherwise well-intentioned students with the motivation and skills needed to write ethically. However, it is also possible that the presence of the tutorial simply functions as pure deterrence, encouraging students to suspect that their instructor is more likely to detect and punish plagiarism. A simple economic model provides a framework for illustrating the effects of the intervention and for drawing out some testable implications of its possible “informational” and “deterrence” mechanisms.
In this model, students choose an amount of “honest” writing effort, h, which includes activities that contribute to both understanding and implementing ethical-writing practices. The amount of plagiarism committed is a decreasing function of this effort and of their stock of preexisting information, I, about what plagiarism is and how to avoid it. The resulting level of human capital, K = k(hA), is an increasing function of h and their ability, A. Students view this human capital as having a (private) return of w.2 The utility cost of this effort, c(h,I), is increasing in h but decreasing in I, the stock of information about what plagiarism is and how to avoid it. The probability that a student is caught and punished for plagiarizing, α(h,I), is a decreasing function of h and I.
In this framework students choose h to maximize the following:
where S is the utility sanction when caught plagiarizing. This basic model implies that students will choose h to balance the marginal benefits associated with increasing their human capital and reducing their expected sanction (wkh – αhS) with the marginal cost of expending this effort (ch). Consistent with the available descriptive evidence, this model implies that plagiarism will be more common when sanctions are low, when detection is unlikely (which implies that is αh also small), and when students perceive more modest private returns to writing ethically (that is, when w is low).
This model also provides a framework for considering the effects of our tutorial on the prevalence of student plagiarism. Specifically, we view our randomly assigned plagiarism tutorial, which was completed early in the study period, as creating an exogenous increase in I. It is straightforward to show that this increase in I will increase h* (and reduce plagiarism) when the information conveyed by the tutorial lowers the marginal utility cost of honest writing effort (chI < 0) and increases its marginal benefit with regard to lowering the probability of detection (αhI < 0).
As suggested above, one interpretation of this result is through the lens of information and human capital. By providing students with an understanding of plagiarism and practical strategies for avoiding it, an increase in I lowers the marginal
cost of undertaking such effort (chI < 0) and amplifies the effect of h in reducing the probability of detection (αhI < 0). In other words, after the tutorial, the honest effort students expend on avoiding plagiarism may become both easier and more effective. As a result, students increase their chosen h, which implies a reduction in the prevalence of plagiarism.
However, an alternative interpretation of this treatment effect is as a pure deterrence phenomenon. That is, when students are presented with the tutorial, they do not really learn about plagiarism but do update their priors about the probability of detection (that is, the perceived α(h,I) increases) and, correspondingly, their perception of the impact of h in reducing the threat of detection increases as well (αhI < 0).
The overall impact of our tutorial on the prevalence of plagiarism is observationally equivalent with both of these interpretations. However, only the deterrence interpretation implies that the probability of detection perceived by students, α(h,I), increases in response to the treatment. For this reason, we conducted an ex-post survey of the students who participated in this field experiment and included questions about their perceptions of the likelihood that plagiarism would be detected as well as questions about their knowledge about plagiarism. We present evidence on how these student perceptions and knowledge differed across students who were randomly assigned to the treatment and control conditions in order to discriminate between these two explanations for the tutorial’s impact.
Another interesting question involves whether the tutorial, if effective, would be more or less effective for students with different levels of ability. Our simple model does not provide clear predictions about the likely patterns of treatment heterogeneity. However, one plausible conjecture is that the informational benefits have less relevance for higher-ability students who already possess a comparative capacity to convert their ethical-writing efforts, h, into human capital. Our analysis addresses this question empirically by examining the tutorial’s impact across students with different levels of baseline SAT score.
III. The Experiment
The setting for our field research is a single, highly selective postsecondary institution in the United States. Specifically, we collected and analyzed electronic versions of anonymized student papers from 28 undergraduate socialscience and humanities courses during the fall 2007 semester.3 The collection of student papers occurred largely through the Blackboard classroom-management web page for each course and the participating students were unaware of the study’s existence. As part of the human-subject protocols for this research project, we do not identify the participating institution and all student papers were anonymized prior to analysis. It is important to note that the study institution does not include plagiarism training as an explicit part of either its orientation scheduling or its regular curriculum. (In contrast, the orientation does include mandatory alcohol-use and sexual-harassment workshops). Hence, the relevant treatment we study here is arguably a completely new source of information to students, and not simply an add-on or reminder to previously discussed material.
A. Study Recruitment and Randomization
We began the recruitment of courses by identifying all the social-science and humanities courses offered during the Fall 2007 semester. We excluded quantitative-methods courses, small-scale seminars, and research colloquia as well as independent study and thesis-related courses. We approached the instructors for 46 classes that had comparatively large enrollments and solicited their participation in a campuswide study on student writing. The motivation for our emphasis on larger classes (that is, typically 18 or more students) was both increased statistical power for our research effort and a potential increase in the external validity of our inferences for institutions that, on average, have larger class sizes than the participating institution. To complement the block-randomization strategy we describe below, we also recruited courses with somewhat smaller enrollments in situations where those courses were taught by the instructor of another recruited course.
The instructors were asked whether they would be willing to include their course in an IRB-approved, field-research project on the characteristics of student writing. They were told that participation would not involve any substantive change in their course. Participation would simply require using Blackboard’s classroom-management software to collect student papers electronically and to provide students with information on their writing assignments. To encourage participation, the research team made it clear that they would design and manage this aspect of Blackboard as well as provide participating instructors with printed or electronic versions of all their submitted writing assignments. The instructors for nine of the 46 recruited courses provided no responses to recruitment queries. Four other courses had no valid writing assignment. The instructors for five additional courses refused participation.4
The remaining 28 courses were randomly assigned to treatment and control conditions. Courses in the control state merely had students use Blackboard to submit their writing assignments. In courses assigned to the treatment state, students also submitted their writing assignments through Blackboard. However, before they were allowed to do so, they also had to complete a Blackboard-based tutorial and quiz on plagiarism. This intervention is described in more detail below.
Our course-based randomization avoids the contamination that might have occurred if students within the same courses had been randomly assigned to the treatment. However, a potential drawback of randomizing over only 28 units is that the treatment and control courses might not be balanced with regard to observed and, more important, unobserved baseline traits. To reduce this possibility, we employed a simple block randomization strategy, pairing participating courses on baseline traits and then randomizing within those pairs.
In an ideal situation, we would be able to match each course to another course with a similar propensity for student plagiarism by using baseline traits that are highly predictive of the prevalence of plagiarism. Unfortunately, reliable baseline variables of this sort are unavailable in this context. However, in light of the prior evidence on the importance of contextual factors (McCabe, Treviño, and Butterfield 2001), we conjectured that the likelihood of plagiarism would be related to the many unobservables associated with particular instructors (for example, types of writing assignment, writing support, and the apparent threat of detection) and with particular academic disciplines. Our review of the syllabi and writing assignments for the participating courses provided some confirmation for these priors (for example, the presence of a plagiarism warning on the syllabus and the extent to which writing assignments involved a student’s response to instructor-chosen source material as opposed to researching a topic through self-identified references).
Based on the available baseline information about the courses and their instructors, we paired courses prior to randomization in the following manner. First, for 12 of the participating courses, we were able to form pairs among courses taught by the same professor in the same department (and, in six of these cases, randomization was also within sections of the same course). Second, for ten other courses, randomization occurred among courses taught in the same department. In cases where there were multiple courses from a given department, we paired courses that had similar writing assignments as indicated by the syllabi (for example, research content of the assignments and the presence of a plagiarism warning). The remaining six courses were paired to another course in the same academic division (that is, social sciences or humanities) using the same data on the character of the writing assignments.
B. The Treatment
In the courses assigned to the treatment, students were required to complete a Blackboard-based tutorial on understanding and avoiding plagiarism. The tutorial was adapted for Blackboard from resources available at the Plagiarism Resource Site (https://ats.bates.edu/cbb/) developed by staff at Colby, Bates, and Bowdoin Colleges.5 The tutorial required students to click through 18 sequential screens with text that defined different forms of plagiarism. This tutorial also provided explicit examples of what constitutes plagiarism by showing side-by-side examples of source material along with examples of the correct and incorrect use of that material in a student paper. The tutorial also outlined effective strategies for avoiding plagiarism (for example, not procrastinating and careful note-taking). At the end of this sequence of material, students completed a nine-question quiz consisting of several detailed and example-driven questions on plagiarism. Each response triggered detailed feedback on why that answer was either correct or incorrect before proceeding to the next question. Appendix 1 contains several illustrative screenshots of this tutorial.
This intervention was deployed on the Blackboard sites of the treatment courses at the beginning of the third week of the semester (that is, immediately after the date at which students could drop the course with no record of having been enrolled). The Blackboard sites made it clear that students would not be allowed to upload their completed writing assignments (that is, the upload mechanism would not activate) until students had completed the tutorial. However, because of the role that early research and note-taking can play in unintentional plagiarism, the instructors within the participating treatment courses were encouraged to email students early in the semester about the need to complete the Blackboard tutorial. They were also provided with the names and email addresses of students who had not promptly completed the tutorial and encouraged to provide targeted followup reminders. As a result of this effort, there are no empirically meaningful distinctions between the effects associated with the intent-to-treat and the effect of the treatment-on-the-treated in this study. Over 97 percent of students in the treatment courses fully completed the tutorial while an additional 1.5 percent partially completed the tutorial.6
An interesting and important feature of this study is that the participating students were not aware that they were participating in a research study.7 However, as noted earlier, the participating instructors did know that their courses were involved in a writing study. In theory, the general awareness among instructors that their student papers were being externally evaluated in some way may have muddied the treatment contrast by encouraging all instructors to manage these assignments in a manner that reduced plagiarism. The existence of sizable treatment effects suggests that, if there were any effects associated with this general awareness, they were not empirically confounding. Several of the participating instructors were also clearly aware of the broad intent of the research study (that is, the focus on plagiarism). Four participating instructors asked to be aware of the study goals as a condition of participation and a fifth clearly inferred the study goals because he or she actively managed other components of their Blackboard sites. Fortunately, our block-randomization strategy implies that there is uniform treatment and control variation within these instructors. Three of these instructors each taught two of the participating courses so they were paired with other courses they taught prior to randomization. The remaining two instructors taught courses in the same department and these courses were also paired with each other. This pattern of pairing implies that any effects that might be associated with an awareness of the study’s focus should again create an attenuation bias in the estimated treatment effects.
C. Identifying Plagiarism
We relied on the proprietary web service, Turnitin.com, to analyze the participating papers for plagiarism.8 For each submitted paper, Turnitin.com generates a “similarity score” that identifies the percentage of submitted text that matches their continually updated database of journal articles, newspapers, magazine articles, books, and web pages. An “originality report” also makes it possible to connect suspicious text to the potentially plagiarized source. At a pilot stage for this project, we compared the performance of this service to that of other available software and found that it was particularly discriminating both with respect to its extensive database and with regard to identifying plagiarized text that may have been lightly edited.
Settings for the originality reports allowed most quoted text and bibliographies to be ignored in generating similarity scores. Nonetheless, our review of the similarity scores from the papers in this field experiment indicated that a large share of the highest similarity scores reflected false positives. This occurred when the software failed to recognize correct citations of quoted material and when it flagged oddly formatted bibliographies as plagiarized text. For example, some high similarity scores occurred when a paper legitimately quoted other text but used margin offsets instead of quotation marks. We also found that some high similarity scores were simply due to the accumulation of common word fragments used throughout a given paper.9
Given the pervasive amount of measurement error in the similarity scores generated by Turnitin.com, we adopted a straightforward rating strategy using multiple reviewers. We first reviewed the papers with high similarity scores (that is, 15 or higher). Roughly half of these had clearly plagiarized content while the remaining papers appeared to be false positives exclusively. We then reviewed each paper with a similarity score between 11 and 15. Of these papers, roughly one-third had plagiarized content. We then reviewed the papers with similarity scores between eight and ten. Only 16 percent of these papers were judged to have plagiarized content. As we moved to (and through) the third strata, the probability of having identifiable plagiarism clearly dropped. Furthermore, within the lower stratum, the extent of plagiarism in papers with plagiarized content was substantially lower. Interestingly, our exhaustive review of these papers indicated that the plagiarism that did occur was predominately of the “mosaic” variety (for example, copied sentences, sentence clauses and phrases).
Our analysis focuses on a binary dependent variable that indicates whether a paper had plagiarized content and was in the two rating strata defined by similarity scores of 11 or higher. This focus reflected a judgment that the plagiarism that occurred in the highest two strata was distinctly more consequential in scale. However, as robustness checks, we also present results based on binary indicators for more and less restrictive measures of plagiarism (that is, papers identified as having plagiarized content with similarity scores of 15 or higher and eight or higher, respectively).
D. Data Description
The 28 participating courses had collective enrollment of 697 students.10 The writing assignments in these 28 courses and the corresponding course enrollments implied that there were 1,329 potential papers to be collected. Because attrition from the collection of papers is a potential threat to both the internal and external validity of our study, we made an aggressive effort, in cooperation with the participating instructors, to obtain physical copies of papers that were submitted as print outs rather than through the web-based upload mechanisms. More specifically, nearly 6 percent of the potential papers (that is, 79 of 1,329) were obtained as printouts. Through the use of scanning and optical character recognition (OCR) software, we were able to convert these papers to searchable text and include them in our analysis.
Our final, analytical sample consisted of 1,259 papers, implying a fairly low attrition rate of 5.3 percent (that is, 70/1,329). The attrition of these 70 papers was due in part to students who withdrew from courses or had taken a grade of incomplete (n = 19). The remaining papers (n = 51) were either not submitted or were submitted (n = 51) directly to the instructor as (n = 51) printed copies that we could not obtain. As we discuss below, the differences in attrition across treatment and control classrooms were small and statistically insignificant.
We were able to identify a number of student traits (for example, race, gender, SAT scores) through access to the institution’s administrative data (Table 1). The composite (math and verbal) SAT scores were imputed for those who only had ACT composite data using a concordance table available from the College Board. We were also able to identify other student traits (for example, class status, pass/fail status) from the class enrollment data.11 We also identified several class-level observables (for example, class size, the presence of a plagiarism warning on the syllabus, the number of required papers for the course, the academic rank, and gender of the instructor) that may be relevant determinants of student plagiarism.
Summary Statistics
IV. Empirical Strategy
The randomized nature of the field experiment alleviates many of the common selection concerns associated studies of plagiarism, and allows for a straightforward analysis of the data. We estimate variants of the following OLS regression:
(1)
where i denotes individuals and c denotes classrooms.12 Tc is a binary indicator for whether the class was in the treatment group, Xi is a vector of student characteristics and Cc is a vector of classroom characteristics. The αc term is a classroom-specific error term that will be a concern in properly estimating the precision of our regression estimates, which we discuss in greater detail below.
In this section, we discuss three issues of particular concern in cluster-randomized trials such as this: (1) treatment-control balance, (2) sample attrition, and (3) proper treatment of the clustered nature of our data for the purposes of statistical inference.
A. Treatment-Control Balance
In expectation, the randomization of classrooms to treatment and control conditions will ensure that all observable and unobservable characteristics of students and classrooms are balanced across the two groups. In small samples, however, it is possible for a specific realization of random assignment to result in poor balance. Our block-randomization strategy was explicitly designed to avoid this potential problem. Nonetheless, it is still important to explore the realized balance of baseline traits across the treatment and control conditions.
Table 1 presents summary statistics that speak to this concern. The first column shows sample means for the full sample. We can see that the sample contains very high-performing students (that is, an average SAT score of 1407), who are typical of the participating institution. There are an equal proportion of males and females, and reasonable distribution of different race/ethnicity types, with 9 percent African-American, 11 percent Hispanic, 18 percent Asian and 56 percent Caucasian. Importantly, 12 percent of students do not indicate a race/ethnicity on school records. The majority of the sample is composed of freshman and sophomores (21 and 33 percent respectively). All freshmen take their fall courses pass/fail and roughly 6.8 percent of other students are taking pass/fail courses in our data. Note that the percentage of freshman (20.5), the percent pass/fail voluntarily (6.8), and the percent taking the class for credit (73.2) sums to 1 (with some rounding error). A bit less than 31 percent of classes included some warning about plagiarism in the syllabus. Female professors taught roughly one-third of the courses in our sample.
Columns 2 and 3 present classroom-level summary statistics for the treatment and control groups. In none of these comparisons do we find that the treatment-control differences are statistically significant. This is striking because, in conducting these “multiple comparisons,” one might expect to sometimes reject some null hypotheses of no difference, even when the null hypotheses are true. Various procedures (for example, Bonferroni and Benjamini-Hochberg) are designed to correct for the Type I errors in multiple comparisons. These corrections would only imply that the p-values in Table 1 are even larger. Moreover, it is useful to note that the only two characteristics that show economically meaningful distinctions between the two groups are the share taking the class for a grade and the share with a plagiarism warning.
However, multiple-comparison procedures that also allow for a joint error structure across these comparisons might create a more powerful test of the treatment-control differences in baseline covariates. As an alternative way to test whether we are likely to observe this distribution of covariates under the null hypothesis of random assignment, we conduct a permutation test analogous to Fischer’s exact test.13 To do so, we conduct a 1,000 replications in which we randomly assign treatment status to 14 of the 28 classrooms (keeping the covariates in the classrooms fixed as they are in reality). We then run a seemingly unrelated regression with each of the covariates as outcomes and a treatment indicator as the single predictor in each equation. We obtain the F-statistic from a joint test of the null hypothesis that the coefficients on the treatment indicator for all equations are equal to zero. We then ascertain at which point the true F-statistic we obtained in our sample would fall in the distribution of the 1,000 F-statistics we obtained from our permutations. The p-value from this exercise indicates that we cannot reject the null hypothesis of random assignment. We also conducted an identical permutation test, but instead of running a SUR, we estimate a single equation OLS model in which the treatment indicator is the dependent variable and all 20 of the covariates are predictors.
In summary, we cannot reject the null hypothesis of random assignment, and thus it appears that our covariates are reasonably well balanced. However, the results in Table 1 suggest that there were some noticeable treatment-control differences in classroom-level traits. For example, treatment classrooms were substantially more likely to have a female instructor and less likely to be taught be a full professor. We examine the robustness of our impact estimates to these statistically insignificant differences through regression adjustments for classroom observables.
B. Sample Attrition
Even if randomization results in good balance across treatment and control groups, differential sample attrition between the conditions may still result in a biased estimate of the treatment effect. For example, if students in treatment classrooms were more likely to drop the class when they feel uncertain about their writing skill, the result may be that control classrooms have a disproportionate fraction of good writers who may have a lower propensity to plagiarize even in the absence of the treatment. This dynamic would lead our empirical strategy to underestimate any beneficial impact of the treatment.
To test for the presence of differential attrition, we estimate specifications similar to Equation 1 where the outcome is a binary indicator for whether we have any outcome data for the student-paper observation. We estimate a variety of different variations on this basic specification, and in no case does assignment to the treatment group have a statistically significant or substantively important impact on attrition. Hence, sample attrition does not appear to be a concern with respect to the internal validity of our results. Furthermore, the low level of attrition also suggests that its implications for the external validity of this study are negligible.
C. Estimation and Statistical Inference
Although our final analysis sample contains over 1,200 student-paper observations, the treatment was randomly assigned across only 28 classrooms. As others have pointed out, the nested structure of the data has important implication for accurately estimating the precision of the treatment effects and conducting statistical inference (Bertrand et al. 2004; Cameron et al. 2008; Donald and Lang 2007; Angrist and Pischke 2009). Specifically, statistical inference must take into account the within-group dependence in the data. A common approach is to report cluster-robust standard errors that generalize the White (1980) heteroskedasticity-consistent estimates of OLS standard errors. Such cluster-robust standard errors provide consistent estimates as the number of clusters goes to infinity. In practice, however, many applied studies use samples with a small number of clusters.
Several recent papers demonstrate that cluster-robust standard errors may not be consistent when the number of clusters is as small (Cameron et al. 2008, Donald and Lang 2007). More importantly, the direction of the bias generally leads one to over-reject the null hypothesis. In the analysis below, we present a several alternative estimates suggested in the recent literature. There are two broad approaches we pursue.
The first strategy utilizes group-level data. In our case, this means that we will collapse our data to the classroom level and estimate specifications like the following using the 28 classroom-level observations:
(2)
where yc is the rate of plagiarism in Classroom C. For the purpose of inference, we calculate bias-adjusted robust standard errors to account for heteroskedasticity. The bias adjustment we use is HC2 (described in Angrist and Pischke 2009) and is meant to adjust for the finite-sample bias of the commonly used “robust” (White 1980) standard errors.
A variant of this group-data approach is a two-step procedure that allows us to incorporate the student-level covariates we have in an effort to gain greater precision. In the first step, we estimate
(3)
where μc provide estimates of the covariate-adjusted group effects, in our case the adjusted plagiarism rate in each classroom. In Step 2, we regress these adjusted group effects on a set of classroom-level variables, which can include our treatment indicator as well as other classroom covariates (and pair effects):
(4)
We show GLS estimates of Equation 4 that use the inverse of var(
) from Equation 3 as weights. We also report estimates of Equation 4 that are unweighted and a third set that are weighted based on the number of student-papers within each classroom. Following the suggestion of Donald and Lang (2007), we use the t-distribution with C-K degrees of freedom (where C is the number of group-level observations and K is the number of regressors) to conduct inference on estimates from all group-level models.
Our second broad approach directly utilizes the micro (that is, student-level) data. One of the virtues of using the microdata is that they facilitate identifying heterogeneity in treatment effects by student-level traits. However, our approach still needs to account for the nested structure of the data and the relatively small number of classrooms in our sample. To do so, we calculate and report bias-corrected clustered standard errors using the method proposed by Bell and McCaffrey (2002). This procedure, called bias-reduced linearization or BRL, is essentially a generalization of the HC2 correction for the case of clustering. In recognition of the within-group dependence and the small number of clusters, we conduct inference based on a t-distribution with C-K degrees of freedom despite the fact that the estimation utilizes student-level observations.14
Furthermore, we also show results from the bootstrap-based approaches recommended in Cameron et al. (2008). These authors propose cluster bootstrap-t procedures to improve inference in cases with a small number of clusters. Bootstrap estimates of a t-statistic provide “asymptotic refinement” because the asymptotic distribution of the t-distribution does not depend on any unknown parameters (unlike regression coefficients, whose asymptotic distribution depends on the unknown residual variance). In a bootstrap-t procedure, one calculates a t-statistic for each bootstrap sample, and compares the t-statistic from the original sample to the distribution of t-statistics from the bootstrap replications. If the absolute value of the original t-statistic is above the 95th percentile of the absolute values from the bootstrap distribution, one rejects the null hypothesis at the 5 percent level. While this approach provides some efficiency gains, it does not yield standard errors, which might be of independent interest, for example, to calculate a confidence interval. On the basis of Monte Carlo simulations, Cameron et al. (2008) recommend using a wild-cluster bootstrap rather than a simple block bootstrap. The wild-cluster bootstrap resamples residuals while holding the regressors fixed. A key advantage of this approach is that it avoids bootstrap replications in which β or var(β) are inestimable, as can happen more frequently with a small number of clusters when the treatment varies exclusively at the cluster level.15
V. Results
We begin by presenting in Table 2 some descriptive evidence on the relationship between various student and class characteristics and the prevalence of plagiarism. For this exercise, we use our main plagiarism indicator (that is, papers with similarity scores of 11 or higher that were rated as plagiarism by multiple raters) and simply note that the results are qualitatively the same using the other measures. Column 1 presents estimates from a series of bivariate OLS regressions that model an indicator for plagiarism as a function of a single student or classroom characteristics. The standard errors for the student level regressors are clustered by student whereas the standard errors for the classroom regressors are clustered by class with a bias-reducing linearization (BRL) adjustment.16 Column 2 presents estimates from a single regression model in which all of the student and classroom characteristics shown are entered jointly as regressors. Hence, Column 1 shows the unconditional relationship between a particular regressor and the outcome whereas Column 2 shows the conditional relationship between the regressor and the outcome.
Relationship between Plagiarism and Student and Class Characteristics
Several interesting patterns emerge. In Column 1, we see that African-American and Asian students are more likely to plagiarize than other students while students with higher SAT scores are less likely to plagiarize. Indeed, the relationship between SAT score and plagiarism appears convex. The joint model in Column 2 indicates that there is still a significant relationship between SAT score and plagiarism, even after controlling for other factors. The large positive coefficient on African-American students disappears, but the positive effect for Asian students remains significant.
The relationship between SAT score and plagiarism is quite strong. The bottom quintile of students at the school, who scored between 1,000 and 1,200 on the SAT, had plagiarism rates of nearly 14 percent. Using the estimates from Column 2 and extrapolating out of sample, we would predict that students scoring at the national average on the SAT (a score of 1,017) would have rates of 17.7 percent and students at the 25th percentile of national SAT scores (that is, 850) would have a plagiarism rate of 31.7 percent. These results are consistent with prior work indicating that lower-performing students are more apt to plagiarize (McCabe, Treviño, and Butterfield 2001). And perhaps not surprisingly, students taking classes pass/fail are significantly less likely to plagiarize than those taking the course for a letter grade.
As an initial view of the treatment effect, Figure 1 shows histograms of unadjusted plagiarism rates by classroom organized so that the class pairs are adjacent to each other. The asterisks below the horizontal axis indicate pairs comprised of one instructor teaching two classes. In 11 out of 14 pairs, the average plagiarism rate in the control classroom exceeds that in the treatment classroom. In two out of three of the other cases, there were no cases of plagiarism in either classroom. Hence, the plagiarism rate in the control classroom exceeded the rate in the paired treatment class in only one of 14 pairs. Simple nonparametric tests based on these means suggest that the intervention substantially reduced the prevalence of plagiarism. For example, a Wilcoxon rank-sum test for matched-pair data rejects the null hypothesis of equality across the treatment and control conditions with a p-value of .008 for a two-sided test. An analogous test using adjusted classroom means derived from a regression that includes all of the student and classroom covariates shown in Table 2 also rejects the null of equality with a p-value of 0.048.
Unadjusted plagiarism rates in treatment and control classes
A. Baseline Estimates
Tables 3 and 4 present parametric estimates of the treatment effect to properly quantify the magnitude and determine the statistical precision of the effect. We start with specifications that use classroom-aggregate data as a conservative estimate of the precision of our treatment effect estimates. In Panel A of Table 3, the dependent variable is the unadjusted classroom mean plagiarism rate, which ranges from zero to 0.31. Column 1 shows the results from a model that includes no other covariates besides the treatment indicator, weights each classroom observation equally and does not make any adjustment to the standard errors. The resulting point estimate of −0.036 suggests that the treatment reduced plagiarism by roughly 3.6 percentage points, a very large effect given the classroom-level control mean of 5.6 percent (that is, a decrease of 64 percent). With a standard error of 0.024, however, this point estimate is not statistically different than zero. However, after controlling for either class-level observables or pair fixed effects, this impact estimate becomes statistically significant. In particular, introducing pair fixed effects has virtually no effect on the impact estimate but increases its statistical precision appreciably. Furthermore, the robust and HC2 standard errors are quite similar to the conventional standard errors in the pair fixed-effects specification.
Group Data Estimates of the Effect of Treatment Status on Plagiarism
Microdata Estimates of the Effect of Treatment Status on Plagiarism
Panel B shows the key results from the two-step estimation procedure described above, including different sets of controls and with different weights (Equation 4). Our preferred specification in Column 6 includes student characteristics and pair fixed effects, and weights the second-step regression with the inverse of the covariance matrix on the classroom fixed effects in step one. The resulting point estimate of −0.023 is smaller but statistically significant, with a p-value of 0.027. This impact estimate implies a 41 percent reduction in plagiarism relative to the control group mean.
Table 4 presents the estimated treatment effect conditional on student covariates and the pair fixed effects. Interestingly, the BRL-adjusted standard errors shown in Column 3 are nearly identical to the conventional standard errors shown in Column 1, both of which are substantially larger than the typical cluster-robust standard errors shown in Column 2. Indeed, the standard error in Column 3 is virtually identical to the GLS estimate from Table 3, Panel B, Column 6. Columns 4 and 5 present results from the standard block bootstrap-t and the wild cluster bootstrap-t suggested by Cameron et al. (2008), which should provide more efficient estimates. The 95 percent confidence intervals of the t-statistics come from 10,000 replications of the bootstrap. The p-value reported indicates the fraction of the 10,000 replications in which the t-statistic was larger, in absolute value, than −2.684, the t-statistic from the full sample.
Taken together, these results suggest a treatment effect that is large in magnitude, statistically significant and robust to a variety of alternative strategies to calculating the standard errors.17 Table 5 presents several additional robustness checks. Column 1 replicates the estimates from Column 3 of Table 4 as a baseline. Columns 2 and 3 show that using more and less restrictive definitions of plagiarism (that is, similarity scores 15 or higher and eight or higher, respectively) do not materially change our results.
Robustness Tests
Some students in our sample appear in multiple classes. As a result, roughly 11 percent of students were simultaneously in at least one treatment and control classroom. For these students, it is possible that exposure to the treatment in one class might have influenced behavior in other classes. For this reason, the specification in Column 4 assigns treatment status to all observations of students who were in at least one treatment class. The results are virtually identical to the baseline.
In Column 5, we limit the sample to classroom pairs in which the instructor taught both the treatment and control classes. In Column 6, we reestimate the main specification dropping all observations from the four pairs in which one of the instructors in the pair was known to be broadly aware of the objective of the study.18 Although the point estimates differ slightly across these specifications, they are not significantly different than the baseline results.
Columns 7–9 show results for logit and probit models as well as OLS. With the full set of baseline student and classroom controls, many observations drop from the nonlinear models. For this reason, the specifications in Columns 7–9 include a limited set of covariates.19 In addition, we report conventional standard errors, which Table 4 indicates are virtually identical to the BRL-adjusted standard errors. The OLS estimate in Column 7 is slightly large than the baseline estimate shown in Column 1, as one would expect since we drop classroom pairs with no observed cases of plagiarism (and thus no potential treatment effect). More importantly, however, the average marginal effect from the probit (Column 8) and logit (Column 9) models yield very similar results to those in Column 7. This suggests that our baseline OLS results are robust to the use of alternative specifications.
B. Treatment Heterogeneity
Prior literature as well as the analysis shown in Table 2 suggests that the prevalence of plagiarism varies systematically with student characteristics. It thus seems likely that the impact of any particular intervention may also vary across students. Table 6 shows treatment effects separately for several key subgroups. In Columns 1 and 2, we see that the intervention had a similar effect on male and female college students.20 Columns 3–6 show the results separately by year in college. The point estimates are roughly equivalent for all but sophomores (for which the point estimate is essentially zero). More specifically, though the treatment effects for juniors and seniors are estimated with comparative precision, the differences in treatment effects by class are not statistically significant.
Robustness by Student Characteristics
To explore the relationship between initial achievement/cognitive ability and the intervention, we estimate models that allow the treatment effect to vary with a student’s SAT score. Figure 2 shows the treatment effects and control mean estimated by a model that include a quadratic in SAT interacted with the treatment indicator, along with all of the covariates from the primary specification discussed above. Confidence intervals use BRL corrected standard errors. The treatment effect model includes all covariates from the primary specification described above.21
The impact of the web-based tutorial on plagiarism, by student SAT score
To begin, note that the mean plagiarism rate among control students is more than10 percent for students at the bottom of the SAT distribution. The rate declines steadily as SAT score rises, asymptoting to nearly zero at the upper end of the SAT distribution. More interestingly, we see that the intervention had a much larger impact on students at the bottom tail of the SAT distribution.22 Based on the estimates from this model, we would conclude that the intervention reduced the likelihood of plagiarism by roughly 10 percentage points among students with SAT scores below 1,200. While these students comprise just less than 10 percent of the students in our sample, the national average of math and verbal SAT scores among all test-takers in 2007 was 1,017 and the 25th percentile was 850. While the external validity of any intervention trial is open to question, this treatment heterogeneity suggests that the intervention may have a large impact on the typical college student.
In theory, it is possible that the treatment effect may have varied with classroom or instructor characteristics. Unfortunately, with only 28 classrooms and 14 pairs, our ability to detect such differences is quite limited. The results shown in Table 7 suggest that impacts may have been larger in classes with female professors, professors below the rank of full professor and in classes that did not include a warning on the syllabus regarding plagiarism. However, none of these treatment effect differences are statistically significant at conventional levels.
Robustness by Class Characteristics
C. Human Capital or Deterrence?
Our intervention was designed to reduce the prevalence of plagiarism by educating students about what constitutes plagiarism and providing them with effective strategies for avoidance. However, it may also (or even exclusively) be that this intervention reduced plagiarism simply by increasing the perceived likelihood that plagiarism would be detected and prosecuted. To assess the mediating mechanisms by which this intervention was effective, we fielded a web-based survey of the participating students approximately one month after the end of the semester and after the collection of writing assignments for this study had concluded. The survey contained ten questions tapping student attitudes regarding the course and the instructor, along with three true-false questions assessing the student’s knowledge of plagiarism (see Appendix 2). The response rate was 51 percent and did not differ significantly across treatment and control groups.23
Table 8 presents results from analyses that examine the impact of the treatment on survey responses. More specifically, Table 8 reports the estimated treatment effects from OLS regressions that condition on student traits and pair fixed effects, with BRL-adjusted clustered standard errors. The results in the first row of Table 8 indicate that, though students in treatment courses were somewhat more likely to not complete the survey, this difference was not statistically significant. In other words, the response rate to the survey appears balanced across treatment and control conditions.
Impacts on Student Attitudes and Perceptions
Perhaps most interestingly, the next row in Table 8 indicates that students in treatment classes were substantially more likely to correctly answer all the three quiz items, which assessed student understanding of plagiarism. This is not surprising insofar as these items were based directly on the information provided in the online tutorial. The fact that roughly 87 percent of control students answered all three items correctly indicates that many students were aware of much of the information contained in the tutorials. However, the fact that virtually 100 percent of students in treatment classes answered all three items correctly confirms that the intervention provided information to a nontrivial fraction of students and that these students retained such information for at least one semester.24
The remaining rows in Table 8 identify the treatment-control differences for the other survey responses where responses were on a scale of one to five with one indicating strong disagreement with the statement and five reflecting strong agreement.25 Interestingly, the data from Question 6 indicate that students in the treatment courses were significantly more likely to agree that they had a good understanding of plagiarism, a finding consistent with the quiz results and the educational intent of the tutorial. In contrast, the results to Questions 8, 9, and 10 suggest that the intervention did not have a statistically significant deterrent effect. That is, respondents in treatment courses were not significantly more likely to think that a professor would detect plagiarism, respond to it in some way or report it to a judiciary committee. As a composite measure of student perceptions regarding the likelihood of detection and sanction for plagiarism, we also calculated the average of survey Questions 8, 9, and 10. In the last row of Table 8, we see that assignment to the treatment condition is not significantly related to this composite measure.26 Although not definitive, these results suggest that the primary mediating mechanism for the intervention was education rather than deterrence.
VI. Discussion
Rapid technological advances (for example, access to full-text resources and cut-and-paste word processing) have raised concern that plagiarism by college students—already disturbingly high according to many accounts—has dramatically increased. Despite the degree of concern and the corresponding calls for reform, there has been surprisingly little credible evidence on how much student plagiarism actually occurs and on the policy determinants of this illicit behavior. The results presented above shed light on these issues.
Our results demonstrate that a short educational tutorial can sharply reduce the prevalence of plagiarism. The costs of this intervention are quite modest, suggesting it could be scaled easily. It involves very little instructor involvement, requires only 15 minutes on the part of students and the tutorial itself is freely available. Moreover, our evidence suggests that the intervention has the largest impact on lower-ability students, which may make it even more beneficial at a wide range of public and private institutions with less selective admissions than the highly selective institution we study.
However, this study was fielded at one institution so the issue of external validity suggests the need for replication efforts. Another complication noted earlier is that this study may understate the general equilibrium effects of the widespread adoption of this tutorial if there are social multipliers not captured by our class-level design (that is, at the institution level). We also stress that the efficacy of our intervention at scale could conceivably turn on specific design features, especially embedding our tutorial in the context of the specific class rather than in an environment like a student orientation.
In addition, the mechanism through which the tutorial likely operates provides insight that may be applicable beyond plagiarism. An ex-post survey of the participants in this experiment suggested that this tutorial was effective by increasing student knowledge about plagiarism rather than by increasing the perceived probabilities of detection and punishment. These results are consistent with the stylized facts suggesting that students do not understand plagiarism or ethical writing strategies particularly well and that this equilibrium can persist because college instructors often view policing plagiarism and teaching students about it as outside their responsibilities. Similar circumstances (that is, incomplete and costly-to-acquire information about how to behave legally combined with low probabilities of detection and meaningful punishment) are likely to characterize other illicit or illegal behavior such as tax evasion and various types of regulatory compliance. The results presented here suggest that well-designed and targeted information initiatives will provide promising policy levers in these contexts as well.
Appendix 1
Appendix 2
Followup Survey
Questions 1 through 10, which are listed below, had five possible responses: Strongly Agree, Agree, Neither Agree nor Disagree, Disagree, Strongly Disagree.
Question 1—Overall, I enjoyed this class.
Question 2—I found this class to be fairly difficult academically.
Question 3—I found the writing assignment(s) for this class somewhat stressful.
Question 4—I tended to get an early start, rather than procrastinate, on writing assignments for this class.
Question 5—When working on the writing assignments for this class, I paid particular attention to avoiding plagiarism.
Question 6—I have a good understanding of what constitutes plagiarism in academic writing.
Question 7—I know how to avoid plagiarism in my writing assignments.
Question 8—If my writing assignments for this class contained any plagiarism, this instructor would detect it.
Question 9—If this instructor felt that one of my writing assignments contained any plagiarism, he or she would ignore it.
Question 10—If this instructor felt that one of my writing assignments contained any plagiarism, he or she would report it to the [institutional judiciary authority].
Questions 11, 12, and 13 were true/false questions.
Question 11—Suppose you are writing a research paper. You have cut and pasted a lot of information from articles you found on web sites and databases into a Word file on your computer. While writing your essay, you find yourself patching together pieces from different sources, and you have occasionally lost track of which ideas were your own and which were from various articles and websites. You could go back to the original sources but the prospect is daunting. Fortunately, if your professor queries your sources, you can legitimately claim that you didn’t plagiarize because it wasn’t intentional.
Question 12—Suppose it would be quite easy for you to retool whole sections of a paper you have written for a previous to satisfy the requirements of another course you are currently taking. It is acceptable practice to resubmit this edited paper—without checking with either professor—because you are writing a paper for a different professor and a different course.
Question 13—Plagiarism is not limited to taking something from a book; it also includes stealing ideas from a movie, a professor’s lecture, or from an interview on a radio news program.
Footnotes
Thomas S. Dee is a Professor of Public Policy and Economics at the Frank Batten School of Leadership and Public Policy, University of Virginia; Brian A. Jacob is the Walter H. Annenberg Professor of Education Policy at the Gerald R. Ford School of Public Policy, University of Michigan. The authors thank Elias Wash and Jonathan Bartik for their research assistance and Doug Miller for providing advice and programs underlying some of the statistical analysis in the paper. The data used in this article can be obtained beginning October 2012 through September 2015 from Brian Jacob, 735 S. State St. #5318 Ann Arbor, MI 48109-3091, bajacob{at}umich.edu.
↵1. In a survey fielded at nine institutions, Scanlon and Neumann (2002) find that 9.6 percent of students admitted to copying text without attribution. In a survey of students at 83 different campuses, McCabe (2005) reports that nearly 40 percent admitted to paraphrasing or copying a few sentences from a print or Internet source without attribution sometime in the past year. The variation in these estimates is likely to reflect in part differences across the sampled institutions and in the design of the questions as well as possible survey artifacts. For example, Brown and Emmett (2001) examined data from multiple studies of student dishonesty and found that the number of student practices included in the study was related to the overall level of student cheating. Objective measurement of plagiarism, like that used in this study, is less common. However, two small-scale field studies that analyzed the papers from a single college class (Lau et al. 2005; Bilic-Zulle, Azman, Frkovic, and Petrovecki 2008) also suggest that student plagiarism is not a rare event.
↵2. For the sake of simplicity, we allow the effects of this effort on course grades to be conflated with this broader human-capital measure and its private return, w.
↵3. College classrooms are the relevant field setting in this context so this natural field experiment (Harrison and List 2004) should not be confused with laboratory experiment that are sometimes conducted in classroom settings as well.
↵4. One instructor provided no reason for refusing while a second instructor was uncomfortable with using Blackboard despite the facilitation by the research team. Three other instructors refused because they were uncomfortable with the “deception” of students, despite the data-security protocols.
↵5. We secured permission for the use of this material, which was also available for sharing and adaptation under a Creative Commons license. To avoid unintended irony, the tutorial clearly made an attribution to its source. Consistent with the license conditions, our use of this material was noncommercial and our adaption of this material is available for sharing upon request.
↵6. Students who exited the tutorial before completing its entire sequence were allowed to upload their writing assignments. A small number of students may not have completed the tutorial at all because they dropped the course or because they submitted hard copy papers directly to the instructor. We collected all available hard copies for our analysis and assess the implications of study attrition for our key inferences.
↵7. Nonetheless, the use of electronic paper collection as opposed to printed copies could conceivably constitute a study-wide deterrent to plagiarism. In theory, this could compromise the external validity of our results for papers that are collected as printed copies. And the fairly low prevalence of plagiarism in our study suggests this caveat. If such an effect existed it could also muddy our treatment contrast, which would bias us toward finding no effect of the intervention. However, given the magnitude of the apparent treatment effects in this study, this seems less problematic. Furthermore, we suspect that the electronic submission of papers is an increasingly common mechanism.
↵8. As part of the human-subject protocols for this research, all of the collected papers were assigned random identifiers and anonymized (for example, names removed from file name, paper titles and headers) prior to analysis.
↵9. In his recent book on plagiarism, Posner (2007, page 84) discusses Turnitin.com and notes it generates false positives because of indented quotations and the flagging of incidental phrases.
↵10. However, because some unique students were enrolled in more than one participating course, there were 573 unique students in the study.
↵11. A small number of potential papers (n = 31), only one of which was lost to attrition, were from students who were taking a course at the participating institution but were not enrolled there. Because of their unique enrollment status, some data (for example, SAT scores) were unavailable for these students. However, their gender was accurately identified from their first names and other public sources.
↵12. Here we abstract away from student-by-paper as the unit of observation.
↵13. Given the small number of observations (28) and the relatively large number of covariates we have (20), many standard regression techniques that rely on asymptotic results do not work. For example, the SUR and OLS regression approaches described above failed miserably in simulation exercises. We generated a test data set with 28 observations and 20 covariates, which were drawn at random but were set to match the means and covariances of the 20 covariates in our actual data. We then randomly assigned treatment status to 14 of the 28 classrooms, and estimated the SUR and OLS models described above.
↵14. Angrist and Pischke (2009) recommend using the maximum of robust and conventional standard errors for inference since robust standard errors can be subject to considerable sampling variance. In practice, the BRL standard errors are virtually identical to the conventional standard errors, both of which are larger than the standard cluster-robust standard errors. In discussing our results, we present the BRL standard errors, effectively adopting the conservative rule of thumb recommended by Angrist and Pischke (2009).
↵15. We are very grateful to Doug Miller for providing STATA code we use to implement the wild-cluster bootstrap method used in Cameron et al. (2008).
↵16. More specifically, the standard errors reflect cluster adjustments and a “bias reducing linearization” (BRL) adjustment, which we discuss below.
↵17. In results not reported here but available upon request, we examined the impact of the intervention on student grades in subsequent courses. We found no statistically significant or substantively important impacts.
↵18. While we did not systematically inform instructors of the purpose of the study, several instructors insisted on knowing as a condition of participation and others inadvertently discovered the objective while working with the electronic submission system we used.
↵19. These specifications drop observations with missing SAT score (no variation in outcome), drop four classrooms with no variation in outcome, drop the pass-fail indicator variable (no variation), drop the indicators for assistant, associate, and visiting professor (almost no variation), and combine the Hispanic and other race indicators (because the other race indicator has no variation in outcome).
↵20. The treatment estimate for female students becomes smaller and statistically insignificant in models that control for classroom observables instead of pair fixed effects.
↵21. Using a student’s SAT score relative to his or her classroom peers yields virtually identical results.
↵22. The confidence intervals shown in Figure 2 illustrate the limited statistical power we have to detect heterogeneity in treatment effects. If we divide our sample into quintiles based on SAT score, and allow the treatment effect to differ for students in the bottom quintile vs. all other students, we find that the coefficient (standard error) on the interaction Treatment x Bottom Quintile is −0.053 (0.029), with a p-value of 0.069.
↵23. Because of the operating constraints implied by Blackboard’s survey mechanism, student identifiers were not available for the student-level survey responses within participating courses. Blackboard’s design also implied that one course pair could not participate in this followup survey. One instructor effectively merged the Blackboard site for two sections of the same course. This did not complicate the treatment, which could be viewed only by the treatment course students in this pair. However, because Blackboard’s survey mechanism strips individual identifiers, it was not possible to separate treatment and control responses for this pair of courses.
↵24. We chose to present results from the linear model here despite the fact that it yields out-of-sample predictions (that is, 0.866 + 0.157 = 1.02 or 102 percent) because it allows us to present BRL-adjusted standard errors, consistent with the other estimates in the paper. In results available upon request, we confirm that the marginal effects from logit and probit models produce qualitatively similar results.
↵25. In results available upon request, we confirm that we obtain comparable results if we use binary outcomes reflecting the top two categories (that is, agree or strongly agree) instead of the continuous level of agreement measure.
↵26. It should be noted that a specification that includes classroom covariates in addition instead of pair fixed effects yields a point estimate of roughly 0.13, which represents a moderate size effect (that is, roughly 0.25 of the standard deviation of the measure) that is statistically different than zero. This is one of the few instances in which the inclusion of classroom covariates instead of (or in addition to) pair fixed effects leads to any substantive change in our estimates. However, the estimate of 0.13 is not significantly different than the estimate of 0.06 shown in Table 8 here. Our read of these results is that the present study does present compelling evidence in either direction with regard to potential deterrent effects.
- Received November 2010.
- Accepted May 2011.
