## Abstract

We consider statistical inference for regression when data are grouped into clusters, with regression model errors independent across clusters but correlated within clusters. Examples include data on individuals with clustering on village or region or other category such as industry, and state-year differences-in-differences studies with clustering on state. In such settings, default standard errors can greatly overstate estimator precision. Instead, if the number of clusters is large, statistical inference after OLS should be based on cluster-robust standard errors. We outline the basic method as well as many complications that can arise in practice. These include cluster-specific fixed effects, few clusters, multiway clustering, and estimators other than OLS.

## I. Introduction

In an empiricist’s day-today practice, most effort is spent on getting unbiased or consistent point estimates. That is, a lot of attention is given to the parameters (). In this paper, we focus on getting accurate statistical inference, a fundamental component of which is obtaining accurate standard errors (*se,* the estimated standard deviation of ). We begin with the basic reminder that empirical researchers should also really care about getting this part right. An asymptotic 95 percent confidence interval is , and hypothesis testing is typically based on the Wald “*t*-statistic” . Both and *se* are critical ingredients for statistical inference, and we should be paying as much attention to getting a good *se* as we do to obtaining .

In this paper, we consider statistical inference in regression models where observations can be grouped into clusters, with model errors uncorrelated across clusters but correlated within cluster. One leading example of “clustered errors” is individual-level cross-section data with clustering on geographical region, such as village or state. Then model errors for individuals in the same region may be correlated while model errors for individuals in different regions are assumed to be uncorrelated. A second leading example is panel data. Then model errors in different time periods for a given individual (for example, person, firm, or region) may be correlated while model errors for different individuals are assumed to be uncorrelated.

Failure to control for within-cluster error correlation can lead to very misleadingly small standard errors, and consequent misleadingly narrow confidence intervals, large *t*-statistics, and low *p*-values. It is not unusual to have applications where standard errors that control for within-cluster correlation are several times larger than default standard errors that ignore such correlation. As shown below, the need for such control increases not only with increase in the size of within-cluster error correlation, but the need also increases with the size of within-cluster correlation of regressors and with the number of observations within a cluster. A leading example, highlighted by Moulton (1986, 1990), is when interest lies in measuring the effect of a policy variable, or other aggregated regressor, that takes the same value for all observations within a cluster.

One way to control for clustered errors in a linear regression model is to additionally specify a model for the within-cluster error correlation, consistently estimate the parameters of this error correlation model, and then estimate the original model by feasible generalized least squares (FGLS) rather than ordinary least squares (OLS). Examples include random effects estimators and, more generally, random coefficient and hierarchical models. If all goes well, this provides valid statistical inference as well as estimates of the parameters of the original regression model that are more efficient than OLS. However, these desirable properties hold only under the very strong assumption that the model for within-cluster error correlation is correctly specified.

A more recent method to control for clustered errors is to estimate the regression model with limited or no control for within-cluster error correlation, and then, post-estimation, obtain “cluster-robust” standard errors proposed by White (1984, pp. 134–142) for OLS with a multivariate dependent variable (directly applicable to balanced clusters), by Liang and Zeger (1986) for linear and nonlinear models, and by Arellano (1987) for the fixed effects estimator in linear panel models. These cluster-robust standard errors do not require specification of a model for within-cluster error correlation but do require the additional assumption that the number of clusters, rather than just the number of observations, goes to infinity.

Cluster-robust standard errors are now widely used, popularized in part by Rogers (1993), which incorporated the method in Stata, and by Bertrand, Duflo, and Mullainathan (2004), which pointed out that many differences-in-differences studies failed to control for clustered errors, and those that did often clustered at the wrong level. Cameron and Miller (2011) and Wooldridge (2003, 2006) provide surveys, and lengthy expositions are given in Angrist and Pischke (2009) and Wooldridge (2010).

One goal of this paper is to provide the practitioner with the methods to implement cluster-robust inference. To this end, we include in the paper reference to relevant Stata commands (for Version 13) because Stata is the computer package most often used in applied microeconometrics research. And we will post on our websites more expansive Stata code and the data sets used in this paper. A second goal is presenting how to deal with complications such as determining when there is a need to cluster, incorporating fixed effects, and inference when there are few clusters. A third goal is to provide an exposition of the underlying econometric theory as this can aid in understanding complications. In practice, the most difficult complication to deal with can be “few” clusters, see Section VI. There is no clear-cut definition of “few”; depending on the situation, “few” may range from less than 20 to less than 50 clusters in the balanced case.

We focus on OLS, for simplicity and because this is the most commonly used estimation method in practice. Section II presents the basic results for OLS with clustered errors. In principle, implementation is straightforward as econometrics packages include cluster-robust as an option for the commonly used estimators; in Stata it is the vce(cluster) option. The remainder of the survey concentrates on complications that often arise in practice. Section III addresses how the addition of fixed effects impacts cluster-robust inference. Section IV deals with the obvious complication that it is not always clear what to cluster over. Section V considers clustering when different ways exist to do so and these ways are not nested in each other. Section VI considers how to adjust inference when there are just a few clusters as, without adjustment, test statistics based on the cluster-robust standard errors overreject and confidence intervals are too narrow. Section VII presents extension to the full range of estimators—instrumental variables, nonlinear models such as logit and probit, and generalized method of moments. Section VIII presents both empirical examples and real-data based simulations. Concluding thoughts are given in Section IX.

## II. Cluster-Robust Inference

In this section, we present the fundamentals of cluster-robust inference. For these basic results, we assume that the model does not include cluster-specific fixed effects, that it is clear how to form the clusters, and that there are many clusters. We relax these conditions in subsequent sections.

Clustered errors have two main consequences: They (usually) reduce the precision of , and the standard estimator for the variance of , is (usually) biased downward from the true variance. Computing cluster-robust standard errors is a fix for the latter issue. We illustrate these issues, initially in the context of a very simple model and then in the following subsection in a more typical model.

### A. A Simple Example

For simplicity, we begin with OLS with a single regressor that is nonstochastic, and assume no intercept in the model. The results extend to multiple regression with stochastic regressors.

Let *y _{i}* = β

*x*+

_{i}*u*,

_{i}*i*= 1, …,

*N*, where

*x*is nonstochastic and E[

_{i}*u*] = 0. The OLS estimator can be reexpressed as , so in general

_{i}(1)

If errors are uncorrelated over *i*, then . In the simplest case of homoskedastic errors, V[*u _{i}*] = σ

^{2}and Equation 1 simplifies to .

If instead errors are heteroskedastic, then Equation 1 becomes

using since E[*u _{i}*] = 0. Implementation seemingly requires consistent estimates of each of the

*N*error variances . In a very influential paper, one that extends naturally to the clustered setting, White (1980) noted that instead all that is needed is an estimate of the scalar , and that one can simply use , where is the OLS residual, provided

*N*→ ∞. This leads to estimated variance

The resulting standard error for is often called a robust standard error, though a better, more precise, term is heteroskedastic-robust standard error.

What if errors are correlated over *i*? In the most general case where all errors are correlated with each other,

so

The obvious extension of White (1980) is to use but this equals zero since . Instead, one needs to first set a large fraction of the error correlations E[*u _{i}*

*u*] to zero. For time series data with errors assumed to be correlated only up to, say,

_{j}*m*periods apart as well as heteroskedastic, White’s result can be extended to yield a heteroskedastic-and autocorrelation-consistent (HAC) variance estimate; see Newey and West (1987).

In this paper, we consider clustered errors, with E[*u _{i}*

*u*] = 0 unless observations

_{j}*i*and

*j*are in the same cluster (such as same region). Then

(2)

where the indicator function **1**[*A*] equals 1 if event *A* happens and equals 0 if event *A* does not happen. Provided the number of clusters goes to infinity, we can use the variance estimate

(3)

This estimate is called a cluster-robust estimate though, more precisely, it is heteroskedastic-and cluster-robust. This estimate reduces to in the special case that there is only one observation in each cluster.

Typically exceeds due to the addition of terms when *i* ≠ *j*. The amount of increase is larger (1) the more positively associated are the regressors across observations in the same cluster (via *x _{i}*

*x*in Equation 3), (2) the more correlated are the errors (via E[

_{j}*u*

_{i}*u*] in Equation 2), and (3) the more observations are in the same cluster (via

_{j}**1**[

*i*,

*j*in same cluster] in Equation 3).

There are several take-away messages. First there can be great loss of efficiency in OLS estimation if errors are correlated within cluster rather than completely uncorrelated. Intuitively, if errors are positively correlated within cluster, then an additional observation in the cluster no longer provides a completely independent piece of new information. Second, failure to control for this within-cluster error correlation can lead to using standard errors that are too small, with consequent overly narrow confidence intervals, overly large *t*-statistics, and overrejection of true null hypotheses. Third, it is straightforward to obtain cluster-robust standard errors, though they do rely on the assumption that the number of clusters goes to infinity (see Section VI for the few clusters case).

### B. Clustered Errors and Two Leading Examples

Let *i* denote the *i ^{th}* of

*N*individuals in the sample and

*g*denote the

*g*of

^{th}*G*clusters. Then for individual

*i*in cluster

*g*the linear model with (one-way) clustering is

(4)

where * x_{ig}* is a

*K*× 1 vector. As usual, it is assumed that E[

*u*|

_{ig}*] = 0. The key assumption is that errors are uncorrelated across clusters while errors for individuals belonging to the same cluster may be correlated. Thus*

**x**_{ig}(5)

#### 1. Example 1: individuals in cluster

Hersch (1998) uses cross-section individual-level data to estimate the impact of job injury risk on wages. As there is no individual-level data on job-injury rate, a more aggregated measure such as job injury risk in the individual’s industry is used as a regressor. Then for individual *i* (with *N* = 5,960) in industry *g* (with *G* = 211)

The regressor *x _{g}* is perfectly correlated within industry. The error term will be positively correlated within industry if the model systematically overpredicts (or underpredicts) wages in a given industry. In this case, default OLS standard errors will be downward biased.

To measure the extent of this downward bias, suppose errors are equicorrelated within cluster, so Cor[*u _{ig}*,

*u*] = ρ for all

_{jg}*i*≠

*j*. This pattern is suitable when observations can be viewed as exchangeable, with ordering not mattering. Common examples include the current one, individuals or households within a village or other geographic unit (such as state), individuals within a household, and students within a school. Then a useful approximation is that for the

*k*regressor the default OLS variance estimate based on

^{th}*s*

^{2}(

*)*

**X′X**^{-1}, where

*s*is the standard error of the regression, should be inflated by

(6)

where ρ* _{xk}* is a measure of the within-cluster correlation of

*x*, ρ

_{igk}*is the within-cluster error correlation, and is the average cluster size. The result, Equation 6, is exact if clusters are of equal size (“balanced” clusters) and ρ*

_{u}*= 1 for all regressors (Kloek 1981); see Scott and Holt (1982) and Greenwald (1983) for the general result with a single regressor.*

_{xk}This very important and insightful result is that the variance inflation factor is increasing in

(1) the within-cluster correlation of the regressor

(2) the within-cluster correlation of the error

(3) the number of observations in each cluster.

For clusters of unequal size replace in (6) by ; see Moulton (1986, p. 387). Note that there is no cluster problem if any one of the following occur: .

In an influential paper, Moulton (1990) pointed out that in many settings the inflation factor τ_{k} can be large even if ρ* _{u}* is small. It considered a log earnings regression using March CPS data (

*N*= 18,946), regressors aggregated at the state level (

*G*= 49), and errors correlated within state . The average group size was 18,946 / 49 = 387, ρ

_{xk}= 1 for a state-level regressor, so Equation 6 yields . The weak correlation of errors within state was still enough to lead to cluster-corrected standard errors being times larger than the (incorrect) default standard errors!

In such examples of cross-section data with an aggregated regressor, the cluster-robust standard errors can be much larger despite low within-cluster error correlation because the regressor of interest is perfectly correlated within cluster and there may be many observations per cluster.

#### 2. Example 2: differences-in-differences (DiD) in a state-year panel

Interest may lie in how wages respond to a binary policy variable *d _{ts}* that varies by state and over time. Then at time

*t*in state

*s*

where we assume independence over states, so the ordering of subscripts (*t*, *s*) corresponds to (*i*, *g*) in Equation 4, and α* _{s}* and δ

*are state and year effects.*

_{t}The binary regressor *d _{ts}* equals one if the policy of interest is in effect and equals zero otherwise. The regressor

*d*is often highly serially correlated since, for example,

_{ts}*d*will equal a string of zeroes followed by a string of ones for a state that switches from never having the policy in place to forever after having the policy in place. The error

_{ts}*u*is correlated over time for a given state if the model systematically overpredicts (or underpredicts) wages in a given state. Again, the default standard errors are likely to be downward biased.

_{ts}In the panel data case, the within-cluster (that is, within-individual) error correlation decreases as the time separation increases so errors are not equicorrelated. A better model for the errors is a time-series model such as an autoregressive error of order one that implies that Cor [*u _{ts}*,

*u*] =ρ

_{t′s}^{|t-t′|}. The true variance of the OLS estimator will again be larger than the OLS default although the consequences of clustering are less extreme than in the case of equicorrelated errors (see Cameron and Miller 2011, Section 2.3, for more detail).

In such DiD examples with panel data, the cluster-robust standard errors can be much larger than the default because both the regressor of interest and the errors are highly correlated within cluster. Note also that this complication can exist even with the inclusion of fixed effects (see Section III).

The same problems arise if we additionally have data on individuals, so that

In an influential paper, Bertrand, Duflo, and Mullainathan (2004) demonstrated the importance of using cluster-robust standard errors in DiD settings. Furthermore, the clustering should be on state, assuming error independence across states. The clustering should not be on state-year pairs because, for example, the error for California in 2010 is likely to be correlated with the error for California in 2009.

The issues raised here are relevant for any panel data application, not just DiD studies. The DiD panel example with binary policy regressor is often emphasized in the cluster-robust literature because it is widely used and it has a regressor that is highly serially correlated, even after mean-differencing to control for fixed effects. This serial correlation leads to a potentially large difference between cluster-robust and default standard errors.

### C. The Cluster-Robust Variance Matrix Estimate

Stacking all observations in the *g ^{th}* cluster, the Model 4 can be written as

where * y_{g}* and

*are*

**u**_{g}*N*× 1 vectors,

_{g}*is an*

**X**_{g}*N*×

_{g}*K*matrix, and there are

*N*observations in cluster

_{g}*g*. Further stacking

*,*

**y**_{g}*and*

**X**_{g}*over the*

**u**_{g}*G*clusters then yields the model

The OLS estimator is

In general, the variance matrix conditional on * X* is

(7)

with

(8)

Given error independence across clusters, V[* u*|

*] has a block-diagonal structure, and Equation 8 simplifies to*

**X**(9)

The matrix **B**_{clu}, the middle part of the “sandwich matrix” Equation 7, corresponds to the numerator of Equation 2. **B**_{clu} can be written as:

where ω* _{ig, jg}* = E[

*u*

_{ig}*u*|

_{jg}

**X***g*] is the error covariance for the

*ig*and

^{th}*jg*observations. We can gain a few insights from inspecting this equation. The term

^{th}*(and hence ) will be bigger when: (1) regressors within cluster are correlated, (2) errors within cluster are correlated so ω*

**B***is nonzero, (3)*

_{ig, jg}*N*is large, and (4) the within-cluster regressor and error correlations are of the same sign (the usual situation). These conditions mirror the more precise Moulton result for the special case of equicorrelated errors given in Equation 6. Both examples in Section II had high within-cluster correlation of the regressor, the DiD example additionally had high within-cluster (serial) correlation of the error and the Moulton (1990) example additionally had

_{g}*N*large.

_{g}Implementation requires an estimate of **B**_{clu} given in Equation 9. The cluster-robust estimate of the variance matrix (CRVE) of the OLS estimator is the sandwich estimate

(10)

where

(11)

and is the vector of OLS residuals for the *g ^{th}* cluster. Formally, Equation 10 and Equation 11 provides a consistent estimate of the variance matrix if as

*G*→ ∞. Initial derivations of this estimator, by White (1984, pp. 134–42) for balanced clusters, and by Liang and Zeger (1986) for unbalanced, assumed a finite number of observations per cluster. Hansen (2007a) showed that the CRVE can also be used if

*N*→ ∞, the case for long panels, in addition to

_{g}*G*→ ∞. Carter, Schnepel, and Steigerwald (2013) considers unbalanced panels with either

*N*fixed or

_{g}*N*→ ∞. The sandwich formula for the CRVE extends to many estimators other than OLS; see Section VII.

_{g}Algebraically, the estimator Equation 10 and Equation11 equals Equations 7 and 9 with E[**u**_{g}* u^{′}_{g}*] replaced with . What is striking about this is that for each cluster

*g*, the

*N*×

_{g}*N*matrix is bound to be a very poor estimate of the

_{g}*N*×

_{g}*N*matrix E[

_{g}

**u**_{g}*]—there is no averaging going on to enable use of a Law of Large Numbers. The “magic” of the CRVE is that despite this, by averaging across all*

**u**^{′}_{g}*G*clusters in Equation 11, we are able to get a consistent variance estimate. This fact helps us to understand one of the limitations of this method in practice—the averaging that makes accurate for is an average based on the number of clusters

*G*. In applications with few clusters, this can lead to problems that we discuss below in Section VI.

Finite-sample modifications of Equation 11 are typically used to reduce downward bias in due to finite *G*. Stata uses in Equation 11 rather than , with

(12)

In general *c* ≃ *G* / (*G* – 1), though Section IIIB addresses an important exception when fixed effects are directly estimated. Some other packages such as SAS use *c* = *G* / (*G* – 1), a simpler correction that is also used by Stata for extensions to nonlinear models. Either choice of *c* usually lessens, but does not fully eliminate, the usual downward bias in the CRVE. Other finite-cluster corrections are discussed in Section VI but there is no clear best correction.

### D. Feasible GLS

If errors are correlated within cluster, then in general OLS is inefficient and feasible GLS may be more efficient.

Suppose we specify a model for Ω* _{g}* = E[

**u**_{g}*|*

**u′**_{g}*] in Equation 9, such as within-cluster equicorrelation. Then the GLS estimator is (*

**X**_{g}*Ω*

**X′**^{−1}

*)*

**X**^{−1}

*Ω*

**X′**^{−1}

*, where Ω = Diag[Ω*

**y**_{g}]. Given a consistent estimate of Ω, the feasible GLS estimator of

**β**is

(13)

The FGLS estimator is second-moment efficient, with variance matrix

(14)

under the strong assumption that the error variance Ω is correctly specified.

Remarkably, the cluster-robust method of the previous section can be extended to FGLS. Essentially, OLS is the special case where Ω* _{g}* = σ

^{2}

*. The cluster-robust estimate of the asymptotic variance matrix of the FGLS estimator is*

**I**_{Ng}(15)

where . This estimator requires that * u_{g}* and

*are uncorrelated when*

**u**_{h}*g*≠

*h*, and that

*G*→ ∞, but permits E[

**u**_{g}*|*

**u′**_{g}*≠ Ω*

**X**_{g}_{g}]. The approach of specifying a model for the error variances and then doing inference that guards against misspecification of this model is especially popular in the biostatistics literature that calls Ω

*a “working” variance matrix (see, for example, Liang and Zeger 1986).*

_{g}There are many possible candidate models for Ω* _{g}*, depending on the type of data being analyzed.

For individual-level data clustered by region, the example in Section IIB1, a common starting point model is the random effects (RE) model. The error in Model 4 is specified to have two components:

(16)

where α_{g} is a cluster-specific error or common shock that is assumed to be independent and identically distributed (iid) , and *ε** _{ig}* is an idiosyncratic error that is assumed to be iid . Then and . It follows that the intraclass correlation of the error so this model implies equicorrelated errors within cluster. Richer models that introduce heteroskedasticity include random coefficients models and hierarchical linear models.

For panel data, the example in Section IIB2, a range of time series models for *u _{it}* may be used, including autoregressive and moving average error models. Analysis of within-cluster residual correlation patterns after OLS estimation can be helpful in selecting a model for Ω

*.*

_{g}Note that in all cases if cluster-specific fixed effects are included as regressors and *N _{g}* is small then bias-corrected FGLS should be used; see Section IIIC.

The efficiency gains of FGLS need not be great. As an extreme example, with equicorrelated errors, balanced clusters, and all regressors invariant within cluster (* x_{ig}* =

*) the FGLS estimator equals the OLS estimator—and so there is no efficiency gain to FGLS. With equicorrelated errors and general*

**x**_{g}*, Scott and Holt (1982) provides an upper bound to the maximum proportionate efficiency loss of OLS, compared to the variance of the FGLS estimator, of 1 / [1 + (4(1 − ρ*

**X***)[1 + (*

_{u}*N*

_{max}− 1)ρ

*) / (*

_{u}*N*

_{max}× ρ

*)*

_{u}^{2}],

*N*

_{max}= max{

*N*

_{1}, …,

*N*}. This upper bound is increasing in the error correlation ρ

_{G}*and the maximum cluster size*

_{u}*N*

_{max}. For low ρ

*, the maximal efficiency gain can be low. For example, Scott and Holt (1982) notes that for ρ*

_{u}*= 0.05 and*

_{u}*N*

_{max}= 20 there is at most a 12 percent efficiency loss of OLS compared to FGLS. With ρ

*= 0.2 and*

_{u}*N*

_{max}= 100, the efficiency loss could be as much as 86 percent, though this depends on the nature of

*.*

**X**There is no clear guide to when FGLS may lead to considerable improvement in efficiency, and the efficiency gains can be modest. However, especially in models without cluster-specific fixed effects, implementation of FGLS and use of Equation 15 to guard against misspecification of Ω* _{g}* is straightforward. And even modest efficiency gains can be beneficial. It is remarkable that current econometric practice with clustered errors ignores the potential efficiency gains of FGLS.

### E. Implementation for OLS and FGLS

For regression software that provides a cluster-robust option, implementation of the CRVE for OLS simply requires defining for each observation a cluster identifier variable that takes one of *G* distinct values according to the observation’s cluster, and then passing this cluster identifier to the estimation command’s cluster-robust option. For example, if the cluster identifier is *id_clu*, then Stata OLS command *regress y x* becomes *regress y x, vce(cluster id_clu)*.

Wald hypothesis tests and confidence intervals are then implemented in the usual way. In some cases, however, joint tests of several hypotheses and of overall statistical significance may not be possible. The CRVE is guaranteed to be positive semi-definite, so the estimated variance of the individual components of are necessarily nonnegative. But is not necessarily positive definite, so it is possible that the variance matrix of linear combinations of the components of is singular. The rank of equals the rank of defined in Equation 11. Because , where is a *K* × *G* matrix, it follows that the rank of , and hence that of , is at most the rank of * C*. Because , the rank of equals min(

*K*,

*G*– 1), though it can be less than this in some examples such as perfect collinearity of regressors and cluster-specific dummy regressors (see Section IIIB for the latter).

A common setting is to have a richly specified model with thousands of observations in far fewer clusters, leading to more regressors than clusters. Then rank-deficient, so it will not be possible to perform an overall *F* test of the joint statistical significance of all regressors. And in a log-wage regression with occupation dummies and clustering on state, we cannot test the joint statistical significance of the occupation dummies if there are more occupations than states. But it is still okay to perform statistical inference on individual regression coefficients and to do joint tests on a limited number of restrictions (potentially as many as min(*K*, *G* – 1)).

Regression software usually also includes a panel data component. Panel commands may enable not only OLS with cluster-robust standard errors but also FGLS for some models of within-cluster error correlation with default (and possibly cluster-robust) standard errors. It is important to note that those panel data commands that do not explicitly use time-series methods—an example is FGLS with equicorrelation of errors within-cluster—can be applied more generally to other forms of clustered data, such as individual-level data with clustering on geographic region.

For example, in Stata first give the command *xtset id_clu* to let Stata know that the cluster-identifier is variable *id_clu*. Then the Stata command *xtreg y x, pa corr(ind) vce(robust)* yields OLS estimates with cluster-robust standard errors. Note that for Stata xt commands, option *vce(robust)* is generally interpreted as meaning cluster-robust; this is always the case from Version 12.1 on. The xt commands use standard normal critical values whereas command *regress* uses Student’s *T*(*G* – 1) critical values; see Sections VI and VIIA for further discussion.

For FGLS estimation, the commands vary with the model for Ω* _{g}*. For equicorrelated errors, a starting point for the example in Section IIB1, the FGLS estimator can be obtained using command

*xtreg y x, pa corr(exch)*or command

*xtreg y x, re*; slightly different estimates are obtained due to slightly different estimates of the equicorrelation. For FGLS estimation of hierarchical models that are richer than a random effects model, use Stata command

*mixed*(Version 13) or

*xtmixed*(earlier versions). For FGLS with panel data and time variable time, first give the command

*xtset id_clu time*to let Stata know both the cluster-identifier and time variable. A starting point for the example in Section IIB2 is an autoregressive error of order 1, estimated using command

*xtreg y x, pa corr(ar 1)*. Stata permits a wide range of possible models for serially correlated errors.

In all of these FGLS examples, the reported standard errors are the default ones that assume correct specification of Ω* _{g}*. Better practice is to add option

*vce(robust)*for

*xtreg*commands, or option

*vce(cluster id_clu)*for mixed commands, as this yields standard errors that are based on the cluster-robust variance defined in Equation 15.

### F. Cluster-Bootstrap Variance Matrix Estimate

Not all econometrics packages compute cluster-robust variance estimates, and even those that do may not do so for all estimators. In that case, one can use a pairs cluster bootstrap that, like the CRVE, gives a consistent estimate of when errors are clustered.

To implement this bootstrap, do the following steps *B* times: (1) form *G* clusters by resampling with replacement *G* times from the original sample of clusters, and (2) compute the estimate of **β**, denoted in the *b ^{th}* bootstrap sample. Then, given the

*B*estimates , compute the variance of these

where , and *B* = 400 should be more than adequate in most settings. It is important that the resampling be done over entire clusters rather than over individual observations. Each bootstrap resample will have exactly *G* clusters, with some of the original clusters not appearing at all while others of the original clusters may be repeated in the resample two or more times. The terms “pairs” is used as (* y_{g}*,

*) are resampled as a pair. The term “nonparametric” is also used for this bootstrap. Some alternative bootstraps hold*

**X**_{g}*fixed while resampling. For finite clusters, if uses in Equation 11 then for comparability should be multiplied by the constant*

**X**_{g}*c*defined in Equation 12. The pairs cluster bootstrap leads to a cluster-robust variance matrix for OLS with rank

*K*even if

*K*>

*G*.

An alternative resampling method that can be used is the leave-one-cluster-out jackknife. Then, letting denote the estimator of **β** when the *g ^{th}* cluster is deleted,

where . This older method can be viewed as an approximation to the bootstrap that does not work as well for nonlinear estimators. It is used less often than the bootstrap and has the same rank as the CRVE.

Unlike a percentile-*t* cluster bootstrap, presented later, the pairs cluster bootstrap and cluster jackknife variance matrix estimates are no better asymptotically than the CRVE so it is best and quickest to use the CRVE if it is available. But the CRVE is not always available, especially for estimators more complicated than OLS. In that case, one can instead use the pairs cluster bootstrap, though the end of Section VIC indicates potential pitfalls if there are few clusters, or even the cluster jackknife.

In Stata, the pairs cluster bootstrap for OLS without fixed effects can be implemented in several equivalent ways including: *regress y x, vce(boot, cluster(id_clu) reps(400) seed(10101)); xtreg y x, pa corr(ind) vce(boot, reps(400) seed(10101))*; and *bootstrap, cluster(id_clu) reps(400) seed(10101) : regress y x*. The last variant can be used for estimation commands and user-written programs that do not have a *vce(boot)* option. We recommend 400 bootstrap iterations for published results and for replicability one should always set the seed.

For the jackknife the commands are instead, respectively, *regress y x, vce(jack, cluster(id_clu)); xtreg y x, pa corr(ind) vce(jack)*; and *jackknife, cluster(id_clu): regress y x*. For Stata *xt* commands, options *vce(boot)* and *vce(jack)* are generally interpreted as meaning cluster bootstrap and cluster jackknife; always so from Stata 12.1 on.

## III. Cluster-Specific Fixed Effects

The cluster-specific fixed effects (FE) model includes a separate intercept for each cluster, so

(17)

where *dh _{ig}*, the

*h*of

^{th}*G*dummy variables, equals one if the

*ig*observation is in cluster

^{th}*h*and equals zero otherwise.

There are several different ways to obtain the same cluster-specific fixed effects estimator. The two most commonly used are the following. The least squares dummy variable (LSDV) estimator directly estimates the second line of Equation 17, with OLS regression of *y _{ig}* on

*and the*

**x**_{ig}*G*dummy variables

*d*1

*, …,*

_{ig}*dG*, in which case the dummy variable coefficients where and . The within estimator, also called the fixed effects estimator, estimates

_{ig}**β**just by OLS regression in the within or mean-differenced model

(18)

The main reason that empirical economists use the cluster-specific FE estimator is that it controls for a limited form of endogeneity of regressors. Suppose in Equation 17 that we view α_{g} + *u*_{ig} as the error, and the regressors * x_{ig}* are correlated with this error but only with the cluster-invariant component, that is, Cov[

*, α*

**x**_{ig}*] ≠ 0 while Cov[*

_{g}*,*

**x**_{ig}*u*] = 0. Then OLS and FGLS regression of

_{ig}*y*on

_{ig}*, as in Section II, leads to inconsistent estimation of*

**x**_{ig}**β**. Mean-differencing Equation 17 leads to the within Model 18 that has eliminated the problematic cluster-invariant error component α

_{g}. The resulting FE estimator is consistent for

**β**if either

*G*→ ∞ or

*N*→ ∞.

_{g}The cluster-robust variance matrix formula given in Section II carries over immediately to OLS estimation in the FE model, again assuming *G* → ∞.

In the remainder of this section, we consider some practicalities. First, including fixed effects generally does not control for all the within-cluster correlation of the error and one should still use the CRVE. Second, when cluster sizes are small and degrees-of-freedom corrections are used, the CRVE should be computed by within rather than LSDV estimation. Third, FGLS estimators need to be bias-corrected when cluster sizes are small. Fourth, tests of fixed versus random effects models should use a modified version of the Hausman test.

### A. Do Fixed Effects Fully Control for Within-Cluster Correlation?

While cluster-specific effects will control for part of the within-cluster correlation of the error, in general they will not completely control for within-cluster error correlation (not to mention heteroskedasticity). So the CRVE should still be used. There are several ways to make this important point.

Suppose we have data on students in classrooms in schools. A natural model, a special case of a hierarchical model, is to suppose that there is both an unobserved school effect and, on top of that, an unobserved classroom effect. Letting *i* denote individual, *s* school, and *c* classroom, we have . A regression with school-level fixed effects (or random effects) will control for within-school correlation but not the additional within-classroom correlation.

Suppose we have a short panel (*T* fixed, *N* → ∞) of uncorrelated individuals and estimate . Then the error *u _{it}* may be correlated over time (that is, within-cluster) due to omitted factors that evolve progressively over time. Inoue and Solon (2006) provides a test for this serial correlation. Cameron and Trivedi (2005, p. 710) presents an FE individual-level panel data log-earnings regressed on log-hours example with cluster-robust standard errors four times the default. Serial correlation in the error may be due to omitting lagged

*y*as a regressor. When

*y*

_{i,t–1}is included as an additional regressor in the FE model, the Arellano-Bond estimator is used and, even with

*y*

_{i,t–1}included, the Arellano-Bond methodology requires that we first test whether the remaining error

*u*is serially correlated.

_{it}Finally, suppose we have a single cross-section (or a single time series). This can be viewed as regression on a single cluster. Then in the model (or ), the intercept is the cluster-specific fixed effect. There are many reasons for why the error *u _{i}* (or

*u*) may be correlated in this regression.

_{t}### B. Cluster-Robust Variance Matrix with Fixed Effects

Since inclusion of cluster-specific fixed effects may not fully control for cluster correlation (and/or heteroskedasticity), default standard errors that assume *u _{ig}* to be iid may be invalid. So one should use cluster-robust standard errors.

Arellano (1987) showed that defined in Equations 10 and 11 remains valid for the within estimator that controls for inclusion of *G* cluster-specific fixed effects, provided *G* → ∞ and *N _{g}* is small. If instead one obtains the LSDV estimator, the CRVE formula gives the same CRVE for as that for the within estimator, with the important proviso that the same degrees-of-freedom adjustment must be used—see below. The fixed effects estimates obtained for the LSDV estimator are essentially based only on

*N*observations, so is inconsistent for , just as is inconsistent for α

_{g}_{g}.

Hansen (2007a, p. 600) shows that this CRVE can also be used if additionally *N _{g}* → ∞, for both the case where within-cluster correlation is always present (for example, for many individuals in each village) and for the case where within-cluster correlation eventually disappears (for example, for panel data where time series correlation disappears for observations far apart in time). The rates of convergence are in the first case and in the second case but the same asymptotic variance matrix is obtained in either case. Kézdi (2004) analyzed the CRVE in FE models for a range of values of

*G*and

*N*.

_{g}It is important to note that, while LSDV and within estimation lead to identical estimates of **β**, they can yield different standard errors due to different finite sample degrees-of-freedom correction.

It is well known that if default standard errors are used, that is, it is assumed that *u _{ig}* in Equation 17 is iid, then one can safely use standard errors after LSDV estimation as this correctly views the number of parameters as

*G*+

*K*rather than

*K*. If instead the within estimator is used, however, manual OLS estimation of Equation 18 will mistakenly view the number of parameters to equal

*K*rather than

*G*+

*K*. (Built-in panel estimation commands for the within estimator—that is, a fixed effects command—should remain okay to use because they should be programmed to use

*G*+

*K*in calculating the standard errors.)

It is not well known that if cluster-robust standard errors are used, and cluster sizes are small, then inference should be based on the within estimator standard errors. We thank Arindrajit Dube and Jason Lindo for bringing this issue to our attention. Within and LSDV estimation lead to the same cluster-robust standard errors if we apply Formula 11 to the respective regressions, or if we multiply this estimate by *c* = *G* / (*G* – 1). Differences arise, however, if we multiply by the small-sample correction *c* given in Equation12. Within estimation sets *c* = [*G* / (*G* −1)] / [(*N* −1) / (*N* − *K* + 1)] since there are only (*K* – 1) regressors—the within model is estimated without an intercept. LSDV estimation uses *c* = [*G* / (*G* −1)] / [(*N* −1) / (*N* − *G* – *K* + 1)] since the *G* cluster dummies are also included as regressors. For balanced clusters with *N _{g}* =

*N*

_{*}and

*G*large relative to

*K*,

*c*≃ 1 for within-estimation and

*c*≃

*N*

_{*}/ (

*N*

_{*}– 1) for LSDV estimation. Suppose there are only two observations per cluster, due to only two individuals per household or two time periods in a panel setting, so

*N*=

_{g}*N*

_{*}= 2. Then

*c*≃ 2 / (2 – 1) = 2 for LSDV estimation, leading to CRVE that is twice that from within estimation. Within estimation leads to the correct finite-sample correction.

In Stata, the within command *xtreg y x, fe vce(robust)* gives the desired CRVE. The alternative LSDV commands *regress y x i.id_clu*, *vce(cluster id_clu)* and, equivalently, *regress y x, absorb(id_clu) vce(cluster id_clu)* use the wrong degrees-of-freedom correction. If a CRVE is needed, then use *xtreg*. If there is reason to instead use *regress i.id*, then the cluster-robust standard errors should be multiplied by the square root of [*N* − (*K* − 1)] / [*N* − *G* − (*K* − 1)], especially if *N _{g}* is small.

The inclusion of cluster-specific dummy variables increases the dimension of the CRVE but does not lead to a corresponding increase in its rank. To see this, stack the dummy variable *dh _{ig}* for cluster

*g*into the

*N*× 1 vector

_{g}*. Then , by the OLS normal equations, leading to the rank of falling by one for each cluster-specific effect. If there are*

**dh**_{g}*k*regressors varying within cluster and

*G*– 1 dummies then, even though there are

*K*+

*G*– 1 parameters

**β**, the rank of is only the minimum of

*K*and

*G*– 1. And a test that α

_{1}, …, α

*are jointly statistically significant is a test of*

_{G}*G*– 1 restrictions (since the intercept or one of the fixed effects needs to be dropped). So even if the cluster-specific fixed effects are consistently estimated (that is, if

*N*→ ∞), it is not possible to perform this test if

_{g}*K*<

*G*– 1, which is often the case.

If cluster-specific effects are present, then the pairs cluster bootstrap must be adapted to account for the following complication. Suppose Cluster 3 appears twice in a bootstrap resample. Then if clusters in the bootstrap resample are identified from the original cluster-identifier, the two occurrences of Cluster 3 will be incorrectly treated as one large cluster rather than two distinct clusters.

In Stata, the bootstrap option *idcluster* ensures that distinct identifiers are used in each bootstrap resample. Examples are *regress y x i.id_clu*, *vce(boot, cluster(id_clu) idcluster(newid) reps(400) seed(10101))* and, more simply, *xtreg y x, fe vce(boot, reps(400) seed(10101))*, as in this latter case Stata automatically accounts for this complication.

### C. Feasible GLS with Fixed Effects

When cluster-specific fixed effects are present, more efficient FGLS estimation can become more complicated. In particular, if asymptotic theory relies on *G* → ∞ with *N _{g}* fixed, the α

*cannot be consistently estimated. The within estimator of*

_{g}**β**is nonetheless consistent, as α

*disappears in the mean-differenced model. But the resulting residuals are contaminated because they depend on both and , and these residuals will be used to form a FGLS estimator. This leads to bias in the FGLS estimator so one needs to use bias-corrected FGLS unless*

_{g}*N*→ ∞. The correction method varies with the model for Ω

_{g}*= V[*

_{g}*], and currently there are no Stata user-written commands to implement these methods.*

**u**_{g}For panel data, a commonly used model specifies an AR(p) model for the errors *u _{ig}* in Equation 17. If fixed effects are present, then there is a bias (of order

*N*

_{g}^{-1}) in estimation of the AR(p) coefficients. Hansen (2007b) obtains bias-corrected estimates of the AR(p) coefficients and uses these in FGLS estimation. Hansen (2007b) in simulations shows considerable efficiency gains in bias-corrected FGLS compared to OLS.

Brewer, Crossley, and Joyce (2013) considers a DiD model with individual-level U.S. panel data with *N* = 750,127, *T* = 30, and a placebo state-level law, so clustering is on state with *G* = 50. It finds that bias-corrected FGLS for AR(2) errors, using the Hansen (2007b) correction, leads to higher power than FE estimation. In its example, ignoring the bias correction does not change results much, perhaps because *T* = 30 is reasonably large.

For balanced clusters with Ω* _{g}* the same for all

*g*, say Ω

*= Ω*

_{g}_{*}, and for

*N*small, then the FGLS estimator in Equation 13 can be used without need to specify a model for Ω

_{g}_{*}. Instead we can let have

*ij*entry , where are the residuals from initial OLS estimation. These assumptions may be reasonable for a balanced panel. Two complications can arise. First, even without fixed effects there may be many off-diagonal elements to estimate and this number can be large relative to the number of observations. Second, the fixed effects lead to bias in estimating the off-diagonal covariances. Hausman and Kuersteiner (2008) presents fixes for both complications.

^{th}### D. Testing the Need for Fixed Effects

FE estimation is often accompanied by a loss of precision in estimation, as only within-cluster variation is used (recall we regress on ). Furthermore, the coefficient of a cluster-invariant regressor is not identified because then . Thus, it is standard to test whether it is sufficient to estimate by OLS or FGLS, without cluster-specific fixed effects.

The usual test is a Hausman test based on the difference between the FE estimator, , and the RE estimator, . Let **β**_{1} denote a subcomponent of **β**, possibly just the coefficient of a single regressor of key interest; at most, **β**_{1} contains the coefficients of all regressors that are not invariant within cluster or, in the case of panel data, that are not aggregate time effects that take the same value for each individual. The chi-squared distributed test statistic is

where is a consistent estimate of .

Many studies use the standard form of the Hausman test. This obtains under the strong assumption that is fully efficient under the null hypothesis. This requires the unreasonably strong assumptions that α_{g} and ε_{ig} in Equation 16 are iid, requiring that neither α_{g} and ε_{ig} is heteroskedastic and that ε_{ig} has no within-cluster correlation. As already noted, these assumptions are likely to fail and one should not use default standard errors. Instead, a CRVE should be used. For similar reasons, the standard form of the Hausman test should not be used.

Wooldridge (2010, p. 332) instead proposes implementing a cluster-robust version of the Hausman test by the following OLS regression

where * w_{g}* denotes the subcomponent of

*that varies within cluster and . If*

**x**_{ig}*H*

_{0}: =

**γ = 0**is rejected using a Wald test based on a cluster-robust estimate of the variance matrix, then the fixed effects model is necessary. The Stata user-written command

*xtoverid*, due to Schaffer and Stillman (2010), implements this test.

An alternative is to use the pairs cluster bootstrap to obtain , in each resample obtaining and , leading to *B* resample estimates of . We are unaware of studies comparing these two cluster-robust versions of the Hausman test.

## IV. What to Cluster Over?

It is not always clear what to cluster over—that is, how to define the clusters—and there may even be more than one way to cluster.

Before providing some guidance, we note that it is possible for cluster-robust errors to actually be smaller than default standard errors. First, in some rare cases errors may be negatively correlated, most likely when *N _{g}* = 2, in which case Equation 6 predicts a reduction in the standard error. Second, cluster-robust is also heteroskedastic-robust and White heteroskedastic-robust standard errors in practice are sometimes larger and sometimes smaller than the default. Third, if clustering has a modest effect so cluster-robust and default standard errors are similar in expectation, then cluster-robust may be smaller due to noise. In cases where the cluster-robust standard errors are smaller, they are usually not much smaller than the default, whereas in other applications they can be much, much larger.

### A. Factors Determining What to Cluster Over

There are two guiding principles that determine what to cluster over.

First, given defined in Equations 7 and 9, whenever there is reason to believe that both the regressors and the errors might be correlated within cluster, we should think about clustering defined in a broad enough way to account for that clustering. Going the other way, if we think that either the regressors or the errors are likely to be uncorrelated within a potential group, then there is no need to cluster within that group.

Second, is an average of *G* terms that gets closer to only as *G* gets large. If we define very large clusters, so that there are very few clusters to average over in Equation 11, then the resulting can be a very poor estimate of . This complication, and discussion of how few is “few,” is the subject of Section VI.

These two principles mirror the bias-variance tradeoff that is common in many estimation problems—larger and fewer clusters have less bias but more variability. There is no general solution to this tradeoff, and there is no formal test of the level at which to cluster. The consensus is to be conservative and avoid bias and to use bigger and more aggregate clusters when possible, up to and including the point at which there is concern about having too few clusters.

For example, suppose your data set included individuals within counties within states, and you were considering whether to cluster at the county level or the state level. We have been inclined to recommend clustering at the state level. If there was within-state cross-county correlation of the regressors and errors, then ignoring this correlation (for example, by clustering at the county level) would lead to incorrect inference. In practice, researchers often cluster at progressively higher (that is, broader) levels and stop clustering when there is relatively little change in the standard errors. This seems to be a reasonable approach.

There are settings where one may not need to use cluster-robust standard errors. We outline several though note that in all these cases it is always possible to still obtain cluster-robust standard errors and contrast them to default standard errors. If there is an appreciable difference, then use cluster-robust standard errors.

If a key regressor is randomly assigned within clusters, or is as good as randomly assigned, then the within-cluster correlation of the regressor is likely to be zero. Thus, there is no need to cluster standard errors even if the model’s errors are clustered. In this setting, if there are additionally control variables of interest, and if these are not randomly assigned within cluster, then we may wish to cluster our standard errors for the sake of correct inference on the control variables.

If the model includes cluster-specific fixed effects, and we believe that within-cluster correlation of errors is solely driven by a common shock process, then we may not be worried about clustering. The fixed effects will absorb away the common shock, and the remaining errors will have zero within-cluster correlation. More generally, control variables may absorb systematic within-cluster correlation. For example, in a state-year panel setting, control variables may capture the state-specific business cycle.

However, as already noted in Section IIIA, the within-cluster correlation is usually not fully eliminated. And even if it is eliminated, the errors may still be heteroskedastic. Stock and Watson (2008) shows that, surprisingly, applying the usual White (1980) heteroskedastic-consistent variance matrix estimate to the FE estimator leads to inconsistent estimation of if *N _{g}* is small (though it is correct if

*N*= 2). It derives a bias-corrected formula for heteroskedastic-robust standard errors. Alternatively, and more simply, the CRVE is consistent for even if the errors are only heteroskedastic, though this estimator of is more variable.

_{g}Finally, as already noted in Section IID, we can always build a parametric model of the correlation structure of the errors and estimate by FGLS. If we believe that this parametric model captures the salient features of the error correlations, then default FGLS standard errors can be used.

### B. Clustering Due to Survey Design

Clustering routinely arises due to the sampling methods used in complex surveys. Rather than randomly draw individuals from the entire population, costs are reduced by sampling only a randomly selected subset of primary sampling units (such as a geographic area), followed by random selection, or stratified selection, of people within the chosen primary sampling units.

The survey methods literature uses methods to control for clustering that predate the cluster-robust approach of this paper. The loss of estimator precision due to clustered sampling is called the design effect: “The design effect or Deff is the ratio of the actual variance of a sample to the variance of a simple random sample of the same number of elements” (Kish 1965, p. 258). Kish and Frankel (1974) gives the variance inflation formula Equation 6 in the nonregression case of estimation of the mean. Pfeffermann and Nathan (1981) considers the more general regression case. The CRVE is called the linearization formula, and the common use of *G* – 1 as the degrees of freedom used in hypothesis testing comes from the survey methods literature; see Shah, Holt, and Folsom (1977), which predates the econometrics literature.

Applied economists routinely use data from complex surveys, controlling for clustering by using a cluster-robust variance matrix estimate. At the minimum, one should cluster at the level of the primary sampling unit though often there is reason to cluster at a broader level, such as clustering on state if regressors and errors are correlated within state.

The survey methods literature additionally controls for two other features of survey data—weighting and stratification. These methods are well-established and are incorporated in specialized software as well as in some broad-based packages such as Stata. Bhattacharya (2005) provides a comprehensive treatment in a GMM framework.

If sample weights are provided, then it is common to perform weighted least squares. Then the CRVE for is that given in Equation 15 with replaced by * W_{g}*. The need to weight can be ignored if stratification is on only the exogenous regressors and we assume correct specification of the model so that in our sample E[

*|*

**y***] =*

**X***. In that special case, both weighted and unweighted estimators are consistent, and weighted OLS actually may be less efficient if, for example, model errors are iid; see, for example, Solon, Haider, and Wooldridge (2013). Another situation in which to use weighted least squares, unrelated to complex surveys, is when data for the*

**X***ig*observation is obtained by in turn averaging

^{th}*N*observations and

_{ig}*N*varies.

_{ig}Stratification of the sample can enable more precise statistical inference. These gains can be beneficial in the nonregression case, such as estimating the monthly national unemployment rate. The gains can become much smaller once regressors are included because these can partially control for stratification; see, for example, the application in Bhattacharya (2005). Econometrics applications therefore usually do not adjust standard errors for stratification, leading to conservative inference due to some relatively small overestimation of the standard errors.

## V. Multiway Clustering

The discussion to date has presumed that if there is more than one potential way to cluster, these ways are nested in each other, such as households within states. But when clusters are nonnested, traditional cluster-robust inference can only deal with one of the dimensions.

In some applications it is possible to include sufficient regressors to eliminate concern about error correlation in all but one dimension, and then do cluster-robust inference for that remaining dimension. A leading example is that in a state-year panel there may be clustering both within years and within states. If the within-year clustering is due to shocks that are the same across all observations in a given year, then including year fixed effects as regressors will absorb within-year clustering, and inference then need only control for clustering on state.

When this approach is not applicable, the one-way cluster robust variance can be extended to multiway clustering. Before discussing this topic, we highlight one error that we find some practitioners make, which is to cluster at the intersection of the two groupings. In the preceding example, some might be tempted to cluster at the state-year level. A Stata example is to use the command *regress y x, vce(cluster id_styr)* where *id_styr* is a state-year identifier. This will be very inadequate because it imposes the restriction that observations are independent if they are in the same state but in different years. Indeed, if the data is aggregated at the state-year level, there is only one observation at the state-year level, so this is identical to using heteroskedastic-robust standard errors—that is, not clustering at all. This point was highlighted by Bertrand, Duflo, and Mullainathan (2004), which advocated clustering on the state.

### A. Multiway Cluster-Robust Variance Matrix Estimate

The cluster-robust estimate of defined in Equations 10 and 11 can be generalized to clustering in multiple dimensions. In a change of notation, suppress the subscript for cluster and more simply denote the model for an individual observation as

(19)

Regular one-way clustering is based on the assumption that E[*u _{i}*

*u*|

_{j}*,*

**x**_{i}

**x**_{j}] = 0, unless observations

*i*and

*j*are in the same cluster. Then Equation 11 sets [

*i*,

*j*in same cluster], where . In multiway clustering, the key assumption is that E[

*u*

_{i}*u*|

_{j}*,*

**x**_{i}*] = 0, unless observations*

**x**_{j}*i*and

*j*share any cluster dimension. Then the multiway cluster robust estimate of replaces Equation 11 with

(20)

This method relies on asymptotics that are in the number of clusters of the dimension with the fewest number of clusters. This method is thus most appropriate when each dimension has many clusters.

Theory for two-way cluster robust estimates of the variance matrix is presented in Cameron, Gelbach, and Miller (2006, 2011), Miglioretti and Heagerty (2006), and Thompson (2006, 2011). See also empirical panel data applications by Acemoglu and Pischke (2003), which clustered at individual level and at region×time level, and by Petersen (2009), which clustered at firm level and at year level. Cameron, Gelbach, and Miller (2011) presents an extension to multiway clustering. Like one-way cluster-robust, the method can be applied to estimators other than OLS.

For two-way clustering, this robust variance estimator is easy to implement given software that computes the usual one-way cluster-robust estimate. First, obtain three different cluster-robust “variance” matrices for the estimator by one-way clustering in, respectively, the first dimension, the second dimension, and by the intersection of the first and second dimensions. Then add the first two variance matrices and, to account for double-counting, subtract the third. Thus

(21)

where the three component variance estimates are computed using Equations 10 and 11 for the three different ways of clustering.

We spell this out in a step-by-step fashion.

(1) Identify your two ways of clustering. Make sure you have a variable that identifies each way of clustering. Also create a variable that identifies unique “group 1 by group 2” combinations. For example, suppose you have individual-level data spanning many U.S. states and many years, and you want to cluster on state and on year. You will need a variable for state (for example, California), a variable for year (for example, 1990), and a variable for state-by-year (California

*and*1990).(2) Estimate your model, clustering on “group 1.” For example, regress

*y*on, clustering on state. Save the variance matrix as .**x**(3) Estimate your model, clustering on “group 2.” For example, regress

*y*on, clustering on year. Save the variance matrix as .**x**(4) Estimate your model, clustering on “group 1 by group 2.” For example, regress

*y*on, clustering on state-by-year. Save the variance matrix as .**x**(5) Create a new variance matrix . This is your new two-way cluster robust variance matrix for .

(6) Standard errors are the square root of the diagonal elements of this matrix.

If you are interested in only one coefficient, you can also just focus on saving the standard error for this coefficient in Steps 2–4 above, and then create .

In taking these steps, you should watch out for some potential pitfalls. With perfectly multicollinear regressors such as inclusion of dummy variables—some of which are redundant—a statistical package may automatically drop one or more variables to ensure a nonsingular set of regressors. If the package happens to drop different sets of variables in Steps 2, 3, and 4, then the resulting s will not be comparable, and adding them together in Step 5 will give a nonsense result. To prevent this issue, manually inspect the estimation results in Steps 2, 3, and 4 to ensure that each step has the same set of regressors, the same number of observations, etc. The only things that should be different are the reported standard errors and the reported number of clusters.

### B. Implementation

Unlike the standard one-way cluster case, is not guaranteed to be positive semidefinite, so it is possible that it may compute negative variances. In some applications with fixed effects, may be non positive-definite, but the subcomponent of associated with the regressors of interest may be positive-definite. This may lead to an error message, even though inference is appropriate for the parameters of interest. Our informal observation is that this issue is most likely to arise when clustering is done over the same groups as the fixed effects. Few clusters in one or more dimensions can also lead to being a nonpositive-semidefinite matrix. Cameron, Gelbach, and Miller (2011) presents an eigendecomposition technique used in the time series HAC literature that zeroes out negative eigenvalues in to produce a positive semidefinite variance matrix estimate.

The Stata user-written command *cmgreg*, available at the authors’ websites, implements multiway clustering for the OLS estimator with, if needed, the negative eigen-value adjustment. The Stata add-on command *ivreg2*, due to Baum, Schaffer, and Still-man (2007), implements two-way clustering for OLS, IV and linear GMM estimation. Other researchers have also posted code, available from searching the Web.

Cameron, Gelbach, and Miller (2011) applies the two-way method to data from Hersch (1998) that examines the relationship between individual wages and injury risk measured separately at the industry level and the occupation level. The log-wage for 5,960 individuals is regressed on these two injury risk measures, with standard errors obtained by two-way clustering on 211 industries and 387 occupations. In that case, two-way clustering leads to only a modest change in the standard error of the industry job risk coefficient compared to the standard error with one-way clustering on industry. Because industry job risk is perfectly correlated within industry, by Result 6 we need to cluster on industry if there is any within-industry error correlation. By similar logic, the additional need to cluster on occupation depends on the within-occupation correlation of job industry risk, and this correlation need not be high. For the occupation job risk coefficient, the two-way and one-way cluster (on occupation) standard errors are similar. Despite the modest difference in this example, two-way clustering avoids the need to report standard errors for one coefficient clustering in one way and for the second coefficient clustering in the second way.

Cameron, Gelbach, and Miller (2011) also applies the two-way cluster-robust method to data on volume of trade between 98 countries with 3,262 unique country pairs. In that case, two-way clustering on each of the countries in the country pair leads to standard errors that are 36 percent larger than one-way clustering and 230 percent more than heteroskedastic-robust standard errors. Cameron and Miller (2012) studies such dyadic data in further detail. It notes that two-way clustering does not pick up all the potential correlations in the data. Instead, more general cluster-robust methods, including one proposed by Fafchamps and Gubert (2007), should be used.

### C. Feasible GLS

Similar to one-way clustering, FGLS is more efficient than OLS, provided an appropriate model for Ω = E[* uu′* |

*] is specified and is consistently estimated.*

**X**The random effects model can be extended to multiway clustering. For individual *i* in clusters *g* and *h*, the two-way random effects model specifies

where the errors α* _{g}*, δ

*, and*

_{h}*ε*

*are each assumed to be iid distributed with mean 0. For example, Moulton (1986) considered clustering due to grouping of regressors (schooling, age, and weeks worked) in a log earnings regression, and estimated a model with common random shock for each year of schooling, for each year of age, and for each number of weeks worked.*

_{igh}The two-way random effects model can be estimated using standard software for (nested) hierarchical linear models. See, for example, Cameron and Trivedi (2009, Chapter 9.5.7) for Stata command *xtmixed* (command *mixed* from Version 13 on). For estimation of a many-way random effects model, see Davis (2002), which modeled film attendance data clustered by film, theater, and time.

The default standard errors after FGLS estimation require that Ω is correctly specified. For two-way and multiway random effects models, FGLS estimation entails transforming the data in such a way that there is no obvious method for computing a variance matrix estimate that is robust to misspecification of Ω. Instead, if there is concern about misspecification of Ω, then one needs to consider FGLS with richer models for Ω and assume that these are correctly specified—see Rabe-Hesketh and Skrondal (2012) for richer hierarchical models in Stata—or do less efficient OLS with two-way cluster-robust standard errors.

### D. Spatial Correlation

Cluster-robust variance matrix estimates are closely related to spatial-robust variance matrix estimates.

In general, for Model 19, in Equation 20 has the form

(22)

where *w*(*i*, *j*) are weights. For cluster-robust inference, these weights are either one (cluster in common) or zero (no cluster in common). But the weights can also decay from one to zero, as in the case of the HAC variance matrix estimate for time series where *w*(*i*, *j*) decays to zero as |*i* – *j*| increases.

For spatial data, it is assumed that model errors become less correlated as the spatial distance between observations grows. For example, with state-level data the assumption that model errors are uncorrelated across states may be relaxed to allow correlation that decays to zero as the distance between states gets large. Conley (1999) provides conditions under which Equations 10 and 22 provide a robust variance matrix estimate for the OLS estimator, where the weights *w*(*i*, *j*) decay with the spatial distance. The estimate (which Conley also generalizes to GMM models) is often called a spatial-HAC estimate rather than spatial-robust, as proofs use mixing conditions (to ensure decay of dependence) as observations grow apart in distance. These conditions are not applicable to clustering due to common shocks, which leads to the cluster-robust estimator with independence of observations across clusters.

Driscoll and Kraay (1998) considers panel data with *T* time periods and *N* individuals, with errors potentially correlated across individuals (and no spatial dampening), though this correlation across individuals disappears for observations that are more than *m* time periods apart. Let *it* denote the typical observation. The Driscoll-Kraay spatial correlation consistent (SCC) variance matrix estimate can be shown to use weight *w*(*it*, *js*) = 1 – *d*(*it*, *js*) / (*m* + 1) in Equation 22, where the summation is now over *i*, *j*, *s*, and *t*, and *d*(*it*, *js*) = | *t* – *s*| if |*t* – *s*| ≥ *m* and *d*(*it*, *js*) = 0 otherwise. This method requires that the number of time periods *T* → ∞, so is not suitable for short panels while *N* may be fixed or *N* → ∞. The Stata add-on command *xtscc*, due to Hoechle (2007), implements this variance estimator.

An estimator proposed by Thompson (2006) allows for across-cluster (in its example firm) correlation for observations close in time in addition to within-cluster correlation at any time separation. The Thompson estimator can be thought of as using *w*(*it*,*js*) = **1**[*i*, *j* share a firm, or *d*(*it*, *js*) ≤ *m*]. Foote (2007) contrasts the two-way cluster-robust and these other variance matrix estimators in the context of a macroeconomics example. Petersen (2009) contrasts various methods for panel data on financial firms where there is concern about both within firm correlation (over time) and across firm correlation due to common shocks.

Barrios et al. (2012) considers state-year panel data on individuals in states over years with state-level treatment and outcome (earnings) that is correlated spatially across states. This spatial correlation can be ignored if the state-level treatment is randomly assigned. But if the treatment is correlated over states (for example, adjacent states may be more likely to have similar minimum wage laws) then one can no longer use standard errors clustered at the state level. Instead, one should additionally allow for spatial correlation of errors across states. The authors additionally contrast traditional model-based inference with randomization inference.

In practice, data can have cluster, spatial, and time series aspects, leading to hybrids of cluster-robust, spatial-HAC, and time-series HAC estimators. Furthermore, it may be possible to parameterize some of the error correlation. For example, for a time series AR(1) error it may be preferable to use based on an AR(1) model rather than . To date, empirical practice has not commonly modeled these combined types of error correlations. This may become more common in the future.

## VI. Few Clusters

We return to one-way clustering and focus on the Wald “*t*-statistic”

(23)

where β is one element in the parameter vector **β**, and the standard error is the square root of the appropriate diagonal entry in . If *G* → ∞ then *w* ~*N*[0,1] under *H*_{0} : β = β_{0}. For finite *G*, the distribution of *w* is unknown even with normal errors. It is common to use the *T* distribution with *G* – 1 degrees of freedom.

It is not unusual for the number of clusters *G* to be quite small. Despite few clusters, may still be a reasonably precise estimate of β if there are many observations per cluster. But with small *G*, the asymptotics have not kicked in. Then can be downward-biased.

One should at a minimum use *T*(*G* – 1) critical values and defined in Equations 10 and 11 with residuals scaled by where *c* is defined in Equation 12 or *c* = *G* / (*G* – 1). Most packages rescale the residuals—Stata uses the first choice of *c* and SAS the second. The use of *T*(*G* – 1) critical values is less common. Stata uses the *T*(*G* – 1) distribution after command *regress y x, vce(cluster)*. But the alternative command *xtreg y x, vce(robust)* instead uses standard normal critical values.

Even with both of these adjustments, Wald tests generally overreject. The extent of overrejection depends on both how few clusters there are and the particular data and model used. In some cases the overrejection is mild, in others cases a test with nominal size 0.05 may have true test size of 0.10.

The next subsection outlines the basic problem and discusses how few is “few” clusters. The subsequent three subsections present three approaches to finite-cluster correction—bias-corrected variance, bootstrap with asymptotic refinement, and use of the *T* distribution with adjusted degrees-of-freedom. The final subsection considers special cases.

### A. The Basic Problems with Few Clusters

There are two main problems with few clusters. First, OLS leads to “overfitting,” with estimated residuals systematically too close to zero compared to the true error terms. This leads to a downward-biased cluster-robust variance matrix estimate. The second problem is that even with bias-correction the use of fitted residuals to form the estimate of * b* leads to overrejection (and confidence intervals that are too narrow) if the critical values are from the standard normal or even the

*T*(

*G*– 1) distribution.

For the linear regression model with independent homoskedastic normally distributed errors, similar problems are easily controlled. An unbiased variance matrix is obtained by estimating the error variance σ^{2} by rather than . This is the “fix” in the OLS setting for the first problem. The analogue to the second problem is that the *N*[0,1] distribution is a poor approximation to the true distribution of the Wald statistic. In the iid case, the Wald statistic *w* can be shown to be exactly *T*(*N* – *K*) distributed. For nonnormal homoskedastic errors, the *T*(*N* – *K*) is still felt to provide a good approximation provided *N* is not too small. Both of these problems arise in the clustered setting albeit with more complicated manifestations and fixes.

For independent heteroskedastic normally distributed errors there are no exact results. MacKinnon and White (1985) considers several adjustments to the heteroskedastic-consistent variance estimate of White (1980) including one called HC2 that is unbiased in the special case that errors are homoskedastic. Unfortunately, if errors are actually heteroskedastic, as expected, the HC2 estimate is no longer unbiased for —an unbiased estimator depends on the unknown pattern of heteroskedasticity and on the design matrix * X*. And there is no way to obtain an exact

*T*distribution result for

*w*even if errors are normally distributed. Other proposed solutions for testing and forming confidence intervals include using a

*T*distribution with data-determined degrees of freedom and using a bootstrap with asymptotic refinement.

In the following subsections, we consider extensions of these various adjustments to the clustered case where the problems can become even more pronounced.

Before proceeding, we note that there is no specific point at which we need to worry about few clusters. Instead, “more is better.” Current consensus appears to be that *G* = 50 is enough for state-year panel data. In particular, Bertrand, Duflo, and Mullainathan (2004, Table 8) finds in its simulations that for a policy dummy variable with high within-cluster correlation, a Wald test based on the standard CRVE with critical value of 1.96 had rejection rates of 0.063, 0.058, 0.080, and 0.115 for number of states (*G*) equal to, respectively, 50, 20, 10, and 6. The simulations of Cameron, Gelbach, and Miller (2008, Table 3), based on a quite different data generating process but again with standard CRVE and critical value of 1.96, had rejection rates of 0.068, 0.081, 0.118, and 0.208 for *G* equal to, respectively, 30, 20, 10, and 5. In both cases, the rejection rates would also exceed 0.05 if the critical value was from the *T*(*G* – 1) distribution.

The preceding results are for balanced clusters. Cameron, Gelbach, and Miller (2008, Table 4, Column 8) considers unbalanced clusters when *G* = 10. The rejection rate with unbalanced clusters, half of size *N _{g}* = 10 and half of size 50, is 0.183, appreciably worse than rejection rates of 0.126 and 0.115 for balanced clusters of sizes, respectively, 10 and 100. Recent papers by Carter, Schnepel, and Steigerwald (2013) and Imbens and Kolesar (2012) provide theory that also indicates that the effective number of clusters is reduced when

*N*varies across clusters; see also the simulations in MacKinnon and Webb (2013). Similar issues may also arise if the clusters are balanced, but estimation is by weighted LS that places different weights on different clusters. Cheng and Hoekstra (2013) documents that weighting can result in overrejection in the panel DiD setting of Bertrand, Duflo, and Mullainathan (2004).

_{g}To repeat a key message, there is no clear-cut definition of “few.” Depending on the situation, “few” may range from less than 20 clusters to less than 50 clusters in the balanced case and even more clusters in the unbalanced case.

### B. Solution 1: Bias-Corrected Cluster-Robust Variance Matrix

A weakness of the standard CRVE with residual is that it is biased for , since . The bias depends on the form of Ω* _{g}* but will usually be downward. Several corrected residuals to use in place of in Equation 11 have been proposed. The simplest, already mentioned, is to use or where

*c*is defined in Equation 12. One should at least use either of these corrections.

Bell and McCaffrey (2002) uses

(24)

where *H** _{gg}* =

*(*

**X**_{g}*)*

**X′X**^{-1}

**X′***g*. This transformed residual leads to unbiased CRVE in the special case that E[

**u**_{g}*] = σ*

**u′**_{g}^{2}

*. This is a cluster generalization of the HC2 variance estimate of MacKinnon and White (1985), so we refer to it as the CR2VE.*

**I**Bell and McCaffrey (2002) also uses

(25)

This transformed residual leads to CRVE that can be shown to equal the delete-one-cluster jackknife estimate of the variance of the OLS estimator. This jackknife correction leads to upward-biased CRVE if in fact E[**u**_{g}* u′_{g}*] = σ

^{2}

*. This is a cluster generalization of the HC3 variance estimate of MacKinnon and White (1985), so we refer to it as the CR3VE.*

**I**Angrist and Pischke (2009, Chapter 8) and Cameron, Gelbach, and Miller (2008) provide a more extensive discussion and cite more of the relevant literature. This literature includes papers that propose corrections for the more general case that E[**u**_{g}* u′_{g}*] ≠ σ

^{2}

*but has a known parameterization, such as an RE model, and extension to generalized linear models.*

**I**Angrist and Lavy (2002) applies the CR2VE Correction 24 in an application with *G* = 30 to 40 and finds that the correction increases cluster-robust standard errors by between 10 and 50 percent. Cameron, Gelbach, and Miller (2008, Table 3) finds that the CR3VE Correction 24 has rejection rates of 0.062, 0.070, 0.092, and 0.138 for *G* equal to, respectively, 30, 20, 10, and 5. These rejection rates are a considerable improvement on 0.068, 0.081, 0.118, and 0.208 for the standard CRVE, but there is still considerable overrejection for very small *G*.

The literature has gravitated to using the CR2VE adjustment rather than the CR3VE adjustment. This reduces but does not eliminate overrejection when there are few clusters.

### C. Solution 2: Cluster Bootstrap with Asymptotic Refinement

In Section IIF, we introduced the bootstrap as it is usually used, to calculate standard errors that rely on regular asymptotic theory. Here, we consider a different use of the bootstrap, one with asymptotic refinement that may lead to improved finite-sample inference.

We consider inference based on *G* → ∞ (formally, has a limit normal distribution). Then a two-sided Wald test of nominal size 0.05, for example, can be shown to have true size 0.05 + *O*(*G*^{–1}) when the usual asymptotic normal approximation is used. For *G* → ∞ this equals the desired 0.05 but for finite *G* this differs from 0.05. If an appropriate bootstrap with asymptotic refinement is instead used, the true size is 0.05 + *O*(*G*^{–3/2}). This is closer to the desired 0.05 for large *G*, as *G*^{–3/2} < *G*^{–1}. Hopefully, this is also the case for small *G*, something that is established using appropriate Monte Carlo experiments. For a one-sided test or a nonsymmetric two-sided test the rates are instead, respectively, 0.05 + *O*(*G*^{–1/2}) and 0.05 + *O*(*G*^{–1}).

Asymptotic refinement can be achieved by bootstrapping a statistic that is asymptotically pivotal, meaning the asymptotic distribution does not depend on any unknown parameters. The estimator is not asymptotically pivotal as its distribution depends on , which in turn depends on unknown variance parameters in V[* u*|

*]. The Wald*

**X***t*-statistic defined in Equation 23 is asymptotically pivotal as its asymptotic distribution is

*N*[0,1], which does not depend on unknown parameters.

#### 1. Percentile-t bootstrap

A percentile-*t* bootstrap obtains *B* draws, , from the distribution of the Wald *t*-statistic as follows. *B* times do the following:

(1) Obtain

*G*clusters by one of the cluster bootstrap methods detailed below.(2) Compute the OLS estimate in this resample.

(3) Calculate the Wald test statistic where is the cluster-robust standard error of and is the OLS estimate of β from the original sample.

Note that we center on and not β_{0}, as the bootstrap views the sample as the population, that is, , and the *B* resamples are based on this “population.” Note also that the standard error in Step 3 needs to be cluster-robust. A good choice of *B* is *B* = 999; this is more than *B* for standard error estimation as tests are in the tails of the distribution, and is such that (*B* + 1)α is an integer for common choices of test size α.

Let denote the ordered values of . These ordered values trace out the density of the Wald *t*-statistic, taking the place of a standard normal or *T* distribution. For example, the critical values for a 95 percent nonsymmetric confidence interval or a 5 percent nonsymmetric Wald test are the lower 2.5 percentile and upper 97.5 percentile of rather than, for example, the standard normal values of –1.96 and 1.96. The *p*-value for a symmetric test based on the original sample Wald statistic *w* equals the proportion of times that .

The simplest cluster resampling method is the pairs cluster resampling introduced in Section IIF. Then in Step 1 above we form *G* clusters by resampling with replacement *G* times from the original sample of clusters. This method has the advantage of being applicable to a wide range of estimators, not just OLS. However, for some types of data the pairs cluster bootstrap may not be applicable—see “Bootstrap with Caution” below.

Cameron, Gelbach, and Miller (2008) found that in Monte Carlos with few clusters the pairs cluster bootstrap did not eliminate test overrejection. The authors proposed using an alternative percentile-*t* bootstrap, the wild cluster bootstrap, that holds the regressors fixed across bootstrap replications.

#### 2. Wild cluster bootstrap

The wild cluster bootstrap resampling method is as follows. First, estimate the main model, imposing (forcing) the null hypothesis *H*_{0} that you wish to test, to give estimate . For example, for test of statistical significance of a single variable, regress *y _{ig}* on all components of

*except the variable that has regressor with coefficient zero under the null hypothesis. Form the residual . Then obtain the*

**x**_{ig}*b*resample for Step 1 above as follows:

^{th}(1a) Randomly assign cluster

*g*the weight*d*= –1 with probability 0.5 and the weight_{g}*d*= 1 with probability 0.5. All observations in cluster_{g}*g*get the same value of the weight.(1b) Generate new pseudo-residuals , and hence new outcome variables .

Then proceed with Step 2 as before, regressing on * x_{ig}*, and calculate as in Step 3. The

*p*-value for the test based on the original sample Wald statistic

*w*equals the proportion of times that .

For the wild bootstrap, the values cannot be used directly to form critical values for a confidence interval. Instead, they can be used to provide a *p*-value for testing a hypothesis. To form a confidence interval, one needs to invert a sequence of tests, profiling over a range of candidate null hypotheses *H*_{0} : β = β_{0}. For each of these null hypotheses, obtain the *p*-value. The 95 percent confidence interval is the set of values of β_{0} for which *p* ≥ 0.05. This method is computationally intensive but conceptually straightforward. As a practical matter, you may want to ensure that you have the same set of bootstrap draws across candidate hypotheses so as to not introduce additional bootstrapping noise into the determination of where the cutoff is.

In principle, it is possible to directly use a bootstrap for bias-reduction, such as to remove bias in standard errors. In practice this is not done, however, as in practice any bias reduction comes at the expense of considerably greater variability. A conservative estimate of the standard error equals the width of a 95 percent confidence interval, obtained using asymptotic refinement, divided by 2 × 1.96.

Note that for the wild cluster bootstrap the resamples have the same * X* in each resample, whereas for pairs cluster both

** and*

**y**** vary across the*

**X***B*resamples. The wild cluster bootstrap is an extension of the wild bootstrap proposed for heteroskedastic data. Essentially, it works because the two-point distribution for forming ensures that and . There are other two-point distributions that also do so, but Davidson and Flachaire (2008) shows that in the heteroskedastic case it is best to use the weights

*d*= {–1,1}, called Rademacher weights.

_{g}The wild cluster bootstrap essentially replaces * y_{g}* in each resample with one of two values, or . Because this is done across

*G*clusters, there are at most 2

*possible combinations of the data, so there are at most 2*

^{G}*unique values of . If there are very few clusters then there is no need to actually bootstrap as we can simply enumerate, with separate estimation for each of the 2*

^{G}*possible data sets.*

^{G}Webb (2013) expands on these limitations. It shows that there are actually only 2^{G–1} possible *t*-statistics in absolute value. Thus with *G* = 5 there are at most 2^{4} = 16 possible values of . So if the main test statistic is more extreme than any of these 16 values, for example, then all we know is that the *p*-value is smaller than 1 / 16 = 0.0625. Full enumeration makes this discreteness clear. Bootstrapping without consideration of this issue can lead to inadvertently picking just one point from the interval of equally plausible *p*-values. As *G* gets to be as large as 11, this concern is less of an issue since 2^{10} = 1024.

Webb (2013) proposes greatly reducing the discreteness of *p*-values with very low *G* by instead using a six-point distribution for the weights *d _{g}* in Step 1b. In this proposed distribution, the weights

*d*have a 1 / 6 chance of taking each value in . In Webb’s simulations, this method outperforms the two-point wild bootstrap for

_{g}*G*< 10 and is the preferred method to use if

*G*< 10.

MacKinnon and Webb (2013) addresses the issue of unbalanced clusters and finds that, even with *G* = 50, tests based on the standard CRVE with *T*(*G* – 1) critical values can overreject considerably if the clusters are unbalanced. By contrast, the two-point wild bootstrap with Rademacher weights is generally reliable.

#### 3. Bootstrap with caution

Regardless of the bootstrap method used, pairs cluster with or without asymptotic refinement or wild cluster bootstrap, an important step when bootstrapping with few clusters is to examine the distribution of bootstrapped values. This is something that should be done whether you are bootstrapping β to obtain a standard error, or bootstrapping *t*-statistics with refinement to obtain a more accurate *p*-value. This examination can take the form of looking at four things: (1) basic summary statistics like mean and variance; (2) the sample size to confirm that it is the same as the number of bootstrap replications (no missing values); (3) the largest and smallest handful of values of the distribution; and (4) a histogram of the bootstrapped values.

We detail a few potential problems that this examination can diagnose.

First, if you are using a pairs cluster bootstrap and one cluster is an outlier in some sense, then the resulting histogram may have a big “mass” that sits separately from the rest of the bootstrap distribution—that is, there may be two distinct distributions, one for cases where that cluster is sampled and one for cases where it is not. If this is the case, then you know that your results are sensitive to the inclusion of that cluster.

Second, if you are using a pairs cluster bootstrap with dummy righthand side variables, then in some samples you can get no or very limited variation in treatment. This can lead to zero or near-zero standard errors. For a percentile-*t* pairs cluster bootstrap, a zero or missing standard error will lead to missing values for *w**, since the standard error is zero or missing. If you naively use the remaining distribution, then there is no reason to expect that you will have valid inference. And if the bootstrapped standard errors are zero plus machine precision noise, rather than exactly zero, very large *t*-values may result. Then your bootstrap distribution of *t*-statistics will have really fat tails and you will not reject anything, even false null hypotheses. No variation or very limited variation in treatment can also result in many of your ^{*}s being “perfect fit” ^{*}s with limited variability. Then the bootstrap standard deviation of the ^{*}s will be too small, and if you use it as your estimated standard error you will overreject. In this case, we suggest using the wild cluster bootstrap.

Third, if your pairs cluster bootstrap samples draw nearly multicollinear samples, you can get huge ^{*}s. This can make a bootstrapped standard error seem very large. You need to determine what in the bootstrap samples “causes” the huge ^{*}s. If this is some pathological but common draw, then you may need to think about a different type of bootstrap, such as the wild cluster bootstrap, or give up on bootstrapping methods. For an extreme example, consider a DiD model with first-order “control” fixed effects and an interaction term. Suppose that a bootstrap sample happens to have among its “treatment group” only observations where “post = 1.” Then the variables “treated” and “treated*post” are collinear, and an OLS package will drop one of these variables. If it drops the “post” variable, it will report a coefficient on “treated*post”; this coefficient will not be a proper interaction term, but instead also will include the level effect for the treated group.

Fourth, with less than ten clusters the wild cluster bootstrap should use the six-point version of Webb (2013).

Fifth, in general, if you see missing values in your bootstrapped *t*-statistics, then you need to figure out why. Don’t take your bootstrap results at face value until you know what’s going on.

### D. Solution 3: Improved Critical Values Using a T-Distribution

The simplest small-sample correction for the Wald statistic is to base inference on a *T* distribution rather than the standard normal, with degrees of freedom at most the number of clusters *G*. Recent research has proposed methods that lead to using degrees of freedom much less than *G*, especially if clusters are unbalanced.

#### 1. G-L degrees of freedom

Some packages, including Stata after command *regress*, use *G* – 1 degrees of freedom for *t*-tests and *F*–tests based on cluster-robust standard errors. This choice emanates from the complex survey literature; Bell and McCaffrey (2002) notes, however, that with normal errors this choice still tends to result in test overrejection so the degrees of freedom should be even less than this.

Even the *T*(*G* – 1) can make quite a difference. For example, with *G* = 10 for a two-sided test at level 0.05, the critical value for *T*(9) is 2.262 rather than 1.960, and if *w* = 1.960, the *p*-value based on *T*(9) is 0.082 rather than 0.05. In Monte Carlo simulations by Cameron, Gelbach, and Miller (2008), this choice works reasonably well, and at a minimum one should use the *T*(*G* – 1) distribution rather than the standard normal.

For models that include *L* regressors that are invariant within cluster, Donald and Lang (2007) provides a rationale for using the *T*(*G* – *L*) distribution. If clusters are balanced and all regressors are invariant within cluster, then the OLS estimator in the model . is equivalent to OLS estimation in the grouped model . If is iid normally distributed, then the Wald statistic is *T*(*G* – *L*) distributed, where and . Note that is iid normal in the random effects model if the error components are iid normal. Usually, if there is a time-invariant regressor there is only one in addition to the intercept, in which case *L* = 2.

Donald and Lang extends this approach to inference on **β** in a model that additionally includes regressors * z_{ig}* that vary within clusters and allow for unbalanced clusters, leading to

*G*–

*L*for the RE estimator. Wooldridge (2006) presents an expansive exposition of the Donald and Lang approach. It also proposes an alternative approach based on minimum distance estimation. See Wooldridge (2006) and, for a summary, Cameron and Miller (2011).

#### 2. Data-determined degrees of freedom

For testing the difference between two means of normally and independently distributed populations with different variances, the *t* test is not exactly *T* distributed. This is known as the Behrens-Fisher problem. Satterthwaite (1946) proposed an approximation that was extended to regression with clustered errors in Bell and McCaffrey (2002) and developed further in Imbens and Kolesar (2012).

The *T*(*N* – *k*) distribution is the ratio of *N*[0,1] to independent . For linear regression under iid normal errors, the derivation of the *T*(*N* – *k*) distribution for the Wald *t*-statistic uses the result that , where is the usual unbiased estimate of . This result no longer holds for non-iid errors, even if they are normally distributed. Instead, an approximation uses the *T*(*v**) distribution where *v** is such that the first two moments of equal the first two moments (*v** and 2*v**) of the χ^{2}(*v**) distribution. Assuming is unbiased for , . And if .

Thus, the Wald *t*-statistic is treated as being *T*(*v**) distributed where . Assumptions are needed to obtain an expression for . For clustered errors with * u* ~

*N*[

**0**, Ω] and using the CRVE defined in Section IIC, or using CR2VE or CR3VE defined in Section VIB, Bell and McCaffrey (2002) shows that the distribution of the Wald

*t*-statistic defined in Equation 23 can be approximated by the

*T*(

*v**) distribution where

(26)

and λ_{j} are the eigenvalues of the *G* ´ *G* matrix , where is consistent for Ω, the *N* × *G* matrix * G* has

*g*column is the

^{th}*N*×

_{g}*N*submatrix for cluster

*g*of the

*N*×

*N*matrix , for CR2VE, and

*is a*

**e**_{k}*K*× 1 vector of zeroes aside from 1 in the

*k*position if . Note that

^{th}*v** needs to be calculated separately, and differs, for each regression coefficient. The method extends to Wald tests based on scalar linear combinations .

The justification relies on normal errors and knowledge of Ω = E[* uu′* |

*]. Bell and McCaffrey (2002) performs simulations with balanced clusters (*

**X***G*= 20 and

*N*= 10) and equicorrelated errors within cluster. It calculates

_{g}*v** assuming Ω = σ

^{2}

*, even though errors are in fact clustered, and finds that its method leads to Wald tests with true size closer to the nominal size than tests based on the conventional CRVE, CRV2E, and CRV3E.*

**I**Imbens and Kolesar (2012) additionally considers calculating *v** where is based on equicorrelated errors within cluster. It follows the Monte Carlo designs of Cameron, Gelbach, and Miller (2008), with *G* = 5 and 10 and equicorrelated errors. It finds that all finite-sample adjustments perform better than using the standard CRVE with *T*(*G* – 1) critical values. The best methods use the CR2VE and *T*(*v**), with slight overrejection with *v** based on (Bell and McCaffrey) and slight underrejection with *v** based on assuming equicorrelated errors (Imbens and Kolesar). For *G* = 5, these methods outperform the two-point wild cluster bootstrap as expected given the very low *G* problem discussed in Section VIC. More surprisingly, these methods also outperform wild cluster bootstrap when *G* = 10, perhaps because Imbens and Kolesar (2012) may not have imposed the null hypothesis in forming the residuals for this bootstrap.

#### 3. Effective number of clusters

Carter, Schnepel, and Steigerwald (2013) proposes a measure of the effective number of clusters. This measure is

(27)

where , is a *K* × 1 vector of zeroes aside from 1 in the *k ^{th}* position if , and . Note that

*G** varies with the regression coefficient considered, and the method extends to Wald tests based on scalar linear combinations .

The quantity δ measures cluster heterogeneity, which disappears if γ* _{g}* = γ for all

*g*. Given the formula for γ

*, cluster heterogeneity (δ ≠ 0) can arise for many reasons including variation in*

_{g}*N*, variation in

_{g}*, and variation in Ω*

**X**_{g}*across clusters. In simulations using standard normal critical values, Carter et al. (2013) finds that test size distortion occurs for*

_{g}*G** < 20. In application, it assumes errors are perfectly correlated within cluster, so Ω

*=*

_{g}*where*

**ll′***is an*

**l***N*× 1 vector of ones. For data from the Tennessee STAR experiment, it finds

_{g}*G** = 192 when

*G*= 318. For the Hersch (1998) data of Section IIB, with very unbalanced clusters, it finds for the industry job risk coefficient and with clustering on industry that

*G** = 19 when

*G*= 211.

Carter et al. (2013) does not actually propose using critical values based on the *T*(*G**) distribution. The key component in obtaining the formula for *v** in the Bell and McCaffrey (2002) approach is determining , which equals given is unbiased for . Carter et al. (2013) instead works with where , defined in this paper, is an approximation to that is good for large *G* (formally . Now , see Carter et al. (2013), where δ is defined in Equation 27. This suggests using the *T*(*G**) distribution as an approximation and that this approximation will improve as *G* increases.

### E. Special Cases

With few clusters, additional results can be obtained if there are many observations in each group. In DiD studies, the few-clusters problem arises if few groups are treated, even if *G* is large. And the few-clusters problem is more likely to arise if there is multiway clustering.

#### 1. Fixed number of clusters with cluster size growing

The preceding adjustments to the degrees of freedom of the *T* distribution are based on the assumption of normal errors. In some settings, asymptotic results can be obtained when *G* is small, provided *N _{g}* → ∞.

Bester, Conley, and Hansen (2011), building on Hansen (2007a), gives conditions under which the *t*-test statistic based on Equation 11 is times *T*_{G–1} distributed. Then using in Equation 11 yields a *T*(*G* – 1) distributed statistic. In addition to assuming *G* is fixed while *N _{g}* → ∞, it is assumed that the within group correlation satisfies a mixing condition (this does not happen in all data settings, although it does for time series and spatial correlation), and that homogeneity assumptions are satisfied, including equality of plim (1/

*N*)

_{g}

*X′*_{g}

**X**_{g}for all

*g*.

Let denote the estimate of parameter β in cluster *g*, denote the average of the *G* estimates, and denote their variance. Suppose that the are asymptotically normal as *N _{g}* → ∞ with common mean β, and consider test of

*H*

_{0}: β = β

_{0}based on . Then Ibragimov and Müller (2010) shows that tests based on the

*T*(

*G*– 1) distribution will be conservative tests (that is, underreject) for level ≤ 0.083. This approach permits correct inference even with extremely few clusters, assuming

*N*is large. However, the requirement that cluster estimates are asymptotically independent must be met. Thus, the method is not directly applicable to a state-year DiD application when there are year fixed effects (or other regressor that varies over time but not states). In that case, Ibragimov and Müller proposes applying its method after aggregating subsets of states into groups in which some states are treated and some are not.

_{g}#### 2. Few treated groups

Problems arise if most of the variation in the regressor is concentrated in just a few clusters, even when *G* is sufficiently large. This occurs if the key regressor is a cluster-specific binary treatment dummy and there are few treated groups.

Conley and Taber (2011) examines a differences-in-differences (DiD) model in which there are few treated groups and an increasing number of control groups. If there are group-time random effects, then the DiD model is inconsistent because the treated groups random effects are not averaged away. If the random effects are normally distributed, then the model of Donald and Lang (2007) applies and inference can use a *T* distribution based on the number of treated groups. If the group-time shocks are not random, then the *T* distribution may be a poor approximation. Conley and Taber (2011) then proposes a novel method that uses the distribution of the untreated groups to perform inference on the treatment parameter.

Abadie, Diamond, and Hainmueller (2010) proposes synthetic control methods that provide a data-driven method to select the control group in a DiD study and that provide inference under random permutations of assignment to treated and untreated groups. The methods are suitable for treatment that affects few observational units.

#### 3. What if you have multiway clustering and few clusters?

Sometimes we are worried about multiway clustering but one or both of the ways has few clusters. Currently, we are not aware of an ideal approach to deal with this problem. One potential solution is to try to add sufficient control variables so as to minimize concerns about clustering in one of the ways, and then use a one-way few-clusters cluster robust approach on the other way. Another potential solution is to model one of the ways of clustering in a parametric way, such as with a common shock or an autoregressive error model. Then you can construct a variance estimator that is a hybrid of the parametric model and cluster robust in the remaining dimension.

## VII. Extensions

The preceding material has focused on the OLS (and FGLS) estimator and tests on a single coefficient. The basic results generalize to multiple hypothesis tests, instrumental variables (IV) estimation, nonlinear estimators, and generalized method of moments (GMM).

These extensions are incorporated in Stata though Stata generally computes test *p*-values and confidence intervals using standard normal and chi-squared distributions rather than *T* and *F* distributions. And for nonlinear models, stronger assumptions are needed to ensure that the estimator of **β** retains its consistency in the presence of clustering. We provide a brief overview.

### A. Cluster-Robust *F-*tests

Consider Wald joint tests of several restrictions on the regression parameters. Except in the special case of linear restrictions and OLS with iid normal errors, asymptotic theory yields only a chi-squared distributed statistic, such as *W*, that is χ^{2}(*h*) distributed where *h* is the number of (linearly independent) restrictions.

Alternatively, we can use the related *F* statistic, *F* = *W* / *h*. This yields the same *p*-value as the chi-squared test if we treat *F* as being *F*(*h*, ∞) distributed. In the cluster case, a finite-sample adjustment instead treats *F* as being *F*(*h*, *G* – 1) distributed. This is analogous to using the *T*(*G* – 1) distribution rather than *N*[0,1] for a test on a single coefficient.

In Stata, the finite-sample adjustment of using the *T*(*G* – 1) for a *t*-test on a single coefficient, and using the *F*(*h*,*G* – 1) for an *F*-test, is only done after OLS regression with command *regress*. Otherwise, Stata reports critical values and *p*-values based on the *N*[0,1] and χ^{2}(*h*) distributions.

Thus, Stata does no finite-cluster correction for tests and confidence intervals following instrumental variables estimation commands, nonlinear model estimation commands, or even after command *regress* in the case of tests and confidence intervals using commands *testnl* and *nlcom*. The discussion in Section VI was limited to inference after OLS regression, but it seems reasonable to believe that for other estimators one should also base inference on the *T*(*G* – 1) and *F*(*h,G* – 1) distributions, and even then tests may overreject when there are few clusters.

Some of the few-cluster methods of Section VI can be extended to tests of more than one restriction following OLS regression. The Wald test can be based on the bias-adjusted variance matrices CR2VE or CR3VE, rather than CRVE. For a bootstrap with asymptotic refinement of a Wald test of *H*_{0} : **R****β** = * r*, in the

*b*resample we compute . Extension of the data-determined degrees of freedom method of Section VID to tests of more than one restriction requires, at a minimum, extension of Theorem 4 of Bell and McCaffrey (2002) from the case that covers β, where β is a single component of β, to

^{th}

**R****β**. An alternative ad hoc approach would be to use the distribution where is an average (possibly weighted by estimator precision) of

*v** defined in Equation 26 computed separately for each exclusion restriction.

For the estimators discussed in the remainder of Section VII, the rank of is again the minimum of *G* – 1 and the number of parameters (*K*). This means that at most *G* – 1 restrictions can be tested using a Wald test, in addition to the usual requirement that *h* ≤ *K*.

### B. Instrumental Variables Estimators

The cluster-robust variance matrix estimate for the OLS estimator extends naturally to the IV estimator, the two-stage least squares (2SLS) estimator, and the linear GMM estimator.

The following additional adjustments must be made when errors are clustered. First, a modified version of the Hausman test of endogeneity needs to be used. Second, the usual inference methods when instruments are weak need to be adjusted. Third, tests of overidentifying restrictions after GMM need to be based on an optimal weighting matrix that controls for cluster correlation of the errors.

#### 1. IV and 2SLS

In matrix notation, the OLS estimator in the model * y* =

**X****β**+

*is inconsistent if E[*

**u***|*

**u***] ≠*

**X****0**. We assume existence of a set of instruments

*that satisfy E[*

**Z***|*

**u***] =*

**Z****0**and satisfy other conditions, notably

*is of full rank with dim[*

**Z***] ≥ dim[*

**Z***] and Cor[*

**X***,*

**Z***] ≠*

**X****0**.

For the clustered case, the assumption that errors in different clusters are uncorrelated is now one of uncorrelated errors conditional on the instruments * Z*, rather than uncorrelated errors conditional on the regressors

*. In the*

**X***g*cluster the matrix of instruments

^{th}*is an*

**Z**_{g}*N*´

_{g}*M*matrix, where

*M*³

*K*, and we assume that E[

*|*

**u**_{g}*] =*

**Z**_{g}**0**and for

*g*≠

*h*.

In the just-identified case, with * Z* and

*having the same dimension, the IV estimator is , and the cluster-robust variance matrix estimate is*

**X**(28)

where are residuals calculated using the consistent IV estimator. We again assume *G* → ∞. As for OLS, the CRVE may be rank-deficient with rank the minimum of *K* and *G* – 1.

In the overidentified case with * Z* having dimension greater than

*, the 2SLS estimator is the special case of the linear GMM estimator in Equation 29 below with*

**X***= (*

**W**

**Z′***)*

**Z**^{-1}, and the CRVE is that in Equation 30 below with

*= (*

**W**

**Z′***)*

**Z**^{-1}and the 2SLS residuals. In the just-identified case, 2SLS is equivalent to IV.

A test for endogeneity of a regressor(s) can be conducted by comparing the OLS estimator to the 2SLS (or IV) estimator that controls for this endogeneity. The two estimators have the same probability limit given exogeneity and different probability limits given endogeneity. This is a classic setting for the Hausman test but, as in the Hausman test for fixed effects discussed in Section IIID, the standard version of the Hausman test cannot be used when errors are clustered. Instead let *X* = [**X**_{1} **X**_{2}], where **X**_{1} is potentially endogenous and **X**_{2} is exogenous, and let denote the residuals from first-stage OLS regression of the endogenous regressors on instruments and exogenous regressors. Then estimate by OLS the model

The regressors **x**_{1} are considered endogenous if we reject *H*_{0} : **γ** = **0** using a Wald test based on a CRVE. In Stata, this is implemented using command *estat endogenous*. (Alternatively, a pairs cluster bootstrap can be used to estimate the variance of ).

#### 2. Weak instruments

When endogenous regressor(s) are weakly correlated with instrument(s), after partialling out the exogenous regressors in the model, there is great loss of precision. Then the standard error for the coefficient of the endogenous regressor is much higher after IV or 2SLS estimation than after OLS estimation.

Additionally, asymptotic theory takes an unusually long time to kick in so that even with large samples the IV estimator can still have considerable bias and the Wald statistic is still not close to normally distributed. See, for example, Bound, Jaeger, and Baker (1995), Andrews and Stock (2007), and textbook discussions in Cameron and Trivedi (2005, 2009).

For this second consequence, called the “weak instrument” problem, the econometrics literature has focused on providing theory and guidance in the case of homoskedastic errors. Not all of the proposed methods extend to errors that are correlated within cluster. And the problem may even be greater in the clustered case, as the asymptotics are then in *G* → ∞ rather than *N* → ∞, though we are unaware of evidence on this.

We begin with a case of a single endogenous regressor. A standard diagnostic for detecting weak instruments is to estimate by OLS the first-stage regression of the endogenous regressor on the exogenous regressors and the additional instrument(s). Then calculate the *F*-statistic for the joint significance of the instruments; in the case of a just-identified model there is only one instrument to test so the *F*-statistic is the square of the *t*-statistic. With clustered errors, this *F*-statistic needs to be based on a cluster-robust variance matrix estimate. It is common practice to interpret the cluster-robust *F*-statistic in the same way as when errors are iid, using the tables of Stock and Yogo (2005) or the popular rule-of-thumb, due to Staiger and Stock (1997), that there may be a weak instrument problem if *F* < 10. But it should be noted that these diagnostics for weak instruments were developed for the simpler case of iid errors. Note also that the first-stage cluster-robust *F*-statistic can only be computed if the number of instruments is less than the number of clusters.

With more than one endogenous variable and iid errors, the *F*-statistic generalizes to the Cragg-Donald minimum eigenvalue statistic, and one can again use the tables of Stock and Yogo (2005). For clustered errors generalizations of the Cragg-Donald minimum eigenvalue statistic have been proposed, see Kleibergen and Paap (2008), but it is again not clear whether these statistics can be compared to the Stock and Yogo tables when errors are clustered.

Now consider statistical inference that is valid even if instruments are weak, again beginning with the case of a single endogenous regressor. Among the several testing methods that have been proposed given iid errors, the Anderson-Rubin method can be generalized to the setting of clustered errors. Consider the model *y _{ig}* = β

*x*+

_{ig}*u*, where the regressor

_{ig}*x*is endogenous and the first-stage equation is . (If there are additional exogenous regressors

**x**_{2}, as is usually the case, the method still works if the variables

*y*,

*x*, and

*are defined after partialling out*

**z**

**x**_{2}.) The two equations imply that , where

*w*=

_{ig}*u*+

_{ig}*v*(β −β*). So a test of β = β* is equivalent to a Wald test of

_{ig}**γ**=

**0**in the model . With clustered errors, the test is based on cluster-robust standard errors.

Additionally, a weak instrument 95 percent confidence interval for β can be constructed by regressing *y _{ig}* −β*

*x*on

_{ig}**z**

*for a range of values of β* and including in the confidence interval for β only those values of β* for which we did not reject*

_{ig}*H*

_{0}:

**γ**=

**0**when testing at 5 percent. As in the iid case, this can yield confidence intervals that are unbounded or empty, and the method loses power when the model is overidentified.

When there is more than one endogenous regressor, this method can also be used, but it can only perform a joint *F*-test on the coefficients of all endogenous regressors rather than separate tests for each of the endogenous regressors.

Chernozhukov and Hansen (2008) provides a simple presentation of the method and an empirical example. Finlay and Magnusson (2009) provides this and other extensions, and provides a command *ivtest* for Stata. We speculate that if additionally there are few clusters, then some of the adjustments discussed in Section VI would help.

Baum, Schaffer, and Stillman (2007) provides a comprehensive discussion of various methods for IV, 2SLS, limited information maximum likelihood (LIML), *k*-class, continuous updating, and GMM estimation in linear models, and present methods using their *ivreg2* Stata command. They include weak instruments methods for errors that are iid, heteroskedastic, or within-cluster correlated errors.

#### 3. Linear GMM

For overidentified models, the linear GMM estimator is more efficient than the 2SLS estimator if E[* uu′* |

*] ≠ σ*

**Z**^{2}

*. Then*

**I**(29)

where * W* is a full rank

*K*×

*K*weighting matrix. The CRVE for GMM is

(30)

where are residuals calculated using the GMM estimator.

For clustered errors, the efficient two-step GMM estimator uses ,where are 2SLS residuals. Implementation of this estimator requires that the number of clusters exceeds the number of instruments as otherwise is not invertible. Here * Z* contains both the exogenous regressors in the structural equation and the additional instruments required to enable identification of the endogenous regressors. When this condition is not met, Baum, Schaffer, and Stillman (2007) proposes doing two-step GMM after first partialling out the instruments

*from the dependent variable*

**z***y*, the endogenous variables in the initial model , and any additional instruments that are not also exogenous regressors in this model.

The overidentifying restrictions (OIR) test, also called a Hansen test or a Sargan test, is a limited test of instrument validity that can be used when there are more instruments than necessary. When errors are clustered, the OIR tests must be computed following the cluster version of two-step GMM estimation; see Hoxby and Paserman (1998).

Just as GLS is more efficient than OLS, specifying a model for Ω = E[* uu′* |

*] can lead to more efficient estimation than GMM. Given a model for Ω, and conditional moment condition E[*

**Z***|*

**u***] =*

**Z****0**, a more efficient estimator is based on the unconditional moment condition E[

*Ω*

**Z′**^{-1}

*] =*

**u****0**. Then we minimize , where is consistent for Ω. Furthermore, the CRVE can be robustified against misspecification of Ω, similar to the case of FGLS. In practice, such FGLS-type improvements to GMM are seldom used, even in simpler settings than the clustered setting. An exception is Shore-Sheppard (1996), which considers the impact of equicorrelated instruments and group-specific shocks in a model similar to that of Moulton. One reason may be that this option is not provided in Stata command

*ivregress*. In the special case of a random effects model for Ω, command

*xtivreg*can be used along with a pairs cluster bootstrap used to guard against misspecification of Ω.

### C. Nonlinear Models

For nonlinear models, there are several ways to handle clustering. We provide a brief summary; see Cameron and Miller (2011) for further details.

For concreteness, we focus on logit regression. Recall that in the cross-section case *y _{i}* takes value 0 or 1 and the logit model specifies that , where Λ(

*Z*) =

*e*/ (1 +

^{z}*e*).

^{z}#### 1. Different models for clustering

The simplest approach is a pooled approach that assumes that clustering does not change the functional form for the conditional probability of a single observation. Thus, for the logit model, whatever the nature of clustering, it is assumed that

(31)

This is called a population-averaged approach, as is obtained after averaging out any within-cluster correlation. Inference needs to control for within-cluster correlation, however, and more efficient estimation may be possible.

The generalized estimating equations (GEE) approach, due to Liang and Zeger (1986) and widely used in biostatistics, introduces within-cluster correlation into the class of generalized linear models (GLM), a class that includes the logit model. One possible model for within-cluster correlation is equicorrelation, with Cor[*y _{ig}*,

*y*

_{jg}|

*,*

**x**_{ig}

**x**_{jg}] = ρ. The Stata command

*xtgee y x, family(binomial) link(logit) corr(exchangeable)*estimates the population-averaged logit model and provides the CRVE assuming the equicorrelation model for within-cluster correlation is correctly specified. The option

*vce(robust)*provides a CRVE that is robust to misspecification of the model for within-cluster correlation. Command

*xtgee*includes a range of models for the within-error correlation. The method is a nonlinear analog of FGLS given in Section IID, and asymptotic theory requires

*G*→ ∞.

A further extension is nonlinear GMM. For example, with endogenous regressors and instruments * z* that satisfy , a nonlinear GMM estimator minimizes

*(β)′)*

**h***(*

**Wh****β**) where . Other choices of

*(*

**h****β**) that allow for intracluster correlation may lead to more efficient estimation, analogous to the linear GMM example discussed at the end of Section VIIB. Given a choice of

*(*

**h****β**), the two-step nonlinear GMM estimator at the second step uses weighting matrix

*that is the inverse of a consistent estimator of V[*

**W***(*

**h****β**)], and one can then use the minimized objection function for an overidentifying restrictions test.

Now suppose we consider a random effects logit model with normally distributed random effect, so

(32)

where . If α* _{g}* is known, the

*N*observations in cluster

_{g}*g*are independent with joint density

Since α* _{g}* is unknown it is integrated out, leading to joint density

where is the density. There is no closed form solution for this integral, but it is only one-dimensional so numerical approximation (such as Gaussian quadrature) can be used. The consequent MLE can be obtained in Stata using the command *xtlogit y x, re*. Note that in this RE logit model Equation 31 no longer holds, so **β** in the Model 32 is scaled differently from **β** in Equation 31. Furthermore, **β** in Equation 32 is inconsistent if the distribution for α* _{g}* is misspecified, so there is little point in using option

*vce(robust)*after command

*xtlogit, re*.

It is important to realize that in nonlinear models such as logit, the population-averaged and random effects approaches lead to quite different estimates of **β** that are not comparable since **β** means different things in the different models. The resulting estimated average marginal effects may be similar, however, just as they are in standard cross-section logit and probit models.

With few clusters, Wald statistics are likely to overreject as in the linear case, even if we scale the CRVE’s given in this section by *G* / (*G* – 1) as is typically done; see Equation 12 for the linear case. McCaffrey, Bell, and Botts (2001) considers bias-correction of the CRVE in generalized linear models. Asymptotic refinement using a pairs cluster bootstrap as in Section VIC is possible. The wild bootstrap given in Section VID is no longer possible in a nonlinear model, aside from nonlinear least squares, as it requires additively separable errors. Instead, one can use the score wild bootstrap proposed by Klein and Santos (2012) for nonlinear models, including maximum likelihood and GMM models. The idea in this paper is to estimate the model once, generate scores for all observations, and then perform a bootstrap in the wild-cluster style, perturbing the scores by bootstrap weights at each step. For each bootstrap replication, the perturbed scores are used to build a test statistic, and the resulting distribution of this test statistic can be used for inference. It finds that this method performs well in small samples and can greatly ease computational burden because the nonlinear model need only be estimated once. The conservative test of Ibragimov and Müller (2010) can be used if *N _{g}* → ∞.

#### 2. Fixed effects

A cluster-specific fixed effects version of the logit model treats the unobserved parameter α* _{g}* in Equation 32 as being correlated with the regressors

*. In that case, both the population-averaged and random effects logit estimators are inconsistent for*

**x**_{ig}**β**.

Instead, we need a fixed effects logit estimator. In general, there is an incidental parameters problem if asymptotics are that *N _{g}* is fixed while

*G*→ ∞, as there are only

*N*observations for each α

_{g}*, and inconsistent estimation of α*

_{g}*spills over to inconsistent estimation of*

_{g}**β**. Remarkably for the logit model it is nonetheless possible to consistently estimate

**β**. The logit fixed effects estimator is obtained in Stata using the command

*xtlogit y x, fe*. Note, however, that the marginal effect in Model 32 is . Unlike the linear FE model, this depends on the unknown α

*. So the marginal effects cannot be computed though the ratio of the marginal effects of the*

_{g}*k*and

^{th}*l*regressor equals β

^{th}*/ β*

_{k}*, which can be consistently estimated.*

_{l}The logit model is one of few nonlinear models for which fixed effects estimation is possible when *N _{g}* is small. The other models are Poisson with , and nonlinear models with , where

*m*(·) is a specified function.

The natural approach to introduce cluster-specific effects in a nonlinear model is to include a full set of cluster dummies as additional regressors. This leads to inconsistent estimation of **β** in all models except the linear model (estimated by OLS) and the Poisson regression model, unless *N _{g}* → ∞. There is a growing literature on bias-corrected estimation in such cases; see, for example, Fernández-Val (2009). This paper also cites several simulation studies that gauge the extent of bias of dummy variable estimators for moderate

*N*, such as

_{g}*N*= 20.

_{g}Yoon and Galvao (2013) considers fixed effects in panel quantile regression models with correlation within cluster and provides methods under the assumption that both the number of individuals and the number of time periods go to infinity.

### D. Cluster-Randomized Experiments

Increasingly, researchers are gathering their own data, often in the form of field or laboratory experiments. When analyzing data from these experiments, they will want to account for the clustered nature of the data. And so when designing these experiments, they should also account for clustering. Fitzsimons et al. (2012) uses a wild cluster bootstrap in an experiment with 12 treated and 12 control clusters.

Traditional guidance for computing power analyses and minimum detectable effects (see Duflo, Glennerster, and Kremer 2007, pp. 3918–22; Hemming and Marsh 2013) are based on assumptions of either independent errors or, in a clustered setting, a random effects common-shock model. Ideally, one would account for more general forms of clustering in these calculations (the types of clustering that motivate cluster-robust variance estimation), but this can be difficult to do ex ante. If you have a data set that is similar to the one you will be analyzing later, then you can assign a placebo treatment and compute the ratio of cluster-robust standard errors to default standard errors. This can provide a sense of how to adjust the traditional measures used in design of experiments.

## VIII. Empirical Example

In this section, we illustrate the most common applications of cluster-robust inference. There are two examples. The first is a Moulton-type setting that uses individual-level cross section data with clustering on state. The second is the Bertrand et al. (2004) example of DiD in a state-year panel with clustering on state and potentially with state fixed effects.

The microdata are from the March CPS, downloaded from IPUMS-CPS (King et al. 2010). We use data covering individuals who worked 40 or more weeks during the prior year and whose usual hours per week in that year was 30 or more. The hourly wage is constructed as annual earnings divided by annual hours (usual hours per week times number of weeks worked), deflated to real 1999 dollars, and observations with real wage in the range ($2, $100) are kept.

The cross-section example uses individual-level data for 2012. The panel example uses data aggregated to the state-year level for 1977 to 2012. In both cases, we estimate log-wage regressions and perform inference on a generated regressor that has zero coefficient. Specifically, we test *H*_{0} : β = 0 using . For each example, we present results for a single data set before presenting a Monte Carlo experiment that focuses on inference when there are few clusters.

We contrast various ways to compute standard errors and perform Wald tests. Even when using a single statistical package, different ways to estimate the same model may lead to different empirical results due to calculation of different degrees of freedom, especially in models with fixed effects, and due to uses of different distributions in computing *p*-values and critical values. To make this dependence clear, we provide the particular Stata command used to obtain the results given below; similar issues arise if alternative statistical packages are employed.

The data and accompanying Stata code (Version 13) are available at our websites.

### A. Individual-Level Cross-Section Data: One Sample

In our first application, we use data on 65,685 individuals from the year 2012. The model is

(33)

where *y _{ig}* is log-wage,

*d*is a randomly generated dummy “policy” variable equal to one for one-half of the states and zero for the other half, and

_{g}*is a set of individual-level controls (age, age squared, and education in years). Estimation is by OLS and by FGLS controlling for state-level random effects.*

**z**_{ig}The policy variable *d _{g}* is often referred to as a “placebo” treatment and should be statistically significant in only 5 percent of tests performed at significance level 0.05.

Table 1 reports the estimated coefficient of the policy variable, along with its standard error computed in several different ways, when there are 51 clusters (states).

OLS results given in the first column of Table 1 are obtained using Stata command *regress*. The default standard error is misleadingly small (*se* = 0.0042), leading to the dummy variable being very highly statistically significant (*t* = –0.0226 / 0.0042 = –5.42) even though it was randomly generated independently of log-wage. The White heteroskedastic-robust standard error, from *regress* option *vce(robust)*, is similar in magnitude. From Section IVA, White standard errors should not be used if *N _{g}* is small but here

*N*is large. The cluster-robust standard error (

_{g}*se*= 0.0229) using option

*vce(cluster state)*is 5.5 times larger and leads to the more sensible result that the regressor is statistically insignificant (

*t*= –0.99). In results not presented in Table 1, the cluster-robust standard errors of the other regressors—age, age squared, and education—were, respectively, 1.2, 1.2, and 2.3 times the default. So ignoring clustering again understates the standard errors. As expected, a pairs cluster bootstrap (without asymptotic refinement) using option

*vce(boot, cluster(state))*, yields very similar cluster-robust standard error.

Note that Formula 6 suggests that the cluster-robust standard errors are 4.9 times the default , close to the observed multiple of 5.5. Formula 6 may work especially well in this example as taking the natural logarithm of wage leads to model error that is close to homoskedastic and equicorrelation is a good error model for individuals clustered in regions.

FGLS estimates for a random effects model with error process defined in Equation 16 are given in the second column of Table 1. These were obtained using command *xtreg, re* after *xtset state*. The cluster-robust standard error defined in Equation 15 and computed using option *vce(robust)* is 0.0214/0.0199 = 1.08 times larger than the default. The pairs cluster bootstrap, implemented using option *vce(boot)*, yields a similar cluster-robust standard error.

In principle, FGLS can be more efficient than OLS. In this example, there is a modest gain in efficiency with the cluster-robust standard error equal to 0.0214 for FGLS compared to 0.0229 for OLS.

Finally, to illustrate the potential pitfalls of pairs cluster bootstrapping for standard errors when there are few clusters, discussed in Section VIC, we examine a modification with only six states broken into treated (AZ, LA, MD) and control (DE, PA, UT). For these six states, we estimate a model similar to that in Table 1. Then with default *se* = 0.0128. We then perform a pairs cluster bootstrap with 999 replications. The bootstrap *se* = 0.0622 is similar to the cluster-robust *se* = 0.0577. However, several problems arise. First, 28 replications cannot be estimated, presumably due to no variation in treatment in the bootstrap samples. Second, a kernel density estimate of the bootstrapped *s* reveals that their distribution is very multimodal and has limited density near the middle of the distribution. Considering these results, we would not feel comfortable using the pairs cluster bootstrap in this data set with these few clusters. Better to base inference on a wild cluster bootstrap.

This example highlights the need to use cluster-robust standard errors even when model errors are only weakly correlated within cluster (the intraclass correlation of the residuals in this application is 0.018), if the regressor is substantially correlated within cluster (here perfectly correlated within cluster), and/or cluster sizes are large (ranging here from 519 to 5,866).

### B. Individual-Level Cross-Section Data: Monte Carlo

We next perform a Monte Carlo exercise to investigate the performance of various cluster-robust methods as the number of clusters becomes small. The analysis is based on the same cross-section regression as in the previous subsection.

In each replication, we generate a data set by sampling (with replacement) states and all their associated observations. For quicker computation of the Monte Carlo simulation, within each state we use only a 20 percent subsample of individuals, so there are on average approximately 260 observations per cluster.

We explore the effect of the number of clusters *G* by performing varying simulations with *G* equal to 6, 10, 20, 30, or 50. Given a sample of states, we assign a dummy “policy” variable to one-half of the states. We run OLS regressions of log-wage on the policy variable and the same controls as used for the Table 1 regressions.

In these simulations, we perform tests of the null hypothesis that the slope coefficient of the policy variable is zero. Table 2 presents rejection rates that with millions of replications should equal 0.05, since we are testing a true hypothesis at a nominal 5 percent level. For *G* = 6 and 10, we perform 4,000 simulations, so we expect that 95 percent of these simulations will yield estimated test size in the range (0.043, 0.057) if the true test size is 0.05. For larger *G*, there are 1,000 simulations and the 95 percent simulation interval is instead (0.036, 0.064).

We begin with lengthy discussion of the many clusters case. These results are given in the final column (*G* = 50) of Table 2. Rows 1–9 report sizes for Wald tests based on where *se* is computed in various ways while Rows 10–15 report sizes for tests using various bootstraps with an asymptotic refinement. Basic Stata commands yield the standard errors in Rows 1–4 and 9, while the remaining rows require additional coding.

Row 1 presents the size of tests using heteroskedastic-robust standard errors, obtained using Stata command *regress, vce(robust)*. Ignoring clustering leads to great overrejection due to considerable underestimation of the standard error. Using Formula 6 for this 20 percent subsample yields a standard error inflation factor of . So *t* = 1.96 using the heteroskedastic-robust standard error is really *t* = 1.96 / 2.38 = 0.82. And using standard normal critical values, an apparent *p* = 0.05 is really *p* = 0.41 since Pr[|z| > 0.82] = 0.41. This crude approximation is fairly close to *p* = 0.498 obtained in this simulation.

Results using cluster-robust standard errors, presented in Rows 2–4 and obtained from *regress, vce(cluster state)*, show that even with 50 clusters the choice of distribution to use in obtaining critical value makes a difference. The rejection rate is closer to 0.05 when *T*(*G* – 1) critical values are used than when *N*[0,1] critical values are used. Using *T*(*G* – 2) in Row 4, suggested by the study of Donald and Lang (2007), leads to slight further improvement, but there is still overrejection.

Results using the bias adjustments CR2 and CR3 discussed in Section VIB, along with *T*(*G* – 1) critical values, are presented in Rows 5–6. Bias adjustment leads to further decrease in the rejection rates, toward the desired 0.05.

Rows 7 and 8 use critical values from the *T* distribution with the data-determined degrees-of-freedom of Section VID, equal to 17 on average when *G* = 50 (see Rows 14 and 17). This leads to further improvement in the Monte Carlo rejection rate.

Bootstrap standard errors obtained from a standard pairs cluster bootstrap, implemented using command *regress, vce(boot, cluster(state))* are used in Row 9. For *G* = 50, the rejection rate is essentially the same as that in Row 3, as expected since this bootstrap has no asymptotic refinement.

Rows 10–15 implement the various percentile-*t* bootstraps with asymptotic refinement presented in Section VIC. Only 399 bootstraps are used here as any consequent bootstrap simulation error averages out over the many Monte Carlo replications. But if these bootstraps were used just once, as in an empirical study, a percentile-*t* bootstrap should use at least 999 replications. Row 10 can be computed using the *bootstrap:* command, see our posted code, while Rows 11–15 require additional coding. For *G* = 50, the various bootstraps give similar results with rejection rates of a bit more than 0.06.

Rows 16–19 give the effective number of clusters. The Imbens and Kolesar (2013) measure *v** in Equation 26, denoted IK, and the Carter, Schnepel, and Steigerwald (2013) measure *G** in Equation 27, denoted CSS, are both equal to 17 on average when *G* = 50. For the IK measure, across the 1,000 simulations the 5^{th} percentile is 9.6 and the 95^{th} percentile is 29.5.

We next examine settings with fewer clusters than *G* = 50. Then most methods lead to rejection rates even further away from the nominal test size of 0.05.

Consider the case *G* = 6. Rows 2–4 and 8 compute the same Wald test statistic but use different degrees of freedom in computing *p*-values. This makes an enormous difference when *G* is small, as the critical value for a Wald test at level 0.05 rises from 2.571 to 2.776 and 3.182 for, respectively, the *T*(5), *T*(4), and *T*(3) distributions, and from Row 16 the IK degrees of freedom averages 3.3 across the simulations. The CSS degrees of freedom is larger than the IK as, from Section VID, it involves an approximation that only disappears as *G* becomes large.

Using a bias-corrected CRVE also makes a big difference. It appears from Rows 6 and 7 that it is best to use the CR3 bias-correction with *T*(*G* – 1) critical values, and the CR2 bias-correction with *T*(*v**) critical values where *v** is the Imbens and Kolesar (2013) calculated degrees of freedom.

A downside to using cluster-robust standard errors is that they provide an estimate of the standard deviation of that is more variable than the default or heteroskedastic-robust standard errors. This introduces a potential bias—variance tradeoff. To see whether this increased variability is an issue, we performed 1,000 Monte Carlo replications using the full cross-section microdata set, resampling the 50 states with replacement. The standard deviation of the cluster-robust standard error across the 1,000 replications equaled 12.3 percent of the mean cluster-robust standard error, while the standard deviation of the heteroskedastic-robust standard error equaled 4.5 percent of its mean. So while the CRVE is less biased than heteroskedastic-robust (or default), it is also more variable. But the increased variability is relatively small, especially compared to the very large bias that can arise if clustering is not controlled for.

Rows 10–15 present various bootstraps with asymptotic refinement. From Row 10, the pairs cluster bootstrap performs extremely poorly for *G* ≤ 10.

Results for the wild cluster bootstrap using a Rademacher 2 point distribution are presented in Rows 11–13. From Section VIB, this bootstrap yields only 2^{6} = 64 possible data sets when *G* = 6, and hence at most 64 unique values for *w**. This leads to indeterminacy for the test *p*-value. Suppose the *p*-value is in the range [a,b]. Then *H*_{0} is rejected in Row 11 if a < 0.05, in Row 12 if (a+b)/2 < 0.05, and in Row 13 if b < 0.05. The indeterminacy leads to substantially different results for *G* as low as six, though not for *G* ≥ 10.

The wild cluster bootstrap using the Webb 6 point distribution, see Row 14, does not have this complication when *G* = 6. And it yields essentially the same results as those using the Rademacher 2 point distribution when *G* ≥ 10.

Comparing Row 15 to Row 12, we see that imposing the null hypothesis in performing the wild bootstrap does not change the rejection rate very much in this set of simulations when *G* ≥ 10, although it appears to matter when *G* = 6. By comparison, we have found a more substantial difference when simulating from the d.g.p. of Cameron et al. (2008).

In summary, the various wild cluster bootstraps lead to test size that is closer to 0.05 than using a standard Wald test with cluster-robust standard errors and *T*(*G* – 1) critical values. But the test size still exceeds 0.05 and the bias adjustments in Rows 6 and 7 appear to do better.

This example illustrates that at a minimum one should use a cluster-robust Wald test with *T*(*G* – 1) critical values. Especially when there are few clusters, it is better still to use bias-adjusted cluster-robust standard errors or to use a wild cluster bootstrap. In this Monte Carlo experiment, with few clusters the test size was closest to the nominal size using a Wald test with cluster-robust standard errors computed using the CR2 correction and *T*(*v**) critical values, or using the larger CR2 correction with *T*(*G* – 1) critical values.

### C. State-Year Panel Data: One Sample

We next turn to a panel difference-in-difference application motivated by Bertrand et al. (2004). The underlying data are again individual-level data from the CPS but now obtained for each of the years 1977 to 2012.

In applications where the policy regressor of interest is only observed at the state-year level, it is common to first aggregate the individual-level data to the state-year level before OLS regression. Several methods are used. We use the following method:

The model estimated for 51 states from 1977–2012 is

(34)

where is the average log-wage in year *t* and state *s* (after partialling out individual level covariates), α* _{s}* and δ

*are state and year dummies, and*

_{t}*d*is a random “policy” variable that turns on and stays on for the last 18 years for one-half of the states. Here

_{ts}*G*= 51,

*T*= 36, and

*N*= 1836.

The individual level covariates (age, age squared, and years of education) are partialled out using a two-step estimation procedure presented in Hansen (2007b). Define *D _{ts}* to be state-by-year dummies. First, we OLS regress log wage (

*y*) on state-by-year dummies

_{its}*D*and on the individual level covariates. And second, in Equation 34 equals the estimated coefficients of the

_{ts}*D*dummies.

_{ts}To speed up bootstraps, and to facilitate computation of the CR2 residual adjustment, we additionally partial out the state fixed effects and year fixed effects in Equation 34 by the standard Frisch-Waugh method. We separately regress and *d _{ts}* on the state dummies and the year dummies. Then β is estimated by regressing the residuals of on the residuals of

*d*with no constant. As noted below, regression using residuals leads to slightly different standard errors due to different degrees of freedom used in calculating the CRVE.

_{ts}Table 3 presents results for the policy dummy regressor, which should have coefficient zero since it is randomly assigned.

We begin with Model 1, OLS controlling for state and year fixed effects. Using default or White-robust standard errors (Rows 1–2) leads to a standard error of 0.0037 that is much smaller than the cluster-robust standard error of 0.0119 (Row 3), where clustering is on state. Similar standard errors are obtained using the CR2 correction (Rows 4 and 5) and bootstrap without asymptotic refinement (Row 6). Note that from Rows 10 and 11 the IK and CSS degrees of freedom are calculated to be, respectively, *G* – 1 and *G*, an artifact of having balanced cluster sizes and a single regressor that is symmetric across clusters.

The inclusion of state and year fixed effects complicates computation of the degrees of freedom (*df*) adjustment used in computing the CRVE. The Row 3, Column 1 results are obtained from regression of residual on residual without intercept, using command *regress, noconstant vce(cluster(state))*. Then, from Formula 12 *df* = [*G* / (*G* −1)] × [*GT* / (*GT* −1)]. If instead we directly regressed log-wage on the state and year fixed effects and the *K* regressors, again using *regress, vce(cluster(state))*, then *df* = [*G* / (*G* −1)] × [*GT* / (*GT* − *T* – *G* −1)] and the cluster-robust standard error equals 0.0122 rather than 0.0119. And if instead we directly estimate the log-wage equation using *xtreg, vce(robust)* after *xtset state*, then *df* = *G* / (*G* −1) and the cluster-robust standard error equals 0.0120. In this example with large *T* and *G*, these adjustments make little difference, but for small *G* or *T* one should use *df* = *G* / (*G* −1) as explained in Section IIIB.

The corresponding *p*-values for tests of the null hypothesis that β = 0, following OLS regression, are given in Column 4 of Table 3. Default and heteroskedastic-robust standard errors lead to erroneously large *t*-statistics (of 0.0156 / 0.0037 = 4.22), so *p* = 0.000 and the null hypothesis is incorrectly rejected. Using various standard errors that control for clustering (Rows 3–6) leads to *p* ≃ 0.20 so that the null is not rejected. Rows 7–9 report *p*-values from several percentile-*t* bootstraps that again lead to rejection of *H*_{0} : β = 0.

For illustrative purposes, we also compute standard errors allowing for two-way clustering, see Section V, with clustering on both state and year. These are computed using the user-generated Stata add-on program cgmreg.ado. Clustering on year is necessary if both the regressor and the model errors are correlated across states in a given year. For this application, the result (s.e. = 0.01167) is very similar to that from clustering on state alone (s.e. = 0.01185). In some other panel applications, the two-way cluster robust standard errors can be substantially larger than those from clustering on state alone.

The main lesson from the Model 1 OLS results is that even after inclusion of state fixed effects one needs to use cluster-robust standard errors that cluster on state. The inclusion of state fixed effects did not eliminate the within-state correlation of the error. In this example, the correct cluster-robust standard errors are 3.6 times larger than the default.

Model 2 again uses OLS estimation but drops the state fixed effects from Equation 34. Dropping these fixed effects leads to much less precise estimation as the cluster-robust standard error (Row 3) increases from 0.0119 to 0.0226 and this cluster-robust standard error is now 0.0226 / 0.0062 = 3.6 times the default, compared to a ratio of 3.6 when state fixed effects were included. Note that inclusion of state fixed effects (Model 1) did soak up some of the within-state error correlation as expected, but there still remained substantial within-cluster correlation of the error so that cluster-robust standard errors need to be used.

For Model 2, the comparisons of the various standard errors and *p*-values are qualitatively similar to those for Model 1 so are not discussed further.

Model 3 estimates the same model as Model 1, except that the state and year fixed effects are directly estimated, and estimation is now by FGLS allowing for an AR(1) process for the errors. Because there are 36 years of data, the bias correction of Hansen (2007b)—see Section IIIC—will make little difference and is not used here. Estimation uses Stata command *xtreg, pa corr(ar 1)* after *xtset state*.

Comparing Rows 1 and 3, again, even with inclusion of state fixed effects, one should obtain standard errors that cluster on state, using *xtreg, pa* option *vce(robust))*. The difference is not as pronounced as for OLS, with FGLS cluster-robust standard error that is 0.0084/0.0062 = 1.4 times the default.

FGLS estimation has led to substantial gain in efficiency, with cluster-robust standard error (Row 3) for FGLS of 0.0084 compared to 0.0119 for OLS.

This example illustrates that, even with state fixed effects included in a state year panel, inference should be based on cluster-robust standard errors. Furthermore, there can be substantial efficiency gains to estimating by FGLS rather than OLS.

### D. State-Year Panel Data: Monte Carlo

We next perform a Monte Carlo exercise to investigate the performance of various cluster-robust methods as the number of clusters becomes small. The analysis is based on the same state-year panel regression as in the previous subsection, with each state-year observation based on log-wages after partialling out the individual level covariates.

In each simulation, we draw a random set of *G* states (with replacement), where *G* takes values 6, 10, 20, 30, and 50. When a state is drawn, we take all years of data for that state. We then assign our DiD “policy” variable to half the states with the policy turned on midway through the time period. In these simulations, we perform tests of the null hypothesis that the slope coefficient of the policy variable is zero. As in Table 2, for *G* = 6 and 10 we perform 4,000 simulations, so we expect that 95 percent of these simulations will yield estimated test size in the range (0.043, 0.057). For larger *G*, there are 1,000 simulations and the 95 percent simulation interval is instead (0.036, 0.064).

We begin with the last column of Table 4, with *G* = 50 states. All tests aside from that based on default standard errors (Row 1) have rejection rates that are not appreciably different from 0.05 once we allow for simulation error.

As the number of clusters decreases, it becomes clear that one should use the *T*(*G* – 1) or *T*(*G* – 2) distribution for critical values and even this leads to mild overrejection with low *G*. The pairs cluster percentile-*t* bootstrap fails with few clusters, with rejection rate of only 0.005 when *G* = 6. For low *G*, the wild cluster percentile-*t* bootstrap has similar results with either two-point or six-point weights with very slight overrejection.

## XI. Concluding Thoughts

It is important to aim for correct statistical inference, many empirical applications feature the potential for errors to be correlated within clusters, and we need to make sure our inference accounts for this. Often this is straightforward to do using traditional cluster-robust variance estimators—but sometimes things can be tricky. The leading difficulties are (1) determining how to define the clusters, and (2) dealing with few clusters; however, other complications can arise as well. When faced with these difficulties, there is no simple hard and fast rule regarding how to proceed. You need to think carefully about the potential for correlations in your model errors and how that interacts with correlations in your covariates. In this essay, we have aimed to present the current leading set of tools available to practitioners to deal with clustering issues.

## Footnotes

Colin Cameron is a professor in the Department of Economics at UC-Davis.

Doug Miller is an associate professor in the Department of Economics at UC-Davis. They thank four referees and the journal editor for very helpful comments and for guidance, participants at the 2013 California Econometrics Conference, a workshop sponsored by the U.K. Programme Evaluation for Policy Analysis, seminars at University of Southern California and at University of Uppsala, and the many people who over time have sent them cluster-related puzzles (the solutions to some of which appear in this paper). Doug Miller acknowledges financial support from the Center for Health and Wellbeing at the Woodrow Wilson School of Public Policy at Princeton University. The data used in this article can be obtained beginning November 2015 through October 2018 from the authors.

- Received March 2013.
- Accepted January 2014.