ITT_CATE_snippets.Rmd

## Research designs to aid the creation of interpretable comparisons

   - Randomized experiments (randomized $Z$ changes $y$) <!-- (more precision from reducing heterogeneity in $Y$) -->
     - No systematic differences between groups
     - Known reference distribution for tests of the sharp null and, in large
       samples, the weak null.
   - Instrumental variables (randomized $Z$ changes $D$ which changes $y$)
   - Natural Experiments / Discontinuities (one $X$ creates $Z$ which \ldots)
   - Semi-/Non-parametric Covariance Adjustment ($Z$ out of our control, but we can choose how to organize and observe given $X$) (matching and difference-in-differences)
   - Parametric covariance adjustment (same as above, but with a model for statistical adjustment, i.e. "controlling for")


# Now: Instruments, the ITT and CACE/LATE

## Instruments, the ITT and CACE/LATE

**Context:** An NGO invites you to help design and evaluate a program to increase donations involving in-person, door-to-door, discussions about the canvasser's own personal experiences. Building rapport with another person should increase willingness to donate, according to the NGO who had just read @broockman2016durably.

\smallskip

**Opportunity for learning about theory** You know have been interested in
understanding why and how social norms influence behavior. So, you ask to
that canvassers appeal to a descriptive social norm ("other people we have
spoken with agree") toward the end of the discussion.

\smallskip

**Design:** The NGO randomizes visits by canvassers to a set of addresses and
measures the amount pledged online in the following month.

\smallskip

**Analysis:** The NGO cares about the cost effectiveness of the approach. Did
it work? You (and the NGO should care) about **why** it worked (or did not
work).


## Defining causal effects

 - $Z_i$ is random assignment to a visit ($Z_i=1$) or not ($Z_i=0$).
 - $d_{i,Z_i=1}=1$ means that person $i$ opened the door to have a conversation when assigned a visit. Opening the door is an outcome of the treatment, is caused by the treatment.
 - $y_{i,Z_i = 1, d_{i,Z_i=1}=1}$ is the potential outcome for people who were assigned a visit and who opened the door.
 - $y_{i,Z_i = 1, d_{i,Z_i=1}=0}$ is the potential outcome for people who were assigned a visit and who did not open the door.
 - $y_{i,Z_i = 0, d_{i,Z_i=0}=1}$ is the potential outcome for people who were not assigned a visit and who opened the door.
 - $y_{i,Z_i = 0, d_{i,Z_i=0}=0}$ is the potential outcome for people who were not assigned a visit and who would not have opened the door.
 - We could also write $y_{i,Z_i = 0, d_{i,Z_i=1}=1}$ for people who were not
   assigned a visit who would have opened the door had they been assigned a
   visit etc..

In this case, $y_{i,Z_i = 0, d_{i,Z_i=1}=1} = y_{i,Z_i = 0, d_{i,Z_i=1}=0}
\equiv y_{i,Z_i=0}$ because you can't open the door unless visited.

And also $y_{i,Z_i = 1, d_{i,Z_i=1}=0} = y_{i,Z_i=0}$ because there can be no
effect of the visit if you don't open the door.

##  Defining our causal effects

People refer to an "Intent To Treat Effect" (ITT) as the (average) causal effect of
**assignment** to treatment. In our case, we can write the ITT as a weighted
sum of the responses to treatment  minus the response to assignment to control
( where $p(d_1=1)$ is the proportion of people opening their doors ).


$$ITT= ( \bar{y}_{Z=1,d_1=1} \cdot p(d_1=1) + \bar{y}_{Z=1,d_1=0} \cdot (1-p(d_1=1)) )  - \bar{y}_{Z=0}$$

\medskip

Notice that if everyone opens the door when visited then $y_{i,Z=1,d_1=1} =
y_{i,Z=1}$  so the ITT=ATE.

\medskip

Why give it a new name?

## A simulation to understand the design and analysis

Setting up the simulation and analysis before going into the field.

```{r}
library(randomizr)
set.seed(20200730)
n <- 1000
## Make a covariate, like age
xtmp <- rnorm(n, mean = 45, sd = 10)
x <- pmax(xtmp, 18)
## Make potential outcome to control
y0tmp <- round(rnorm(n, mean = 0, sd = 10)) + x / 2
y0 <- pmax(y0tmp, 0)
Z <- complete_ra(n)
## Door opening is a function of x (age) and Z (visits)
probd <- (x - min(x)) / (max(x) - min(x))
mean(probd)
d <- ifelse(Z == 1, rbinom(sum(Z == 1), prob = probd / 2, size = 1), 0)
table(Z, d, exclude = c())
mean(d[Z == 1])
```

```{r}
## No effects possible if not visited
y0d0 <- y0
y0d1 <- y0
y1d0 <- y0
taubar <- .25 * sd(y0) ## Effect of .25 sds on compliers/door openers
## The treatment also changes variance of y1d1
y1d1 <- mean(y0) + taubar + (y0 - mean(y0)) / 2
## Define observed outcome
Y <- Z * d * y1d1 + Z * (1 - d) * y1d0 + (1 - Z) * d * y0d1 + (1 - Z) * (1 - d) * y0d0
dat <- data.frame(y1d1, y1d0, y0d1, y0d0, y0, Y, Z, d, x, probd)
dat$y1 <- (1 - Z) * y0 + Z * d * y1d1 + Z * (1 - d) * y1d0
```

## A simulation to understand the design and analysis


```{r}
boxplot(Y ~ Z * d, data = dat)
```


## Learning about the ITT

First, let's learn about the effect of the policy itself, about the ITT.

```{r}
## True ITT with one-sided non-compliance
pd <- mean(dat$d[dat$Z == 1])

bary1 <- mean(dat$y1d1) * pd + mean(dat$y1d0) * (1 - pd)
bary0 <- mean(dat$y0)

trueITT <- bary1 - bary0
```


\smallskip

What evidence do we have against the claim that the policy had no effects? How
would we assess whether this testing procedure is trustworthy --- controls its
false positive error rate?

\smallskip

What is our best guess about the ITT itself? How would we assess whether or
not this estimator is biased or not?

\smallskip

What is our best guess about how our estimate of the ITT would vary from
experiment to experiment?


## Some code notes

```{r, eval=FALSE}
newExp <- function(trt) {
  sample(trt)
}
testStat <- function(outcome, trt) {
  mean(outcome[trt == 1]) - mean(outcome[trt == 0])
}
obsTestStat <- testStat(outcome = dat$Y, trt = dat$Z)
nullDist <- replicate(1000, testStat(outcome = dat$Y, trt = newExp(dat$Z)))
upperP <- mean(nullDist >= obsTestStat)
```


```{r, eval=FALSE}
## taubarhat <- with(dat, mean(Y[Z==1]) - mean(Y[Z==0]))
## or taubarhat <- coef(lm(Y~Z))[["Z"]]
library(estimatr)
taubarhat <- difference_in_means(Y ~ Z, data = dat)
## sqrt( var(Y[Z==1])/sum(Z==1) + var(Y[Z==0])/sum(Z==0) )
```


## The Complier Average Causal Effect or Local Average Treatment Effect (on the Compliers)

Now we would like to learn about the causal effect of answering the door and having the
conversation, the theoretically interesting effect. But, this comparison is confounded
by $x$ (tells us about differences in the outcome due to $x$ in addition to
the difference caused by $D$).

```{r}
with(dat, cor(Y, x))
with(dat, cor(d, x))
with(dat, cor(Z, x)) ## should be near 0
```


## Identifying the causal effect of opening the door

We want to learn about the Complier Average Causal Effect (aka the Local
Average Treatment Effect): CACE$= \bar{y_{i,Z=1,d_1=1}} -
\bar{y_{i,Z=0,d_1=1}}$


Let $p$ be the proportion of compliers ($\sum_{i=1}^N I(d_{i,1} = 1)/N$).

\begin{align*}
ITT & = \left( \bar{y}_{1,d_1=1}p + \bar{y}_{1,d_1=0}(1-p) \right)  - \bar{y}_{0} \\
    & = ( \bar{y}_{1,d_1=1}p + \bar{y}_{1,d_1=0}(1-p) )  - ( \bar{y}_{0,d_0=1}p+ \bar{y}_{0,d_0=0}(1-p) )  \\
    & = ( \bar{y}_{1,d_1=1}p - \bar{y}_{0,d_0=1}p )  +  ( \bar{y}_{1,d_1=0}(1-p)  - \bar{y}_{0,d_0=0}(1-p) )  \\
    & = ( \bar{y}_{1,d_1=1} - \bar{y}_{0,d_0=1} )p  +  ( \bar{y}_{1,d_1=0}  - \bar{y}_{0,d_0=0}) (1-p)  \\
\intertext{but $\bar{y}_{1,d_1=0} = \bar{y}_{0,d_1=0} = \bar{y}_{0,d_0=0}$ so }
  ITT  & = ( \bar{y}_{1,d_1=1} - \bar{y}_{0,d_0=1} ) p \\
  ITT  & = ( CACE ) p \\
\intertext{so we can write the CACE as ITT/p}
  ITT/p & = CACE
\end{align*}

This means that we can write the CACE as $CACE = ITT/\text{proportion of compliers}$

## Code snippets

```{r}
ITT <- with(dat, mean(Y[Z == 1]) - mean(Y[Z == 0]))
pd <- with(dat, mean(d[Z == 1]) - mean(d[Z == 0]))
ITT / pd

library(AER)
iv1 <- ivreg(Y ~ d | Z, data = dat)
```



##  Defining our causal effects: The Intent to Treat Effect

People refer to an "Intent To Treat Effect" (ITT) as the (average) causal
effect of **assignment** to treatment. In our case, we can write the ITT as a
weighted sum of the responses to treatment  minus the response to assignment to
control ( where $p(d_1=1)$ is the proportion of people opening their doors ).
We simplify the left hand term because we assume that you will act the same way
when you are not assigned to a visit --- no visit has the same outcome as
no visit plus an open door, for example.


\begin{equation}
ITT= \left( \bar{y}_{Z=1,d_1=1} \cdot p(d_1=1)  +  \bar{y}_{Z=1,d_1=0} \cdot ( 1-p(d_1=1) ) \right)  - \bar{y}_{Z=0}
\end{equation}

\medskip

Notice that if everyone opens the door when visited then $y_{i,Z=1,d_1=1} =
y_{i,Z=1}$  so the ITT=ATE.

\begin{eqnarray}
ITT &= \left( \bar{y}_{Z=1} ( p(d_1=1)  +  ( 1-p(d_1=1) ) \right)  - \bar{y}_{Z=0} \\
    & = \left( \bar{y}_{Z=1} ( 1 ) \right)  - \bar{y}_{Z=0}
\end{eqnarray}


\medskip

Why give this causal effect a new name? The importance of thinking about the nature of the causal effect.


## Identifying the causal effect of opening the door

We want to learn about the Complier Average Causal Effect (aka the Local
Average Treatment Effect): CACE$= \widebar{y_{i,Z=1,d_1=1}} -
\widebar{y_{i,Z=0,d_1=1}}$

Let $p$ be the proportion of compliers ($\sum_{i=1}^N I(d_{i,1} = 1)/N$).

\begin{align*}
ITT & = \left( \bar{y}_{1,d_1=1}p + \bar{y}_{1,d_1=0}(1-p) \right)  - \bar{y}_{0} \\
\intertext{now expand $\bar{y}_{0}$ into its parts depending on compliance type}
    & = ( \bar{y}_{1,d_1=1}p + \bar{y}_{1,d_1=0}(1-p) )  - ( \bar{y}_{0,d_0=1}p+ \bar{y}_{0,d_0=0}(1-p) )  \\
    \intertext{and collect the compliers together and simplify}
    & = ( \bar{y}_{1,d_1=1}p - \bar{y}_{0,d_0=1}p )  +  ( \bar{y}_{1,d_1=0}(1-p)  - \bar{y}_{0,d_0=0}(1-p) )  \\
    & = ( \bar{y}_{1,d_1=1} - \bar{y}_{0,d_0=1} )p  +  ( \bar{y}_{1,d_1=0}  - \bar{y}_{0,d_0=0}) (1-p)  \\
\intertext{but all forms of not talking with a visitor are the same outcome $\bar{y}_{1,d_1=0} = \bar{y}_{0,d_1=0} = \bar{y}_{0,d_0=0}$ so }
  ITT  & = ( \bar{y}_{1,d_1=1} - \bar{y}_{0,d_0=1} ) p \\
  ITT  & = ( CACE ) p \\
\intertext{so we can write the CACE as ITT/p}
  ITT/p & = CACE
\end{align*}

## Identifying the causal effect of opening the door

We want to learn about the Complier Average Causal Effect (aka the Local
Average Treatment Effect): CACE$= \widebar{y_{i,Z=1,d_1=1}} -
\widebar{y_{i,Z=0,d_1=1}}$

\begin{align*}
  ITT  & = ( \bar{y}_{1,d_1=1} - \bar{y}_{0,d_0=1} ) p \\
  ITT  & = ( CACE ) p \\
\intertext{so we can write the CACE as ITT/p}
  ITT/p & = CACE
\end{align*}

This means that we can write the CACE as $CACE = ITT/\text{proportion of compliers}$