So, We've Done this Linear Deliciousness

$$y_i = \beta_0 + \beta_1 x_i + \epsilon_i$$ $$\epsilon_i \sim \mathcal{N}(0, \sigma)$$

And We've Seen Many Predictors

$$y_i = \beta_0 + \sum^K_{j=1}\beta_j x_{ij} + \epsilon_i$$

What if our X Variable Is Categorical?

Comparing Two Means

$$mass_i = \beta_0 + \beta_1 sex_i + \epsilon_i$$

What if it Has Many Levels

Comparing Many Means

$$mass_i = \beta_1 adelie_i + \beta_2 chinstrap_i + \beta_3 gentoo_i + \epsilon_i$$

Different Types of Categories?

Comparing Many Means in Many categories

$$mass_i = \beta_1 adelie_i + \beta_2 chinstrap_i + \beta_3 gentoo_i + \beta_4 male_i + \epsilon_i$$

Dummy Codding for Dummy Models

1. The Categorical as Continuous

2. Many Levels of One Category

3. Interpretation of Categorical Results.

4. Querying Your Model to Compare Groups

We Know the Linear Model

But, what if $x_i$ was just 0,1?

Consider Comparing Two Means

Consider the Horned Lizard

Horns prevent these lizards from being eaten by birds. Are horn lengths different between living and dead lizards, indicating selection pressure?

The Data

Looking at Means and SE

What is Really Going On?

What if we think of Dead = 0, Living = 1

Let's look at it a different way

First, Recode the Data with Dummy Variables

First, Recode the Data with Dummy Variables

But with an Intercept, we don't need Two Dummy Variables

This is known as a Treatment Contrast structure

This is Just a Linear Regression

$$Length_i = \beta_0 + \beta_1 Status_i + \epsilon_i$$

You're Not a Dummy, Even If You Code a Dummy Variable

$$Length_i = \beta_0 + \beta_1 Status_i + \epsilon_i$$

• Setting $Status_i$ to 0 or 1 (Dead or Living) is called Dummy Coding
  • Or One Hot Encoding in the ML world.

• We can always turn groups into "Dummy" 0 or 1

• We could even fit a model with no $\beta_0$ and code Dead = 0 or 1 and Living = 0 or 1

• This approach works for any unordered categorical (nominal) variable

Homogeneity of Variance Important for CI Estimation

Dummy Codding for Dummy Models

1. The Categorical as Continuous.

2. Many Levels of One Category

3. Interpretation of Categorical Results.

4. Querying Your Model to Compare Groups

Categorical Predictors: Gene Expression and Mental Disorders

The data

Traditional Way to Think About Categories

What is the variance between groups v. within groups?

If We Only Had Control v. Bipolar...

If We Only Had Control v. Bipolar...

Underlying linear model with control = intercept, dummy variable for bipolar

If We Only Had Control v. Bipolar...

Underlying linear model with control = intercept, dummy variable for bipolar

If We Only Had Control v. Schizo...

Underlying linear model with control = intercept, dummy variable for schizo

If We Only Had Control v. Schizo...

Underlying linear model with control = intercept, dummy variable for schizo

Linear Dummy Variable (Fixed Effect) Model

$$\large y_{ij} = \beta_{0} + \sum \beta_{j}x_{ij} + \epsilon_{ij}, \qquad x_{i} = 0,1$$

• i = replicate, j = group

• $x_{ij}$ inidicates presence/abscence (1/0) of level j for individual i

  • A Dummy variable

• This is the multiple predictor extension of a two-category model

• All categories are orthogonal

• One category set to $\beta_{0}$ for ease of fitting, and other $\beta$s are different from it

A Simpler Way to Write: The Means Model

$$\large y_{ij} = \alpha_{j} + \epsilon_{ij}$$
$$\epsilon_{ij} \sim N(0, \sigma^{2} )$$

• i = replicate, j = group

• Different mean for each group

• Focus is on specificity of a categorical predictor

Partioning Model to See What Varies

$$\large y_{ij} = \bar{y} + (\bar{y}_{j} - \bar{y}) + ({y}_{ij} - \bar{y}_{j})$$

• i = replicate, j = group

• Shows partitioning of variation
  • Between group v. within group variation

• Consider $\bar{y}$ an intercept, deviations from intercept by treatment, and residuals

• Can Calculate this with a fit model to answer questions - it's a relic of a bygone era

  • That bygone era has some good papers, so, you should recognize this

Let's Fit that Model

Using Least Squares

brain_lm <- lm(PLP1.expression ~ group, data=brainGene)
tidy(brain_lm) |> 
  select(-c(4:5)) |>
  knitr::kable(digits = 3) |>
  kableExtra::kable_styling()

Dummy Codding for Dummy Models

1. The Categorical as Continuous

2. Many Levels of One Category

3. Interpretation of Categorical Results

4. Querying Your Model to Compare Groups

R Fits with Treatment Contrasts

$$y_{ij} = \beta_{0} + \sum \beta_{j}x_{ij} + \epsilon_{ij}$$

What does this mean?

• Intercept ( $\beta_{0}$ ) = the average value associated with being in the control group

• Others = the average difference between control and each other group

• Note: Order is alphabetical

Actual Group Means

$$y_{ij} = \alpha_{j} + \epsilon_{ij}$$

What does this mean?

Being in group j is associated with an average outcome of y.

What's the best way to see this?

Many Ways to Visualize

Many Ways to Visualize

Many Ways to Visualize

How Well Do Groups Explain Variation in Response Data?

We can look at fit to data - even in categorical data!

# R2 for Linear Regression
       R2: 0.271
  adj. R2: 0.237

But, remember, this is based on the sample at hand.

Adjusted R²: adjusts for sample size and model complexity (k = # params = # groups)

$$R^2_{adj} = 1 - \frac{(1-R^2)(n-1)}{n-k-1}$$

Dummy Codding for Dummy Models

1. The Categorical as Continuous

2. Many Levels of One Category

3. Interpretation of Categorical Results.

4. Querying Your Model to Compare Groups

Which groups are different from each other?

Many mini-linear models with two means....multiple comparisons!

Post-Hoc Means Comparisons: Which groups are different from one another?

• Each group has a mean and SE

• We can calculate a comparison for each

• BUT, we lose precision as we keep resampling the model

• Remember, for every time we look at a system, we have some % of our CI not overlapping the true value

• Each time we compare means, we have a chance of our CI not covering the true value

• To minimize this possibility, we correct (widen) our CIs for this Family-Wise Error Rate

Solutions to Multiple Comparisons and Family-wise Error Rate?

1. Ignore it -

   • Just a bunch of independent linear models

2. Increase your CI given m = # of comparisons

   • If 1 - CI of interest = $\alpha$
   • Bonferroni Correction $\alpha/ = \alpha/m$
   • False Discovery Rate $\alpha/ = k\alpha/m$ where k is rank of test

3. Other multiple comparison corrections

   • Tukey's Honestly Significant Difference
   • Dunnett's Compare to Control

No Correction: Least Square Differences

Bonferroni Corrections

Tukey's Honestly Significant Difference

Visualizing Comparisons (Tukey)

Dunnett's Comparison to Controls

So, Categorical Models...

• At the end of the day, they are just another linear model

• We can understand a lot about groups, though

• We can begin to see the value of queries/counterfactuals

$$\widehat {\textbf{Y} } = \textbf{X}\beta$$ $$\textbf{Y} \sim \mathcal{N}(\widehat {\textbf{Y} }, \Sigma)$$