Multiple Predictor Variables in General Linear Models

https://etherpad.wikimedia.org/p/607-mlr

Data Generating Processes Until Now

• One predictor with one response
  
  
• Except…ANOVA had multiple possible treatment levels

The General Linear Model

\[\Large \boldsymbol{Y} = \boldsymbol{\beta X} + \boldsymbol{\epsilon} \]

• This equation is huge. X can be anything - categorical, continuous, squared, sine, etc.

• There can be straight additivity, or interactions

• So far, the only model we’ve used with >1 predictor is ANOVA

Models with Many Predictors

0. ANOVA

1. ANCOVA

2. Multiple Linear Regression

3. MLR with Interactions

Analysis of Covariance

\[\Large \boldsymbol{Y} = \boldsymbol{\beta X} + \boldsymbol{\epsilon}\]


\[Y = \beta_0 + \beta_{1}x + \sum_{j}^{i=1}\beta_j + \epsilon\]

Analysis of Covariance

\[Y = \beta_0 + \beta_{1}x + \sum_{j}^{i=1}\beta_j + \epsilon\]

• ANOVA + a continuous predictor
  
  
• Often used to correct for a gradient or some continuous variable affecting outcome
  
  
• OR used to correct a regression due to additional groups that may throw off slope estimates
  - e.g. Simpson’s Paradox: A positive relationship between test scores and academic performance can be masked by gender differences

Neanderthals and ANCOVA

Who had a bigger brain: Neanderthals or us?

The Means Look the Same…

But there appears to be a Relationship Between Body and Brain Mass

And Mean Body Mass is Different

Analysis of Covariance (control for a covariate)

ANCOVA: Evaluate a categorical effect(s), controlling for a covariate (parallel lines)
Groups modify the intercept.

The Steps of Statistical Modeling

1. What is your question?
2. What model of the world matches your question?
3. Build a test
4. Evaluate test assumptions
5. Evaluate test results
6. Visualize

Assumptions of Multiway Anova

• Independence of data points

• Normality and homoscedacticity within groups (of residuals)

• No relationship between fitted and residual values

• Additivity of Treatment and Covariate (Parallel Slopes)

The Usual Suspects of Assumptions

Group Residuals

Test for Parallel Slopes

Ye Olde F-Test

What type of Sums of Squares?

II!

Vsualizing Result

Parcelling Out Covariate

Parcelling Out Covariate with PostHocs & Least Square Menas

 contrast             estimate     SE df t.ratio p.value
 neanderthal - recent  -0.0703 0.0282 36   -2.49  0.0175

Evluated at mean of covariate

Models with Many Predictors

0. ANOVA

1. ANCOVA

2. Multiple Linear Regression

3. MLR with Interactions

One-Way ANOVA Graphically

Multiple Linear Regression?

Multiple Linear Regression

Calculating Multiple Regression Coefficients with OLS

\[\boldsymbol{Y} = \boldsymbol{b X} + \boldsymbol{\epsilon}\]

What causes species richness?

• Distance from fire patch
• Elevation
• Abiotic index
• Patch age
• Patch heterogeneity
• Severity of last fire
• Plant cover

Many Things may Influence Species Richness

Our Model

klm <- lm(rich ~ cover + firesev + hetero, data=keeley)

Testing Assumptions

• Data Generating Process: Linearity
  
  
• Error Generating Process: Normality & homoscedasticity of residuals
  
  
• Data: Outliers not influencing residuals
  
  
• Predictors: Minimal multicollinearity

Checking for Multicollinearity: Correlation Matrices

           cover   firesev    hetero
cover    1.00000 -0.437135 -0.168378
firesev -0.43713  1.000000 -0.052355
hetero  -0.16838 -0.052355  1.000000

Checking for Multicollinearity: Variance Inflation Factor

• Consider \(y = \beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \epsilon\)
  
  
• And \(X_{1} = \alpha_{0} + \alpha_{2}x_{2} + \epsilon_j\)
  
  
• \(var(\beta_{1}) = \frac{\sigma^2}{(n-1)\sigma^2_{X_1}}\frac{1}{1-R^{2}_1}\)
  
  \[VIF = \frac{1}{1-R^2_{1}}\]

Checking for Multicollinearity: Variance Inflation Factor

\[VIF_1 = \frac{1}{1-R^2_{1}}\]

  cover firesev  hetero 
 1.2949  1.2617  1.0504 

Other Diagnostics as Usual!

Examine Residuals With Respect to Each Predictor

Which Variables Explained Variation: Type II Marginal SS

If order of entry matters, can use type I. Remember, what models are you comparing?

What’s Going On: Type I and II Sums of Squares

- Type II more conservative for A

The coefficients

R² = 0.40986

Comparing Coefficients on the Same Scale

\[r_{xy} = b_{xy}\frac{sd_{x}}{sd_{y}}\]

   cover  firesev   hetero 
 0.32673 -0.19872  0.50160 

Visualization of 4-D Multivariate Models is Difficult!

Component-Residual Plots Aid in Visualization

Takes effect of predictor + residual of response

Added Variable Plot to Show Unique Contributions when Holding Others Constant

Or Show Regression at Median of Other Variables

Or Show Predictions Overlaid on Data

Models with Many Predictors

0. ANOVA

1. ANCOVA

2. Multiple Linear Regression

3. MLR with Interactions

Problem: What if Continuous Predictors are Not Additive?

Problem: What if Continuous Predictors are Not Additive?

Problem: What if Continuous Predictors are Not Additive?

Model For Age Interacting with Elevation to Influence Fire Severity

\[y = \beta_0 + \beta_{1}x_{1} + \beta_{2}x_{2}+ \beta_{3}x_{1}x_{2}\]

keeley_lm_int <- lm(firesev ~ age*elev, data=keeley)

Testing Assumptions

• Data Generating Process: Linearity
  
  
• Error Generating Process: Normality & homoscedasticity of residuals
  
  
• Data: Outliers not influencing residuals
  
  
• Predictors: Minimal collinearity

Other Diagnostics as Usual!

Examine Residuals With Respect to Each Predictor

Interactions, VIF, and Centering

     age     elev age:elev 
  3.2001   5.5175   8.2871 

Interpretation of Centered Coefficients

\[\huge X_i - \bar{X}\]

• Additive coefficients are the effect of a predictor at the mean value of the other predictors
  
  
• Intercepts are at the mean value of all predictors
  
  
• Visualization will keep you from getting confused!
  

Interactions, VIF, and Centering

\[y = \beta_0 + \beta_{1}(x_{1}-\bar{x_{1}}) + \beta_{2}(x_{2}-\bar{x_{2}})+ \beta_{3}(x_{1}-\bar{x_{1}})(x_{2}-\bar{x_{2}})\]

Variance Inflation Factors for Centered Model:

       age_c       elev_c age_c:elev_c 
      1.0167       1.0418       1.0379 

F-Tests to Evaluate Model

What type of Sums of Squares??

Coefficients!

R² = 0.32352

Note that additive coefficients signify the effect of one predictor in the abscence of all others.

Centered Coefficients!

R² = 0.32352

Note that additive coefficients signify the effect of one predictor at the average level of all others.

Interpretation

• What the heck does an interaction effect mean?
  
  
• We can look at the effect of one variable at different levels of the other
  
  
• We can look at a surface
  
  
• We can construct counterfactual plots showing how changing both variables influences our outcome

Age at Different Levels of Elevation

Elevation at Different Levels of Age

Surfaces and Other 3d Objects

Or all in one plot

Without Data and Including CIs

A Heatmap Approach

We can do a lot with the general linear model!
You are only limited by the biological models you can imagine.