Putting Linear Regression Into Practice with Pufferfish

Question of the day: Does Resembling a Pufferfish Reduce Predator Visits?

Digging Deeper into Regression

1. Assumptions: Is our fit valid?

2. How did we fit this model?

3. How do we draw inference from this model?

You are now a Statistical Wizard. Be Careful. Your Model is a Golem.

(sensu Richard McElreath)

A Case of "Great" versus "Not as Great" Fits...

The Two Fits

Assumptions (in rough descending order of importance)

1. Validity

2. Representativeness

3. Model captures features in the data

4. Additivity and Linearity

5. Independence of Errors

6. Equal Variance of Errors

7. Normality of Errors

8. Minimal Outlier Influence

Validity: Do X and Y Reflect Concepts I'm interested In

What if predator approaches is not a good measure of recognition? Or mimics just don't look like fish?

Representativeness: Does Your Data Represent the Population?

For example, say this is your result...

Representativeness: Does Your Data Represent the Population?

What if you are looking at only a piece of the variation in X in your population?

Representativeness: Does Your Data Represent the Population?

How should you have sampled this population for a representative result?

Representativeness: Does Your Data Represent the Population?

It's better to have more variation in X than just a bigger N

Representativeness: Does Your Data Represent the Population?

• Always question if you did a good job sampling

• Use natural history and the literature to get the bounds of values

• If experimenting, make sure your treatment levels are representative

• If you realize post-hoc they are not, qualify your conclusions

Model captures features in the data

Does the model seem to fit the data? Are there any deviations? Can be hard to see...

Simulating implications from the model to see if we match features in the data

Is anything off?

But what to wolves say to you?

Additivity and Linearity: Should account for all of the variation between residual and fitted values - what you want

Additivity and Linearity: Wolf Problems?

Solutions: Nonlinear transformations or a better model!

Independence of Errors

• Are all replicates TRULY independent

• Did they come from the same space, time, etc.

• Non-independence can introduce BIAS

  • SEs too small (at the least)
  • Causal inference invalid

• Incoporate Non-independence into models (many methods)

Equal Variance of Errors: No Pattern to Residuals and Fitted Values

Equal Variance of Errors: What is up with intermediate Wolf Values

Equal Variance of Errors: Problems and Solutions

• Shapes (cones, footballs, etc.) with no bias in fitted v. residual relationship

• A linear relationship indicates an additivity problem

• Can solve with a better model (more predictors)

• Can solve with weighting by X values, if source of heteroskedasticity known

  • This actually means we model the variance as a function of X
  • $\epsilon_i \sim(N, f(x_i))$

• Minor problem for coefficient estimates

• Major problem for doing inference and prediction as it changes error

Normality of errors: Did we fit the error generating process that we observed?

• We assumed $\epsilon_i \sim N(0,\sigma)$ - but is that right?

• Can assess with a QQ-plot

  • Do quantiles of the residuals match quantiles of a normal distribution?

• Again, minor problem for coefficient estimates

• Major problem for doing inference and prediction, as it changes error

Equal Variance of Errors: Puffers

Equal Variance of Errors: Wolves underpredict at High Levels

    Shapiro-Wilk normality test
data:  residuals(wolf_mod)
W = 0.9067, p-value = 0.02992

Outliers: Cook's D

Want no values > 1

Outliers: Cook's D - wolves OK

Everyone worries about outliers, but...

• Are they real?

• Do they indicate a problem or a nonlinearity?

• Remove only as a dead last resort

• If from a nonlinearity, consider transformation

Assumptions (in rough descending order of importance)

1. Validity: only you know!

2. Representativeness: look at nature

3. Model captures features in the data: compare model v. data!

4. Additivity and Linearity: compare model v. data!

5. Independence of Errors: conisder sampling design

6. Equal Variance of Errors: evaluate res-fit

7. Normality of Errors: evaluate qq and levene test

8. Minimal Outlier Influence: evaluate Cook's D

Digging Deeper into Regression

1. Assumptions: Is our fit valid?

2. How did we fit this model?

3. How do we draw inference from this model?

So, uh.... How would you fit a line here?

Lots of Possible Lines - How would you decide?

Method of Model Fitting

1. Least Squares

   • Conceptually Simple
   • Minimizes distance between fit and residuals
   • Approximations of quantities based on frequentist logic

2. Likelihood

   • Flexible to many error distributions and other problems
   • Produces likelihood surface of different parameter values
   • Equivalent to least square for Gaussian likelihood
   • Approximations of quantities based on frequentist logic

3. Bayesian

   • Incorporates prior knowledge
   • Probability for any parameter is likelihood * prior
   • Superior for quantifying uncertainty
   • With "flat" priors, equivalent to least squares/likelihood
   • Analytic or simulated calculation of quantities

Basic Principles of Least Squares Regression

$\widehat{Y} = \beta_0 + \beta_1 X + \epsilon$ where $\beta_0$ = intercept, $\beta_1$ = slope

Minimize Residuals defined as $SS_{residuals} = \sum(Y_{i} - \widehat{Y})^2$

Analytic Solution: Solving for Slope

$\LARGE b=\frac{s_{xy}}{s_{x}^2}$ $= \frac{cov(x,y)}{var(x)}$

$\LARGE = r_{xy}\frac{s_{y}}{s_{x}}$

Analytic Solution: Solving for Intercept

Least squares regression line always goes through the mean of X and Y

$\Large \bar{Y} = \beta_0 + \beta_1 \bar{X}$

$\Large \beta_0 = \bar{Y} - \beta_1 \bar{X}$

Digging Deeper into Regression

1. Assumptions: Is our fit valid?

2. How did we fit this model?

3. How do we draw inference from this model?

Inductive v. Deductive Reasoning

Deductive Inference: A larger theory is used to devise many small tests.

Inductive Inference: Small pieces of evidence are used to shape a larger theory and degree of belief.

Applying Different Styles of Inference

• Null Hypothesis Testing: What's the probability that things are not influencing our data?

  • Deductive

• Cross-Validation: How good are you at predicting new data?

  • Deductive

• Model Comparison: Comparison of alternate hypotheses

  • Deductive or Inductive

• Probabilistic Inference: What's our degree of belief in a data?

  • Inductive

Null Hypothesis Testing is a Form of Deductive Inference

A Bigger View of Dedictive Inference

https://plato.stanford.edu/entries/lakatos/#ImprPoppScie

Reifying Refutation - What is the probability something is false?

What if our hypothesis was that the resemblance-predator relationship was 2:1. We know our SE of our estimate is 0.57, so, we have a distribution of what we could observe.

Reifying Refutation - What is the probability something is false?

BUT - our estimated slope is 3.

To falsify the 2:1 hypothesis, we need to know the probability of observing 3, or something GREATER than 3.

We want to know if we did this experiment again and again, what's the probability of observing what we saw or worse (frequentist!)

Probability = 0.04

R.A. Fisher and The P-Value For Null Hypotheses

Applying Fisher: Evaluation of a Test Statistic

We use our data to calculate a test statistic that maps to a value of the null distribution.

We can then calculate the probability of observing our data, or of observing data even more extreme, given that the null hypothesis is true.

$$\large P(X \leq Data | H_{0})$$

Problems with P

• Most people don't understand it.

  • See American Statistical Society' recent statements

• Like SE, it gets smaller with sample size!- Neyman-Pearson Null Hypothesis Significance Testing

  • For Industrial Quality Control, NHST was introduced to establish cutoffs of reasonable p, called an $\alpha$
  • This corresponds to Confidence intervals - 1-$\alpha$ = CI of interest
  • This has become weaponized so that $\alpha = 0.05$ has become a norm.... and often determines if something is worthy of being published?
  • Chilling effect on science

• We don't know how to talk about it

How do you talk about results from a p-value?

• Based on your experimental design, what is a reasonable range of p-values to expect if the null is false

• Smaller p values indicate stronger support for rejection, larger ones weaker. Use that language.

• Accumulate multiple lines of evidence so that the entire edifice of your research does not rest on a single p-value!!!!

For example, what does p = 0.061 mean?

• There is a 6.1% chance of obtaining the observed data or more extreme data given that the null hypothesis is true.

• If you choose to reject the null, you have a ~ 1 in 16 chance of being wrong

• Are you comfortable with that?

• OR - What other evidence would you need to make you more or less comfortable?

Common Regression Test Statistics

• Does my model explain variability in the data?

  • Null Hypothesis: The ratio of variability from your predictors versus noise is 1
  • Test Statistic: F distribution (describes ratio of two variances)

• Are my coefficients not 0?

  • Null Hypothesis: Coefficients are 0
  • Test Statistic: T distribution (normal distribution modified for low sample size)

Does my model explain variability in the data?

Ho = The model predicts no variation in the data.

Ha = The model predicts variation in the data.

To evaluate these hypotheses, we need to have a measure of variation explained by data versus error - the sums of squares! $$SS_{Total} = SS_{Regression} + SS_{Error}$$

Sums of Squares of Error, Visually

Sums of Squares of Regression, Visually

Sums of Squares of Regression, Visually

Distance from $\hat{y}$ to $\bar{y}$

Components of the Total Sums of Squares

$SS_{R} = \sum(\hat{Y_{i}} - \bar{Y})^{2}$, df=1

$SS_{E} = \sum(Y_{i} - \hat{Y}_{i})^2$, df=n-2

To compare them, we need to correct for different DF. This is the Mean Square.

MS=SS/DF

e.g, $MS_{E} = \frac{SS_{E}}{n-2}$

The F Distribution and Ratios of Variances

$F = \frac{MS_{R}}{MS_{E}}$ with DF=1,n-2

F-Test and Pufferfish

We reject the null hypothesis that resemblance does not explain variability in predator approaches

Testing the Coefficients

• F-Tests evaluate whether elements of the model contribute to variability in the data
  • Are modeled predictors just noise?
  • What's the difference between a model with only an intercept and an intercept and slope?

• T-tests evaluate whether coefficients are different from 0

• Often, F and T agree - but not always

  • T can be more sensitive with multiple predictors

T-Distributions are What You'd Expect Sampling a Standard Normal Population with a Small Sample Size

• t = mean/SE, DF = n-1
• It assumes a normal population with mean of 0 and SD of 1

Error in the Slope Estimate

$\Large SE_{b} = \sqrt{\frac{MS_{E}}{SS_{X}}}$

95% CI = $b \pm t_{\alpha,df}SE_{b}$

(~ 1.96 when N is large)

Assessing the Slope with a T-Test

$$\Large t_{b} = \frac{b - \beta_{0}}{SE_{b}}$$

DF=n-2

$H_0: \beta_{0} = 0$, but we can test other hypotheses

Slope of Puffer Relationship (DF = 1 for Parameter Tests)

We reject the hypothesis of no slope for resemblance, but fail to reject it for the intercept.

So, what can we say?