Leverage and Influence in Univariate Regression
The interactive tool in the Leverage/Influence tab illustrates the
concepts of leverage and influence in the univariate regression model
yi = β0 + β1xi + εi,
i = 1, 2, …, n. A yi value that seems quite
inconsistent with the line fitting through the other data points is called an outlier and
can have a lot of influence on the regression line, especially on
the slope of the line. This means that the slope estimate changes dramatically whether we
include or exclude the pair (xi, yi). A closely
related concept is leverage, which measures the
potential influence of an outlying y-value for the pair
(xi, yi). If xi is far away
from the center, x, the potential influence if yi
is an outlier is much greater than if xi is near x.
Open the tool to see a set of (x, y) points. The least squares
regression line fits the data and there are no apparent outliers yet. Now perform the
following steps in order.
-
Click on the leftmost point to see it turn red and show its y-value, 3.90. Using
either your mouse or the up-arrow key, increase the y-value to make it a positive
outlier. The text “Δ slope” reports the change in slope. Notice that the
line becomes negatively sloped even though a positively sloped line better fits the other
data points (blue). When the y-value becomes a very large outlier, the line
seems to fit the data very poorly.
-
Now use your mouse or the down-arrow key to decrease the y-value of the leftmost
point. The line becomes steeply positively sloped and fits the other points (blue) very
poorly.
-
Now click on the 5th point from the left to turn it red and report a y-value of
4.10. The leftmost point is reset to its original position and color. Use the mouse or the
up- and down-arrow keys to see the effect of creating a large positive or negative outlier
for the 5th point. Notice that although the line shifts up or down, its slope does not
change (the Δ slope remains 0.0). The 5th data point has no effect on the slope
because its horizontal position is exactly the average horizontal position,
x. Points with x-values close to x
have very little leverage.
-
Now click on the rightmost data point and use the mouse or up- and down-arrow keys to see
the effect of making it a positive or negative outlier. Like the leftmost point, the
rightmost point changes the slope dramatically when it becomes an outlier. The rightmost
point has a lot of leverage because its x-value is far from x.
Summary
- Leverage and influence are important concepts in regression.
-
A point (xi, yi) has a lot of
influence if the equation (usually the slope) of the regression
line changes dramatically if that data point is removed.
-
A point (xi, yi) has a lot of
leverage if it has the potential to be influential when
yi is an outlier. This happens when xi is far away
from x.