Introducing Quasi-Experimental Plus/Minus (QEPM)

In this article, I introduce a variant of Plus/Minus which might be useful to basketball analysts (despite not being a metric).


The Holy Grail of sports analytics is player evaluation. From a managerial point of view, being able to assess the real value of players allows one to avoid those who are overvalued and to recruit those who are undervalued.

A player’s worth is usually assessed based on subjective judgment (including common knowledge) and his individual statistics. However, one can reasonably doubt that these two components capture the real value of a player (this is particularly true since the Moneyball story). The usual individual statistics (first-serve percentage in tennis, batting average in baseball, goals scored in soccer, etc.) are somehow thought to reflect the value of players but they do not reflect their overall value. Therefore, there are some statisticians looking for new stats that would reveal the true value of players.

Basketball is no exception. Everybody knows that the basic individual stats (points, rebounds, assists, steals, blocks, turnovers, shot attempts) do not capture the overall value of a player, especially his defensive contribution. Several stats have been proposed that are supposed to reflect the overall value of players.

The first one is the NBA’s efficiency measure which combines the basic individual stats into a single number:

(points + rebounds + assists + steals + blocks) – [(field goals attempted − field goals made) + (free throws attempted − free throws attempted) + turnovers)]

The second stat is the player efficiency rating (PER) proposed by ESPN analyst John Hollinger. This is also an all-in-one measure but somewhat more elaborated than the NBA’s efficiency measure (see here for the formula).

The third stat is Plus/Minus which we are going to consider in detail.


The core idea of the Plus/Minus approach is that the overall value of a player is his effect on his team’s performance. How to measure the effect of a player on his team’s performance? One has to compare the team’s performance when the player is on the court to the team’s performance when he is off the court. This is basic experimental science: if you want to test the effect of X on Y, you vary X and observe how this variation affects Y. So experimentally speaking, we would say that the target player is the independent variable while his team’s performance is the dependent variable.

There are two main Plus/Minus stats: Net Plus/Minus (NPM) and Adjusted Plus/Minus (APM).

Net Plus/Minus was defined by Roland Beech (who launched 82games during the 2002-03 season). The calculation of NPM is straightforward. There are two equivalent ways to compute NPM. Let’s consider the following example. The table below shows the points scored and allowed by a given player’s team when he was on the court and when he was off the court:

 Player on the courtPlayer off the court
Points scored (per 100 possessions)11598
Points allowed (per 100 possessions)110105

First manner: NPM is the difference between the point differential (points scored – points allowed) when the player is on the court and the point differential when he is off the court.

NPM = (115 – 110) – (98 – 105) = 5 – (– 7) = 12

Second manner: NPM is the difference between a) points scored when the player is on the court minus points scored when he is off the court (Net Offensive Plus/Minus) and b) points allowed when the player is on the court minus points allowed when he is off the court (Net Defensive Plus/Minus).

NPM = (115 – 98) – (110 – 105) = 17 – (5) = 12

The more a player contributes to his team’s performance (offensively and defensively), the higher his NPM. In this example, the player’s team does better when he is on the court than when he is not.

It is easy to see that Net Plus/Minus faces a strong limitation. Indeed, comparing a team’s performance when a player is on the court to the performance when the player is off the court is fundamentally flawed if the teammates and opponents are not the same in both conditions. For instance, a player who is surrounded by good teammates when he is on court will get a higher NPM than a player who is often surrounded by bad teammates (all other things being equal).

Experimentally speaking, if you want to test the effect of X on Y, the conditions that you compare must differ only regarding X. Indeed, if the conditions also differ on Z, any observed variation on Y could be attributed either to X or Z (which is then called a confounding variable). If one wants to draw a valid causal inference, conditions must differ on X all other things being equal.

Adjusted Plus/Minus is Plus/Minus when controlling for teammates and opponents. This is done through a common statistical method called multiple regression. APM has been proposed by Dan Rosenbaum in this 2004 article. Eli Witus (Basketball Operations Analyst for the Houston Rockets) has described precisely how to compute APM here. I have also described how to compute NPM and APM in this article on my blog.


As mentioned above, the main issue regarding Plus/Minus is to control for teammates and opponents. In APM, this issue is addressed statistically through multiple regression. With QEPM, I propose to address this issue experimentally (more precisely, quasi-experimentally). What does it mean?

The goal is to compare how a team performs when a given player is on the court (condition A) to how the team performs when he is off the court (condition B), teammates and opponents being the same in both conditions. In other words, we want to compare a given player to another player of the same team, all other things being equal. Therefore, the question is: in a NBA match, are there couples of game segments that differ only regarding two players of the same team, all other things (teammates and opponents) being equal? A typical NBA game includes about 30 game segments. In certain games, we can actually find such couples of game segments.

Let’s take a look at the raw data file that allows us to identify these couples of game segments. The data file we are going to consider in this example is relative to the 2007-08 season and it shows all games segments of the 1230 games that were played during this season. The file (and those of other seasons) can be found on BasketballValue (see below “Calculating Quasi-Experimental Plus/Minus”)

Here is a sample of the file:


The sample shows the first 13 game segments of a Lakers vs. Nuggets game that took place November 29, 2007 (that game has 24 game segments in total). Basically, each row represents of game segment. For each game segment, there are three main information: the players that were on the court, the points scored and allowed by each team, and the number of possessions of each team. Thus, one can calculate the per-possession Plus/Minus of each lineup for each game segment.

Let’s see if we can find couples of game segments that allow for a comparison all other things being equal. For instance, consider the two following game segments:

5th game segment (5:04 to 6:18)
home players: Farmar, Radmanovic, Turiaf, Vujacic, Walton
away players: Camby, Iverson, Jones, Kleiza, Smith

11th game segment (12:52 to 15:13)
home players: Bryant, Farmar, Odom, Turiaf, Vujacic
away players: Anthony, Camby, Iverson, Najera, Smith

Regarding the home players, there are two differences between the lineups: Radmanovic/Odom and Walton/Bryant. And regarding the away players, there are also two differences between the lineups: Jones/Anthony and Kleiza/Najera. So the comparison of these two game segments is not “all other things being equal”.

Now consider the two following game segments:

6th game segment (6:32 to 8:29)
home players: Farmar, Radmanovic, Turiaf, Vujacic, Walton
away players: Anthony, Iverson, Kleiza, Najera, Smith

10th game segment (12:00 to 12:52)
home players: Bryant, Farmar, Radmanovic, Turiaf, Vujacic
away players: Anthony, Iverson, Kleiza, Najera, Smith

There is only one difference between the home players lineups (Walton/Bryant) while the away players lineups are identical. Therefore, this comparison is valid because it involves only one difference, all other things being equal. There is another valid comparison, between the 9th game segment (10:31 to 12:00) and the 10th game segment (12:00 to 12:52): the only difference is also Walton/Bryant.

In summary, the principle of QEPM is to find, for a given player over a season, all couples of game segments that allow for comparisons “all other things being equal”. Such an approach is called a quasi-experiment.

In an experiment, one tests whether X has a causal influence on Y by comparing conditions that differ only on X. Crucially, subjects are randomly assigned to the conditions. Random assignment ensures that the conditions eventually differ only on X. This is what is basically done when the effect of a new drug is tested (a type of study called randomized controlled trial). Participants are randomly assigned to the experimental (drug) or control (placebo) conditions and eventually, the two groups are compared on one or several criteria.

But there are cases in which the random assignment to conditions is not feasible (for practical or ethical reasons). For instance, imagine that we want to test whether the sexual orientation of parents influences children’s personality. Basically, we compare two conditions: children being raised by a mother and a father on one hand and children being raised by parents of the same sex on the other hand. Obviously, it would not be ethical to randomly assign newborns to these two conditions and assess their personality thereafter.

In such cases, a typical procedure is to select a group of children being raised by a mother and a father and a group of children being raised by parents of the same sex. Because of the lack of random assignment, the two groups might differ on other features than the sexual orientation of parents (age, socioeconomic status, etc.). Some of these confounding variables could account for any observed difference regarding children’s personality. Therefore, the two groups must be made so that they are matched on several important features: one wants to be as close as possible to a comparison “all other things being equal”.

Such a study is called a quasi-experiment: as an experiment, its goal is to test a causal relationship between two variables, but unlike an experiment, it lacks random assignment so that one cannot be entirely sure that any difference on the dependent variable is truly caused by the independent variable.


I have created an algorithm that implements QEPM. The algorithm is a function coded in R. Basically, it takes a player’s ID as input, scans the raw data file, and provides a data frame as output which shows all the comparisons “all other things being equal” for that player.

The raw data file can be downloaded on BasketballValue. Go to: “Downloads”, “2007-2008” (for instance), and download the folder “List of each matchup of one unit against another”. In the folder, the file is “matchups20072008reg20081211.text”. Open the file and save it in csv format. Here is the file: matchups20072008reg20081211

Here are the first 10 rows of the file (content is described just above):


First, load the file in R. When you execute the command below, a box opens and you must select the file “matchups20072008reg20081211.csv” We rename the file “d”:

d <- read.csv2 (file.choose())

Second, let’s work the file just a little bit: we recode the variable “GameID” and we number the games.

nrow_d <- nrow(d)

for (i in 1:nrow_d) {
d$Date[i] <- substr(d$GameID[i], start = 1, stop = 8)
d$AwayTeam[i] <- substr(d$GameID[i], start = 9, stop = 11)
d$HomeTeam[i] <- substr(d$GameID[i], start = 12, stop = 14)
d$GameName[i] <- toString(c(d$Date[i], d$AwayTeam[i], d$HomeTeam[i]))

d <- d[,-which(names(d) %in% c("GameID"))]

d$GameID <- rep(c(0), nrow(d))
d$GameID[1] <- 1

for(i in 2:nrow_d)
if(d$GameName[i] == d$GameName[i-1]) {d$GameID[i] <- d$GameID[i-1]} else {d$GameID[i] <- d$GameID[i-1]+1}

save(d, file = "d.RData")

Now the file looks like this (first 10 rows):

00:00:0100:00:0000:00:01118918610351645930831830530262BrownBryantFisherTuriafWaltonAlstonJamesMcGradyYaoWells93959395000000NULLNULLNULLNULL0000NULLNULLNULLNULL20071030HOULAL20071030, HOU, LAL1
00:00:0200:00:0100:00:011189186575103459308198508318305BrownBryantFarmarFisherWaltonAlstonBattierHayesJamesMcGrady92959395101-100NULLNULLNULLNULL0000NULLNULLNULLNULL20071030HOULAL20071030, HOU, LAL1
00:01:3200:00:0200:01:3090189186575103459308198318305302BrownBryantFarmarFisherWaltonAlstonBattierJamesMcGradyYao83929295936-64522560165-1650010NULL01NULL20071030HOULAL20071030, HOU, LAL1
00:01:3600:01:3200:00:044189186103664459308198318305302BrownBryantFisherKarlWaltonAlstonBattierJamesMcGradyYao80928392303-310300NULLNULLNULL0000NULLNULLNULLNULL20071030HOULAL20071030, HOU, LAL1
00:04:3000:01:3600:02:54174189186103516459308198508305302BrownBryantFisherTuriafWaltonAlstonBattierHayesMcGradyYao76828092410-665680166.667-86.666786.66670013001120071030HOULAL20071030, HOU, LAL1
00:04:4300:04:3000:00:1313186194103516459308198508305302BryantBynumFisherTuriafWaltonAlstonBattierHayesMcGradyYao76827682000000NULLNULLNULLNULL0000NULLNULLNULLNULL20071030HOULAL20071030, HOU, LAL1
00:06:3900:04:4300:01:5611618619410351645930850830530262BryantBynumFisherTuriafWaltonAlstonHayesMcGradyYaoWells69777682752-24417512550-500020NULL01NULL20071030HOULAL20071030, HOU, LAL1
00:06:5400:06:3900:00:1515186194103516459308198508305302BryantBynumFisherTuriafWaltonAlstonBattierHayesMcGradyYao68776977101-1111000100-10000010NULLNULL120071030HOULAL20071030, HOU, LAL1
00:09:4200:06:5400:02:4816818619410351645950831830530262BryantBynumFisherTuriafWaltonHayesJamesMcGradyYaoWells62746877633-37685.71435035.7143-35.71430033001120071030HOULAL20071030, HOU, LAL1
00:10:5300:09:4200:01:117118619427657518850831830530262BryantBynumEvansFarmarMihmHayesJamesMcGradyYaoWells6270627404-44220200-200200011200.50.5120071030HOULAL20071030, HOU, LAL1

Here is the algorithm that computes QEPM (annotations are in French, sorry…). It is a function in R called “qepm”. Note that you must install the two R packages “psych” and “prettyR”.

Now, to get the results of a given player, you just have to execute the “qepm” function. For instance, Steve Nash’s ID is #8. To get his results, execute:

x <- qepm(8)

(the computation might take some time, about 5 minutes on my computer)

Here is a sample of the output of the algorithm:

1420071101PHXSEAAway8488632216, 7, 4, 8, 537488, 16, 7, 4, 537172, 714, 715, 168, 173172, 714, 715, 168, 1731100:32:5500:31:506500:33:1800:32:552332110203-2-3-1.00000000-3.00000000
3820071104PHXCLEHome81651887, 12, 486, 4, 816, 7, 12, 486, 4561, 257, 260, 264, 265561, 257, 260, 264, 2651100:00:4200:00:004200:14:4600:12:0016622762375-12-0.500000000.16666667
3820071104PHXCLEHome8486418816, 7, 12, 4, 816, 7, 12, 486, 4561, 257, 260, 264, 265561, 257, 260, 264, 2651100:15:1600:14:463000:14:4600:12:0016622763375020.000000000.16666667
5020071106PHXCHAAway848883121216, 486, 4, 8, 78488, 16, 486, 4, 78307, 126, 117, 90, 111307, 126, 117, 90, 1111100:31:5600:30:407600:32:4400:31:5648332320042-40.66666667-1.33333333
7520071109PHXMIAAway8486324416, 7, 12, 4, 816, 7, 12, 486, 4243, 201, 206, 185, 196243, 201, 206, 185, 1961100:41:0000:39:0611400:15:1800:14:582044113400-10-0.250000000.00000000
8420071110PHXORLAway8488748816, 12, 4, 8, 78488, 16, 12, 4, 78245, 277, 419, 149, 170245, 277, 419, 149, 1701100:15:3800:14:425600:14:4200:12:00162225540244-22.00000000-0.40000000
8420071110PHXORLAway8488747716, 12, 4, 8, 78488, 16, 12, 4, 78277, 419, 161, 149, 170277, 419, 161, 149, 1701100:40:1100:37:1617500:37:1600:36:0175652382046-40.93333333-1.33333333
10120071113PHXNYKHome81684151512, 4, 8, 131, 7816, 12, 4, 131, 78597, 679, 249, 154, 255597, 679, 249, 154, 2551100:15:4500:14:545100:14:5400:12:0017422550146-1-2-0.50000000-0.40000000
10120071113PHXNYKHome816842212, 4, 8, 131, 7816, 12, 4, 131, 78597, 249, 235, 154, 254597, 249, 235, 154, 2541100:39:2000:38:215900:38:2100:36:0014111662368-1-2-1.00000000-0.33333333
10120071113PHXNYKHome813164151516, 12, 4, 8, 7816, 12, 4, 131, 78597, 679, 249, 154, 255597, 679, 249, 154, 2551100:16:2200:15:453700:14:5400:12:00174115530463-23.00000000-0.40000000
11720071115PHXCHIHome87613316, 12, 486, 4, 816, 7, 12, 486, 4122, 113, 110, 112, 559122, 113, 110, 112, 5591100:08:4200:08:004200:12:4300:12:004312212402-2-20.00000000-2.00000000
11720071115PHXCHIHome8772131316, 12, 4, 8, 7816, 7, 12, 4, 78122, 113, 119, 383, 559122, 113, 119, 383, 5591100:39:4900:38:555400:38:5500:37:1310222340402-4-2-2.00000000-0.50000000
11720071115PHXCHIHome8486419916, 7, 12, 4, 816, 7, 12, 486, 4122, 113, 119, 112, 559122, 113, 119, 112, 5591100:15:1000:14:254500:14:2500:12:4310212330403-4-3-2.00000000-1.00000000

For instance, the first row says this: in the game Phoenix vs. Seattle that took place on November 1, 2007, there is one comparison “all other things being equal” involving player #8, Steve Nash (“TP” column), the other player involved in the comparison being player #488, Marcus Banks (“CP” column).

The two game segments corresponding to that comparison are described in detail. The lineup including Steve Nash is 16, 7, 4, 8, 537 (“TP_team_players” column), the opposing lineup being 172, 714, 715, 168, 173 (“TP_opp_players” column). The lineup including Marcus Banks is 488, 16, 7, 4, 537 (“CP_team_players” column), the opposing lineup being 172, 714, 715, 168, 173 (“CP_opp_players” column).

The per-possession Plus/Minus of the lineup including Steve Nash is -1.0 (“TP_PM_perposs” column) while that of the lineup including Marcus Banks is -3.0 (“CP_PM_perposs” column).

As one might expect, most of games contain no comparisons “all other things being equal”, while some games contain several.

The command below displays the different players to which the target player can be compared:


Here is the output in our example with Steve Nash:

Value   Count   Percent
16      27      31.03
486     16      18.39
7       11      12.64
12      8       9.2
133     8       9.2
488     6       6.9
4       4       4.6
78      3       3.45
131     1       1.15
203     1       1.15
mode=16  Valid n=87   12 categories - only first 10 shown

Steve Nash can be compared “all other things being equal” to:
• player #16 (Leandro Barbosa) in 27 observations
• player #486 (Grant Hill) in 16 observations
• player #7 (Raja Bell) in 11 observations

In some cases, there are enough observations to test whether the difference on per-possession Plus/Minus between the two conditions is statistically significant. The relevant statistical test is a paired t-test. Here is a good example. Let’s consider Manu Ginobili (player #24) as the target player. It turns out that Ginobili can be compared to Michael Finley (player #29) “all other things being equal” in 83 observations, which is a significant amount of observations (both players are shooting guards).

Let’s calculate the mean per-possession Plus/Minus of both players:

comp_P29 <- x[x$CP == 29,]

Ginobili: 0.078, Finley: -0.222. Is that difference statistically significant? The command below performs the paired t-test to compare the two values:

t.test(comp_P29$TP_PM_perposs, comp_P29$CP_PM_perposs, paired = TRUE)

Here is the result:

t = 1.9192, df = 82, p-value = 0.05844
alternative hypothesis: true difference in means is not equal to 0

The difference between the mean per-possession Plus/Minus of Ginobili (0.078) and that of Finley (-0.222) is almost statistically significant. It means that during the 2007-08 season, the Spurs did better when Ginobili was on the court than when Finley was on the court, all other things being equal.


Compared to Net Plus/Minus (NPM) and Adjusted Plus/Minus (APM), Quasi-Experimental Plus/Minus (QEPM) is not a metric. In fact, QEPM is an (quasi) experimental approach of Plus/Minus: it allows one to compare how a team performs when a given player is on the court to how the team performs when another player is on the court, all other things (teammates and opponents) being equal. I think this information can be quite useful to basketball analysts.

Fundamentally, QEPM shows that in certain sports like basketball, it is possible to test rigorously causal relationships even though many people think this is not possible. For instance, Hirsch and Hirsch (2011) have published a book in which they massively criticize the scientific approach of sports. Here is what they write (p.49) about the impossibility to test causal relationships in baseball:

“Take the suggestion, as discussed above, that Roger Maris hit better with Mickey Mantle batting behind him. We supplied data to support that conclusion, but the data hardly prove that manager Ralph Houk was right to bat Mantle behind Maris. The first question he’d need to ask is whether Maris’s success with Mantle batting behind him amounts to a real finding or a random phenomenon rooted in a small sample size. Assuming he determined that is was real, he’d need to consider the converse: might not Maris protect Mantle equally well? How could he establish which way they were better off?

The obvious answer is to switch the order for awhile, and see how Maris, Mantle and the Yanks fare with Maris behind Mantle as compared to how they’d done previously. In fact, this experiment would not yield reliable information. Experiments provide the most useful information when the scientific method is used – all relevant factors kept constant except one (in this case, where Maris and Mantle bat). If you vary only the factor in question, the results can confidently be attributed to the change in the single variable. Baseball does not permit such a controlled experiment. The flipping of Maris and Mantle in the lineup will necessarily be accompanied by numerous other changes as well – the teams the Yankees play, the pitchers they face, the weather, the performance of the batters before and after Maris and Mantle and, most importantly, the players themselves. […] In short, any hypothesis about which batting order best serves the Yankees can never reliably be tested.

That does not hold for basketball which allows for quasi-experiments.

Share on FacebookTweet about this on TwitterEmail this to someone