Tests for Two or More Independent Samples, Discrete Outcome
Here we extend that application of the chi-square test to the case with two or more independent comparison groups. Specifically, the outcome of interest is discrete with two or more responses and the responses can be ordered or unordered (i.e., the outcome can be dichotomous, ordinal or categorical). We now consider the situation where there are two or more independent comparison groups and the goal of the analysis is to compare the distribution of responses to the discrete outcome variable among several independent comparison groups.
The test is called the χ2 test of independence and the null hypothesis is that there is no difference in the distribution of responses to the outcome across comparison groups. This is often stated as follows: The outcome variable and the grouping variable (e.g., the comparison treatments or comparison groups) are independent (hence the name of the test). Independence here implies homogeneity in the distribution of the outcome among comparison groups.
The null hypothesis in the χ2 test of independence is often stated in words as: H0: The distribution of the outcome is independent of the groups. The alternative or research hypothesis is that there is a difference in the distribution of responses to the outcome variable among the comparison groups (i.e., that the distribution of responses "depends" on the group). In order to test the hypothesis, we measure the discrete outcome variable in each participant in each comparison group. The data of interest are the observed frequencies (or number of participants in each response category in each group). The formula for the test statistic for the χ2 test of independence is given below.
Test Statistic for Testing H0: Distribution of outcome is independent of groups
and we find the critical value in a table of probabilities for the chi-square distribution with df=(r-1)*(c-1).
Here O = observed frequency, E=expected frequency in each of the response categories in each group, r = the number of rows in the two-way table and c = the number of columns in the two-way table. r and c correspond to the number of comparison groups and the number of response options in the outcome (see below for more details). The observed frequencies are the sample data and the expected frequencies are computed as described below. The test statistic is appropriate for large samples, defined as expected frequencies of at least 5 in each of the response categories in each group.
The data for the χ2 test of independence are organized in a two-way table. The outcome and grouping variable are shown in the rows and columns of the table. The sample table below illustrates the data layout. The table entries (blank below) are the numbers of participants in each group responding to each response category of the outcome variable.
Table - Possible outcomes are are listed in the columns; The groups being compared are listed in rows.
|
|
Outcome Variable |
|
|||
|
Grouping Variable |
Response Option 1 |
Response Option 2 |
... |
Response Option c |
Row Totals |
|
Group 1 |
|
|
|
|
|
|
Group 2 |
|
|
|
|
|
|
. . . |
|
|
|
|
|
|
Group r |
|
|
|
|
|
|
Column Totals |
|
|
|
|
N |
In the table above, the grouping variable is shown in the rows of the table; r denotes the number of independent groups. The outcome variable is shown in the columns of the table; c denotes the number of response options in the outcome variable. Each combination of a row (group) and column (response) is called a cell of the table. The table has r*c cells and is sometimes called an r x c ("r by c") table. For example, if there are 4 groups and 5 categories in the outcome variable, the data are organized in a 4 X 5 table. The row and column totals are shown along the right-hand margin and the bottom of the table, respectively. The total sample size, N, can be computed by summing the row totals or the column totals. Similar to ANOVA, N does not refer to a population size here but rather to the total sample size in the analysis. The sample data can be organized into a table like the above. The numbers of participants within each group who select each response option are shown in the cells of the table and these are the observed frequencies used in the test statistic.
The test statistic for the χ2 test of independence involves comparing observed (sample data) and expected frequencies in each cell of the table. The expected frequencies are computed assuming that the null hypothesis is true. The null hypothesis states that the two variables (the grouping variable and the outcome) are independent. The definition of independence is as follows:
Two events, A and B, are independent if P(A|B) = P(A), or equivalently, if P(A and B) = P(A) P(B).
The second statement indicates that if two events, A and B, are independent then the probability of their intersection can be computed by multiplying the probability of each individual event. To conduct the χ2 test of independence, we need to compute expected frequencies in each cell of the table. Expected frequencies are computed by assuming that the grouping variable and outcome are independent (i.e., under the null hypothesis). Thus, if the null hypothesis is true, using the definition of independence:
P(Group 1 and Response Option 1) = P(Group 1) P(Response Option 1).
The above states that the probability that an individual is in Group 1 and their outcome is Response Option 1 is computed by multiplying the probability that person is in Group 1 by the probability that a person is in Response Option 1. To conduct the χ2 test of independence, we need expected frequencies and not expected probabilities. To convert the above probability to a frequency, we multiply by N. Consider the following small example.
|
|
Response 1 |
Response 2 |
Response 3 |
Total |
|---|---|---|---|---|
|
Group 1 |
10 |
8 |
7 |
25 |
|
Group 2 |
22 |
15 |
13 |
50 |
|
Group 3 |
30 |
28 |
17 |
75 |
|
Total |
62 |
51 |
37 |
150 |
The data shown above are measured in a sample of size N=150. The frequencies in the cells of the table are the observed frequencies. If Group and Response are independent, then we can compute the probability that a person in the sample is in Group 1 and Response category 1 using:
P(Group 1 and Response 1) = P(Group 1) P(Response 1),
P(Group 1 and Response 1) = (25/150) (62/150) = 0.069.
Thus if Group and Response are independent we would expect 6.9% of the sample to be in the top left cell of the table (Group 1 and Response 1). The expected frequency is 150(0.069) = 10.4. We could do the same for Group 2 and Response 1:
P(Group 2 and Response 1) = P(Group 2) P(Response 1),
P(Group 2 and Response 1) = (50/150) (62/150) = 0.138.
The expected frequency in Group 2 and Response 1 is 150(0.138) = 20.7.
Thus, the formula for determining the expected cell frequencies in the χ2 test of independence is as follows:
Expected Cell Frequency = (Row Total * Column Total)/N.
The above computes the expected frequency in one step rather than computing the expected probability first and then converting to a frequency.
Example:
In a prior example we evaluated data from a survey of university graduates which assessed, among other things, how frequently they exercised. The survey was completed by 470 graduates. In the prior example we used the χ2 goodness-of-fit test to assess whether there was a shift in the distribution of responses to the exercise question following the implementation of a health promotion campaign on campus. We specifically considered one sample (all students) and compared the observed distribution to the distribution of responses the prior year (a historical control). Suppose we now wish to assess whether there is a relationship between exercise on campus and students' living arrangements. As part of the same survey, graduates were asked where they lived their senior year. The response options were dormitory, on-campus apartment, off-campus apartment, and at home (i.e., commuted to and from the university). The data are shown below.
|
|
No Regular Exercise |
Sporadic Exercise |
Regular Exercise |
Total |
|---|---|---|---|---|
|
Dormitory |
32 |
30 |
28 |
90 |
|
On-Campus Apartment |
74 |
64 |
42 |
180 |
|
Off-Campus Apartment |
110 |
25 |
15 |
150 |
|
At Home |
39 |
6 |
5 |
50 |
|
Total |
255 |
125 |
90 |
470 |
Based on the data, is there a relationship between exercise and student's living arrangement? Do you think where a person lives affect their exercise status? Here we have four independent comparison groups (living arrangement) and a discrete (ordinal) outcome variable with three response options. We specifically want to test whether living arrangement and exercise are independent. We will run the test using the five-step approach.
- Step 1. Set up hypotheses and determine level of significance.
H0: Living arrangement and exercise are independent
H1: H0 is false. α=0.05
The null and research hypotheses are written in words rather than in symbols. The research hypothesis is that the grouping variable (living arrangement) and the outcome variable (exercise) are dependent or related.
- Step 2. Select the appropriate test statistic.
The formula for the test statistic is:
.
The condition for appropriate use of the above test statistic is that each expected frequency is at least 5. In Step 4 we will compute the expected frequencies and we will ensure that the condition is met.
- Step 3. Set up decision rule.
The decision rule depends on the level of significance and the degrees of freedom, defined as df = (r-1)(c-1), where r and c are the numbers of rows and columns in the two-way data table. The row variable is the living arrangement and there are 4 arrangements considered, thus r=4. The column variable is exercise and 3 responses are considered, thus c=3. For this test, df=(4-1)(3-1)=3(2)=6. Again, with χ2 tests there are no upper, lower or two-tailed tests. If the null hypothesis is true, the observed and expected frequencies will be close in value and the χ2 statistic will be close to zero. If the null hypothesis is false, then the χ2 statistic will be large. The rejection region for the χ2 test of independence is always in the upper (right-hand) tail of the distribution. For df=6 and a 5% level of significance, the appropriate critical value is 12.59 and the decision rule is as follows: Reject H0 if c 2 > 12.59.
- Step 4. Compute the test statistic.
We now compute the expected frequencies using the formula,
Expected Frequency = (Row Total * Column Total)/N.
The computations can be organized in a two-way table. The top number in each cell of the table is the observed frequency and the bottom number is the expected frequency. The expected frequencies are shown in parentheses.
|
|
No Regular Exercise |
Sporadic Exercise |
Regular Exercise |
Total |
|---|---|---|---|---|
|
Dormitory
|
32 (48.8) |
30 (23.9) |
28 (17.2) |
90 |
|
On-Campus Apartment |
74 (97.7) |
64 (47.9) |
42 (34.5) |
180 |
|
Off-Campus Apartment |
110 (81.4) |
25 (39.9) |
15 (28.7) |
150 |
|
At Home
|
39 (27.1) |
6 (13.3) |
5 (9.6) |
50 |
|
Total |
255 |
125 |
90 |
470 |
Notice that the expected frequencies are taken to one decimal place and that the sums of the observed frequencies are equal to the sums of the expected frequencies in each row and column of the table.
Recall in Step 2 a condition for the appropriate use of the test statistic was that each expected frequency is at least 5. This is true for this sample (the smallest expected frequency is 9.6) and therefore it is appropriate to use the test statistic.
The test statistic is computed as follows:
- Step 5. Conclusion.
We reject H0 because 60.5 > 12.59. We have statistically significant evidence at a =0.05 to show that H0 is false or that living arrangement and exercise are not independent (i.e., they are dependent or related), p < 0.005.
Again, the χ2 test of independence is used to test whether the distribution of the outcome variable is similar across the comparison groups. Here we rejected H0 and concluded that the distribution of exercise is not independent of living arrangement, or that there is a relationship between living arrangement and exercise. The test provides an overall assessment of statistical significance. When the null hypothesis is rejected, it is important to review the sample data to understand the nature of the relationship. Consider again the sample data.
|
|
No Regular Exercise |
Sporadic Exercise |
Regular Exercise |
Total |
|---|---|---|---|---|
|
Dormitory |
32 |
30 |
28 |
90 |
|
On-Campus Apartment |
74 |
64 |
42 |
180 |
|
Off-Campus Apartment |
110 |
25 |
15 |
150 |
|
At Home |
39 |
6 |
5 |
50 |
|
Total |
255 |
125 |
90 |
470 |
Because there are different numbers of students in each living situation, it makes the comparisons of exercise patterns difficult on the basis of the frequencies alone. The following table displays the percentages of students in each exercise category by living arrangement. The percentages sum to 100% in each row of the table. For comparison purposes, percentages are also shown for the total sample along the bottom row of the table.
|
|
No Regular Exercise |
Sporadic Exercise |
Regular Exercise |
|---|---|---|---|
|
Dormitory |
36% |
33% |
31% |
|
On-Campus Apartment |
41% |
36% |
23% |
|
Off-Campus Apartment |
73% |
17% |
10% |
|
At Home |
78% |
12% |
10% |
|
Total |
54% |
27% |
19% |
From the above, it is clear that higher percentages of students living in dormitories and in on-campus apartments reported regular exercise (31% and 23%) as compared to students living in off-campus apartments and at home (10% each).
Test Yourself
Pancreaticoduodenectomy (PD) is a procedure that is associated with considerable morbidity. A study was recently conducted on 553 patients who had a successful PD between January 2000 and December 2010 to determine whether their Surgical Apgar Score (SAS) is related to 30-day perioperative morbidity and mortality. The table below gives the number of patients experiencing no, minor, or major morbidity by SAS category.
|
Surgical Apgar Score |
No morbidity |
Minor morbidity |
Major morbidity or mortality |
|---|---|---|---|
|
0-4 |
21 |
20 |
16 |
|
5-6 |
135 |
71 |
35 |
|
7-10 |
158 |
62 |
35 |
Question: What would be an appropriate statistical test to examine whether there is an association between Surgical Apgar Score and patient outcome? Using 14.13 as the value of the test statistic for these data, carry out the appropriate test at a 5% level of significance. Show all parts of your test.
In the module on hypothesis testing for means and proportions, we discussed hypothesis testing applications with a dichotomous outcome variable and two independent comparison groups. We presented a test using a test statistic Z to test for equality of independent proportions. The chi-square test of independence can also be used with a dichotomous outcome and the results are mathematically equivalent.
In the prior module, we considered the following example. Here we show the equivalence to the chi-square test of independence.
Example:
A randomized trial is designed to evaluate the effectiveness of a newly developed pain reliever designed to reduce pain in patients following joint replacement surgery. The trial compares the new pain reliever to the pain reliever currently in use (called the standard of care). A total of 100 patients undergoing joint replacement surgery agreed to participate in the trial. Patients were randomly assigned to receive either the new pain reliever or the standard pain reliever following surgery and were blind to the treatment assignment. Before receiving the assigned treatment, patients were asked to rate their pain on a scale of 0-10 with higher scores indicative of more pain. Each patient was then given the assigned treatment and after 30 minutes was again asked to rate their pain on the same scale. The primary outcome was a reduction in pain of 3 or more scale points (defined by clinicians as a clinically meaningful reduction). The following data were observed in the trial.
|
Treatment Group |
n |
Number with Reduction of 3+ Points |
Proportion with Reduction of 3+ Points |
|---|---|---|---|
|
New Pain Reliever |
50 |
23 |
0.46 |
|
Standard Pain Reliever |
50 |
11 |
0.22 |
We tested whether there was a significant difference in the proportions of patients reporting a meaningful reduction (i.e., a reduction of 3 or more scale points) using a Z statistic, as follows.
- Step 1. Set up hypotheses and determine level of significance
H0: p1 = p2
H1: p1 ≠ p2 α=0.05
Here the new or experimental pain reliever is group 1 and the standard pain reliever is group 2.
- Step 2. Select the appropriate test statistic.
We must first check that the sample size is adequate. Specifically, we need to ensure that we have at least 5 successes and 5 failures in each comparison group or that:
In this example, we have
Therefore, the sample size is adequate, so the following formula can be used:
.
- Step 3. Set up decision rule.
Reject H0 if Z < -1.960 or if Z > 1.960.
- Step 4. Compute the test statistic.
We now substitute the sample data into the formula for the test statistic identified in Step 2. We first compute the overall proportion of successes:
We now substitute to compute the test statistic.
- Step 5. Conclusion.
We reject H0 because 
. We have statistically significant evidence at α=0.05 to show that there is a difference in the proportions of patients on the new pain reliever reporting a meaningful reduction (i.e., a reduction of 3 or more scale points) as compared to patients on the standard pain reliever.
We now conduct the same test using the chi-square test of independence.
- Step 1. Set up hypotheses and determine level of significance.
H0: Treatment and outcome (meaningful reduction in pain) are independent
H1: H0 is false. α=0.05
- Step 2. Select the appropriate test statistic.
The formula for the test statistic is:
The condition for appropriate use of the above test statistic is that each expected frequency is at least 5. In Step 4 we will compute the expected frequencies and we will ensure that the condition is met.
- Step 3. Set up decision rule.
For this test, df=(2-1)(2-1)=1. At a 5% level of significance, the appropriate critical value is 3.84 and the decision rule is as follows: Reject H0 if χ2 > 3.84. (Note that 1.962 = 3.84, where 1.96 was the critical value used in the Z test for proportions shown above.)
- Step 4. Compute the test statistic.
We now compute the expected frequencies using:
The computations can be organized in a two-way table. The top number in each cell of the table is the observed frequency and the bottom number is the expected frequency. The expected frequencies are shown in parentheses.
|
Treatment Group |
# with Reduction of 3+ Points |
# with Reduction of <3 Points |
Total |
|---|---|---|---|
|
New Pain Reliever |
23 (17.0) |
27 (33.0) |
50 |
|
Standard Pain Reliever |
11 (17.0) |
39 (33.0) |
50 |
|
Total |
34 |
66 |
100 |
A condition for the appropriate use of the test statistic was that each expected frequency is at least 5. This is true for this sample (the smallest expected frequency is 22.0) and therefore it is appropriate to use the test statistic.
The test statistic is computed as follows:
(Note that (2.53)2 = 6.4, where 2.53 was the value of the Z statistic in the test for proportions shown above.)
- Step 5. Conclusion.
We reject H0 because 
. We have statistically significant evidence at α=0.05 to show that H0 is false or that treatment and outcome are not independent (i.e., they are dependent or related). This is the same conclusion we reached when we conducted the test using the Z test above. With a dichotomous outcome and two independent comparison groups, Z2 = χ2 ! Again, in statistics there are often several approaches that can be used to test hypotheses.
Chi-Squared Tests in R
The video below by Mike Marin demonstrates how to perform chi-squared tests in the R programming language.
