Volume 12, Issue 1: 2019

Let Them In: Increasing Access, Completion, and Equity in English Placement Policies at a Two-Year College in California

by Leslie Henson, Butte College and Katie Hern, California Acceleration Project

This article uses a disparate impact analysis framework to assess the impact of a policy change in writing assessment that roughly doubled the proportion of students placing into college English at Butte College, a two-year college in California. After establishing the disparate impact of placement, we tracked how students performed in college English, subsequent college courses, and overall college completion under the new policy. We found that substantially more students completed college English compared to previous cohorts, with Asian, African American, Latinx, and Native American students’ completion of college English doubling or tripling. Upon taking subsequent college courses, students placing into college English under the new policy performed as well as those who had qualified for college English under the more restrictive policy. Overall college completion outcomes, including degree completion and meeting the criteria for transferring to 4-year universities, have generally improved and become more equitable since the 2011 policy change. These findings suggest that broadening access to college English can be a powerful lever for reducing racial and ethnic gaps in the completion of college English and may help to reduce gaps in the attainment of other, longer-term college completion outcomes.

Keywords: placement; disparate impact; college completion; equity; multiple measures.


Butte College and the Need for Disparate Impact Analysis in Writing Assessment

California’s Student Success Scorecard shows a stark divide between “college prepared” and “unprepared” students. When incoming community college students are designated prepared for college-level work in English and math, they go on to complete degrees, certificates, and transfer-related outcomes at a rate of 71% within six years. For students designated as unprepared and required to enroll in developmental courses, that figure is just 41% (California Community Colleges’ Chancellor’s Office, 2017). Unfortunately, most California community college students are assigned to the unprepared group. Statewide, more than 80% of incoming students enroll in one or more developmental courses (Mejia, Rodriguez, & Johnson, 2016). These courses, which we also occasionally refer to as “remedial” or “basic writing” courses, do not carry credit toward bachelor’s degrees.

These statistics are often seen as the inevitable result of students’ academic deficiencies. However, research has shown that the standardized tests community colleges rely on to assess college readiness are a large contributor to the problem. Though these tests are used to determine which students have access to college-level courses, they are simply not very good at predicting students’ performance in college. In one study, analysis of data from a statewide community college system revealed that placement test scores in reading/writing explained less than two percent of the variation in students’ first college-level English grades (Belfield & Crosta, 2012, p. 23). A study of a large, urban community college system estimated that 61% of incoming students could pass college English with a C or higher if allowed to enroll directly, but only 19% were designated college ready by the placement test (Scott-Clayton, 2012). While these studies are relatively recent, more than 20 years ago the Conference on College Composition and Communication (CCCC) Executive Committee (1995) released a position statement on writing assessment that acknowledged the limited usefulness of standardized multiple choice tests in assessing student writing and making decisions about their learning. The organization emphasized that such tests “misrepresent disproportionately the skills and abilities of students of color” (CCCC Executive Committee, 1995). In a 2016 white paper on placement reform, the Two-Year College English Association (TYCA) makes the point emphatically: “High-stakes testing, which even now dominates placement practices at two-year colleges, is unsound and unfair” (p. 3).

In response to such concerns over standardized tests, the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (2014) included more emphasis on fairness when codifying their new standards for educational assessment. However, writing assessment experts note that psychometric standards of fairness involve a certain “self-referential solipsism and silence on consequences” (Slomp, 2016) and do not ensure that the constructs being measured–whether via standardized tests or other writing assessments–are themselves fair to the “knowledges, languages, ways, and values” of all students (Cushman, 2016). Special issues of leading journals in writing studies have attempted to fill in these gaps, focusing around issues of ethics (Kelly Riley & Whithaus, 2016), social justice (Poe & Inoue, 2016), and diversity (Poe, 2014). Within this body of work, writing assessment experts show that fairness is “the first virtue of writing assessment,” and that statistical attention to disparate impact is key to ensuring that writing assessments will be used ethically (Elliot, 2016).

Borrowed from the legal field, disparate (or differential) impact refers to “the unintended racial differences in outcomes resulting from facially neutral policies or practices that on the surface seem neutral” (Poe, Elliot, Cogan, & Nurudeen, 2014, p. 593). Poe and Cogan (2016) emphasize that “differences in test scores alone do not constitute disparate impact; students come to college with different writing proficiencies. Rather, disparate impact occurs when a facially-neutral test places an unfair disadvantage on one group versus another.” Disparate impact analyses correct what Behm and Miller (2012), drawing on the work of Bonilla-Silva, identified as the use of color-blind frames

to explain why students from minority groups perform poorly on placement tests; to rationalize the disproportionate enrollment of minority students in developmental writing courses; or to deflect attention away from how a writing program and its various assessment practices may work unwittingly to maintain white privilege by reducing the opportunities of students of color. (p. 132)

Disparate impact analyses can help writing programs to assess and then remedy the differential effects of writing assessment, allowing programs to meet their goals for student learning without perpetuating disadvantage for various racial and ethnic groups (Poe & Cogan, 2016).

In one example of a disparate impact analysis, Inoue (2015) contended that we should be suspicious of any assessment–including the California State University system’s holistically scored Early Placement Test–in which writing by White students was consistently rated as college ready while writing by students of color was disproportionately rated as remedial (pp. 35-42). Rather than assuming that our constructs and measurements are race-neutral and that students of color just aren’t up to the job of producing college ready writing, Inoue made it clear that we should question our constructs, as well as the way in which they are being measured, when this measurement results in racially disparate outcomes. Inoue’s analysis led to the adoption of a contract grading system in California State University, Fresno’s writing program. Similarly, after finding that African American, Latinx, and Native American students placed into basic writing at higher rates and had lower graduation rates than and European American students, faculty at the pseudonymous Brick City University decided to change their overall approach to placement and curriculum. In the new system, all students would begin in college-level English, and the locally developed placement exam would be given at the beginning of the term to identify students who could benefit from additional support services (Poe et al., 2014, p. 603).

Disparate impact analyses are particularly needed at majority European American institutions where faculty members are disproportionately European American. Butte College is one such institution. A rural college at the base of the Sierra Nevada Mountain Range in California, Butte College sits–though without official acknowledgement by the College–on the ancestral homelands of the Maidu-speaking people of the region. A majority of the College’s service area is European American (75% in 2017), as are a majority of students at the College (57% in 2017). In 2010-2011, when this study began, 63% of the students at the College were European American, followed by Latinx (15%), Asian (6%), African American (3%), and American Indian or Alaskan Native students (2%). Sixty percent of students used the Board of Governor’s fee waiver for low-income students. Along with a majority-European American student body and service area, European American teachers are over-represented in the faculty ranks. In 2010-2011, European Americans represented 87% of all faculty, but just 63% of students. By 2017, the proportion of European American students dropped to 57% of students at the College, but European Americans still comprised 89% of the faculty teaching those students. By proportion, there are now 4 times as many students of color in the student population (43%) as there are teachers of color in the faculty population (11%). Such majority European American demographics, as Coleman, DeLong, DeVore, Gibney, and Kuhne (2016) argued, do not “just happen. They are a result of the cumulative legacies of violent, historical, cumulative, contemporary, and ongoing institutional exclusion and oppression” (p. 368), and they play a role in producing “disproportionate, repeated, and patterned failure for certain students in writing classrooms and programs” (p. 365).

In March of 2011, Butte College began to examine the role of writing assessment in perpetuating the “disproportionate, repeated, and patterned failure” of students of color when the English department replaced the Assessment and Placement Services (APS) English Writing test, a multiple choice test of grammar and sentence editing, with the ACT’s COMPASS English Placement Test, another multiple choice test of grammar and sentence editing. To assist faculty in setting cut scores for the new test, Butte College’s assessment officer, Eric Hoiland, examined ACT’s recommendations on cut scores, averaged cut scores from over a dozen other community colleges, and conducted a version of the Modified Angov method in which three to four faculty members for each course took the COMPASS and responded to the test questions as if they were a student who was "barely ready" for that course. Hoiland also had a sample of students at each level take the COMPASS assessment, and he compared those scores against students’ end-of-semester grades and the other data on cut scores. After faculty set cut scores, the College conducted a consequential validity study during weeks 5 through 7 of the first semester in which students had been placed using the new test. Faculty were asked to rate each student’s preparedness for the course into which they had been placed, using a 5-level Likert scale. In the process, Butte College faculty were surprised to see that under the cut score range they had set for scoring into college English (73-99), many more students were being classified as college ready. Instead of 23% of students who took the assessment test having access to the gateway college-level English composition course, 48% of students did. Butte faculty considered lowering the cut scores in order to maintain the prior ratio of college-ready/basic writing, but, conscious of the high rates of attrition and the inequities in developmental course sequences. they decided to let the new cut scores stand and see how students performed.

This article describes what happened–initially and longer term–using a disparate impact analysis framework to assess the department’s shifting policies. After establishing the disparate impact of placement, we tracked outcomes in four phases. In the first phase, we considered completion of college English, grades, and success rates for students in college English before and after the assessment change. We found that after the 2011 policy change, substantially more students completed college English across all ethnic groups, with gaps between groups narrowing. Students of color–who had fared the worst under the prior policy–saw the greatest gains for this outcome, with Asian, African American, Latinx, and Native American students’ completion of college English doubling or tripling under the new policy. Examining success rates and grade distributions after the 2011 policy change, we found that, among students who previously would have been placed into basic writing coursework, 40% earned As and Bs in the college-level course. In the second phase, we considered whether allowing more students to bypass basic writing could have meant that these students were less prepared for success in downstream coursework. Comparing success rates in downstream coursework for students placing into college English before and after the assessment change, we found that students placing into college English after the policy change succeeded at rates that were virtually identical to those of their counterparts placing into college English under the previous system. In the third phase of our analysis, we considered whether allowing more students to bypass basic writing may have impacted longer-completion outcomes for the incoming student population overall. We found that, across the entire first-time student population, all groups earned degrees at higher rates, with the exception of African Americans, whose degree completion rates remained the same. All groups met the criteria for transferring to four-year universities at higher rates after the assessment change, and the gaps in the rates at which students of different races/ethnicities attained this outcome narrowed. These findings suggest that broadening access to college English can be a powerful lever for reducing racial and ethnic gaps in the completion of college English and may help to reduce gaps in the attainment of other, longer-term outcomes. In the fourth phase, we have speculated on the results of Butte College’s new multiple measures placement policies, effective for students beginning in fall 2017, and a new co-requisite English course, expected to become available beginning in fall of 2018. (Prior to students beginning in Fall of 2017, students were placed primarily by their test scores, with high school grades playing a role only for students with scores near the cut-offs.) While these changes promise to further reduce the disparate impact of placement, we argue that the statewide multiple measures placement guidelines the department adopted in fall of 2017 do not go far enough toward creating equitable access to college English, that math policies need to change along with English policies, and that complying with an existing California educational regulation that protects students’ right to enroll in courses unless they are “highly unlikely” to succeed would produce more ethical and equitable placement and improved student outcomes at Butte College and in California community colleges system-wide.

The Disparate Impact of Placement

College completion outcomes are affected by multiple intersecting factors including race, socio-economic status, gender, sexuality, and other issues. However, in this study, we limited our examination to race/ethnicity because at Butte College, as in community colleges across the US, students of color are disproportionately placed into non-credit-bearing developmental courses. According to 2009 data from the National Center for Education Statistics, 62% of White community college students in the United States took remedial courses, compared to 71% of Black and Latinx students and 68% of Asian students (Witham, Malcom-Piqueux, Dowd, & Bensimon, 2015). More striking is the fact that Black and Latinx students were twice as likely to have to take three or more developmental courses than White students were (43% of Black and Latinx students vs. 22% of White students) (Witham, Malcom-Piqueux, Dowd, & Bensimon, 2015). A 2010 study showed that in California, more than half of Black and Latinx community college students who are placed into developmental coursework begin three or more classes away from a transferable, college-level math course. Students of color are also disproportionately represented in the lowest levels of English coursework: Compared to White students, 3 times as many Black students begin three or more classes below college English, and twice as many Latinx and Asian students do (White: 8%, Black: 25%, Asian: 19%, Latinx: 17%) (Perry, Bahr, Rosin, & Woodward, 2010). In Table 1, we see that Butte College follows these larger trends in placement.

Table 1

A chi-square analysis of these data was conducted to test the chances that the disparities in placement might have been due to random variation. Results of this analysis are included in Appendix A and show that the probability that the disparities between White students and other races/ethnicities occurred by chance was statistically insignificant (p < .05) for all groups excepting American Indian students under the old test. (The sample size for this group may have been too small to allow the chi-square analysis to detect disparity beyond what might be expected to occur by random chance.) With these results, we can reasonably conclude that the tests themselves explain the disparities in placement. Also, note these figures exclude students who took the assessment test but did not enroll at the College. They also exclude students taking no-cost or community-based courses. As seen in Table 1, under Butte College’s more restrictive cut score policy in 2010, 35% of White students were classified as college ready and given access to college English, a rate 2.8 times higher than for Black students. After the assessment change in 2011, all students had greater access to college English, and the gap between groups had narrowed, with White students’ access shrinking to just 1.6 times Black students’ access. However, students of color were still disproportionately excluded from the college-level course, which is required for students to earn an associate’s degree and/or transfer to a four-year university. In comparison, gaps in access between socioeconomic groups have been consistently smaller. For example, in Fall 2016, 47% of Pell recipients had access to college English versus 54% for students not receiving Pell funding, which is not a substantial gap. Pell grant receipt, while not a perfect indication of low income, is a more selective measure than the Board of Governor’s fee waiver and is the current measure Butte College uses to track outcomes for low-income students.

Students of color were also still disproportionately represented in Butte College’s lowest levels of basic writing. During the time of this study, students who placed below college English were required to take between one and four non-transferable English courses before taking college English, depending on their assessment results. In Fall 2012, Latinx students constituted 24% of the students who started three to four courses below college English but only 16% of the overall student population. Similarly, Asian students represented 15% of the students starting three to four courses below college English but only 5% of the Butte student population (California Community Colleges Chancellor’s Office, Management Information Systems Data Mart, Basic Skills Progress Tracker, n.d.).

These disparate placement rates unfairly disadvantage students of color because, for each additional developmental course required, students’ completion of college-level English and math courses declines (Bailey, Jeong, & Cho, 2008; Hern & Snell, 2010; Perry et al., 2010), and students who don’t complete college English and math requirements are ineligible to earn an associates degree or transfer to a four-year university. At Butte College, only 50% of students who began one course below college in writing in Fall 2010 completed college English within two years. For students who began two courses below college in writing, that number dropped to 27%. Among students starting three to four courses below college English, just 18% completed college English within four years (timeframe extended because of time required to progress through the basic writing sequence) (California Community Colleges Chancellor’s Office, Management Information Systems Data Mart, Basic Skills Progress Tracker, n.d.).

The implications of these statistics for students of color are troubling. In one study, initial placement was estimated to explain as much as 50% to 60% of the racial inequity in college completion outcomes (Stoup, 2015). Thus, while placement policies might seem facially neutral, as noted earlier, they can in fact result in unintended racial differences in outcomes (Poe et al., 2014).  

Phase 1: Completion of College English, Grades, and Success Rates in College English

The COMPASS placement test and policy were implemented in Spring 2011, but during the 2011-2012 year, many students enrolling in English courses were placed using the previous APS English writing test. These data therefore exclude the 2011-2012 year and focus on the four years preceding the change and the two years after full implementation. We looked at completion of college English across the entire population of incoming students–those placed into basic writing and those placed directly into the college level. Students enrolled in no-cost or community-based courses were excluded from this analysis. As Figure 1 makes clear, under the less restrictive policy, substantially more students completed college English within one year across all racial/ethnic groups. Students of color–who had fared the worst under the prior policy–saw the greatest gains: Native American and African American students’ completion tripled or nearly tripled, and Latinx and Asian students’ completion more than doubled. And, while gaps between groups persisted, they narrowed. European American students’ completion of college English was 2.9 times higher than African American students under the more restrictive policy; under the new policy, it was just 1.6 times higher.

Figure 1

Appendix B shows a chi-square analysis of these results. Under both assessment instruments, there were statistically significant differences between White students’ completion of college English and that of all other groups, excepting Asian students, at p < .05. Because there is a very low probability that the different rates of completing college English could have occurred through random chance, we can conclude there is still disparate impact for Native American, African American, and Latinx students under the new test.

We also examined two-year data for the 2012-2013 group to see whether the increased completion might be driven by the one-year timeframe of the study. After all, under the more restrictive policy, more students were placed into developmental coursework, which delayed their enrollment in college English. Would they catch up if given more time? We found that, while students in both groups made gains in year 2, completion of college English continued to be higher under the new policy (Fall 2012 to Summer 2014: Across every ethnic group, completion was 12 to 13 percentage points higher than under the more restrictive policy).

An additional question we considered was whether other factors could be driving the increase in completion. The change in placement at the college level meant a reduction in the number of students placed into basic writing courses. The biggest change was in the course two courses below college English. In Fall 2010, 33% of incoming students had been placed two courses below college English; in Fall 2012, 17% were (changes in the placement test appear to have shifted many of these students up to one course below college English). It is possible that these students contributed to the overall completion gains because more of them could have progressed through college English within a year under the new policy. Another possible factor is Butte’s accelerated developmental course. The accelerated course admits students who would otherwise have had to take a sequence of two basic writing courses, enabling them to progress to college English in just one semester. The course has substantially increased completion of college English among students at this placement level, a finding that is consistent with Hern and Snell’s (2010) discussion of how accelerated coursework improves student outcomes by reducing the “exponential attrition” built into the structure of prerequisite developmental sequences.

In the years 2012-2013 and 2013-2014, Butte offered 21 sections of accelerated English, enrolling 478 students. During those years, the College also offered 243 sections of college composition, enrolling 7,007 students. While we were not able to determine the precise degree to which the accelerated course was a factor in the college-wide completion gains, the relatively small scale of these offerings leads us to conclude that much if not most of the improvement was driven by the changes in placement policy.

With so many more incoming students allowed to skip basic writing coursework and enroll directly in college English, the first question most teachers will ask is this: How are they doing in the course? Are they unprepared for the rigor of the college level? Are they failing out at high rates? It is important, then, to look not only at overall completion rates (Figure 1) but at students’ performance within the college course. It should be noted that online sections were not included in the analysis of success rates. Many of these sections were outliers, and we wanted to leave out issues with course modality and its effect on student success. There were two to four online sections of college English offered each semester of this study.

In 2012-2013, the first year of full implementation, there appeared to be a modest decrease in average success rates across sections (students passing the course with a C or higher). Butte offered 119 sections of first-year composition that year, with a median success rate of 63%. The following year, the median success rate was also 63%. Prior to the policy change, the median success rates had varied from about 67% to 72% annually. So, by this measure, students do appear to be performing slightly less well in college English under the new policy.

However, it is important to note that Butte offered 83 to 119 sections of college English during each year of this study, and there was tremendous variability in success rates across sections. In 2013-2014, for example, success rates ranged from a low of 27% to a high of 97% across sections. Further, prior to the new policy, the median success rates varied by as many as 5 percentage points year to year, so a decline of 4 to 9 percentage points in the median is not a substantial deviation, particularly when considering the difference between sections within any given year.

To further investigate student success rates, we analyzed data from English instructors who had taught sections of college English before and after the policy change to determine whether their own success rates had changed (Fall 2007 to Spring 2014). Of these 21 instructors, eight had higher mean success rates after the placement change, three had no change in their mean success rates, and 10 had lower mean success rates. Among instructors whose success rates had increased or decreased, most saw a change of fewer than 10 percentage points, typical of the variation teachers normally see in their classes. Most interesting: Across all 21 instructors, the mean success rate dropped just 2.8 percentage points under the new policy, and the median less than 1 percentage point.

As an additional test of whether students were less prepared to succeed under the new policy, we looked at course grade distributions for students who placed into college English from different scoring ranges on the new test. The data we analyzed included all students who qualified for college English under the new placement test and enrolled in the course, including repeat enrollers. We were particularly curious about the performance of students who would have been assigned to basic writing under the old system but who were now allowed to begin directly in college English. While it was not possible to identify these students with certainty because the testing instrument had changed, we could estimate this group by considering the ratio of students placed into/out of college English under the old system. In using this method, we assumed that differences between the old and new tests and what they measured were less important than the increased access to college English afforded by the cut scores established for the new test. If Butte faculty had decided to narrow the cut score range and revert to previous placement ratios when they implemented the new test, students scoring between 73 and 88 would likely have been placed into remediation, while those scoring between 89 and 99 would likely have had access to college English. Table 2 shows these two groups’ grade distributions under the new placement policy.

Table 2

These data show that students testing into college English in the lower range of scores had slightly lower success rates than their higher-scoring counterparts (41% earned grades of D/F/FW/W compared to 36% of students in the higher-scoring range). They were also less likely to earn As. However, they did not markedly underperform in comparison to the higher-scoring students. We found it noteworthy that the lower scoring group did not receive a disproportionate number of Cs, as might have been expected if they were borderline college ready. In fact, 40% earned As and Bs in a course they would have been excluded from under prior placement ratios.

When considering rates of non-success among lower-scoring students, an important question to ask is whether they would have had better outcomes if required to first enroll in a basic writing course. Among students who began one course below college English in Fall 2010, just 39% completed college English within a year. This makes clear that, while we might be concerned that only 59% of the lower-scoring group succeeded in college English, requiring these students to enroll in a basic writing pre-requisite would not have led to more of them successfully completing the college English course.

It should be acknowledged that this study has not examined evidence of student writing, such as performance on a departmental exam or portfolios of student work; scoring of student writing samples would have added inter-observer reliability to this study. That said, if large numbers of students were unable to produce writing that met their college English instructors' expectations, we would see it in the data on course grades and rates of success. Taken together, course success rates and grades in college English suggest that dramatically increasing the number of students classified as college ready resulted in little change in students’ performance inside college English. While there was a modest decline in the aggregate success rate across sections, we are reluctant to conclude that this is evidence that students were less prepared to succeed. With success rates varying so widely across sections and a smaller drop in course success rates for instructors who taught both before and after the change, it’s clear that instructor-level effects–rather than simple student preparation levels–are playing a role in this outcome. In addition, more recent data show a rise in success rates: In Fall 2016, success rates in college English were 72%, equivalent to average success rates in the course before the assessment change. More study is warranted to see if this trend will continue and, if so, whether it reflects normal year-to-year variation and/or other factors, including recent equity-focused professional development efforts at the College. Regardless of variations in rates of success, more students are completing college English since the policy of broader access was implemented. In raw numbers, roughly 200 to 300 more students have completed college English each fall since the policy change (Michels-Ratliff & Henson, 2017).

Phase 2: Downstream Course Success for Students Placed into College English Before and After the Policy Change

The data showing that more students were completing college English have been shared widely at the College. However, many faculty–from English and other disciplines—have expressed concern that allowing more students to bypass basic writing may have resulted in students who were less prepared to do the writing required in their other college-level courses. While we were conscious that this concern stemmed from the uninformed belief that students’ performance on a single multiple choice test of sentence editing correlates with their writing abilities overall, we were curious to see if faculty members’ apprehensions were borne out by the data.

To get at the question of student preparedness for writing in other college-level courses, Butte College research analyst, Emelia Michels-Ratliff, selected high-enrolled courses that had college English as a pre-requisite or recommended preparation (i.e., “downstream” courses). These included courses in history, communication studies, political science, and English. Of these courses, History 8, History 10, and Communication Studies 2 require 2,500 words of writing, and English 11 requires 6,000 to 8,000 words. If allowing lower-scoring students to take college English had resulted in students who were less prepared for the demands of writing in their other college courses–if college English instructors had lowered their standards and passed unprepared students–we might expect to see lower success rates in downstream courses for the new group of students placing into college English. This is not what these data show. Figure 2 shows success rates in downstream courses for students who placed into college English before and after the assessment change. As shown in Figure 2, students who were placed into college English after the assessment change were not less successful compared to students who were placed into college English previously. Rather, they succeeded in downstream coursework at rates that were virtually identical to those for students placed into college English previously (Michels-Ratliff & Henson, 2017).

Figure 2

Phase 3: Longer-Term Outcomes for All First-Time Students

Phase 2 analysis was limited to examining downstream outcomes for students who were assessed as college-ready in English. However, we also wanted to see what impact the assessment change might have had on longer-term outcomes for the entire incoming student population, not just those placing into college English. Did the policy change allow more students to complete degrees and transfer to four-year universities? To investigate this question, we examined degree completion and transfer-readiness rates for all first-time students–those assessed as college-ready in English and those assessed as needing basic writing–in the three years before and the two years after the assessment change. Consistent with the rest of this study, students with previous concurrent enrollment or who enrolled only in no-cost or community-based courses were excluded from this analysis. As detailed earlier in this study, after the assessment change, a substantially larger share of students could access college English without having to complete a basic writing course first, and another share had access to accelerated English coursework that cut their time in basic writing in half. We were curious to see if removing these barriers early in students’ educational careers might have led to differences in longer-term outcomes college-wide. The longer-term completion outcomes were tracked for four years for all students in this phase of our analysis.

Comparing degree completion rates for all first-time students before and after the assessment change, we found that degree completion increased for all groups except African Americans, whose completion rate remained constant. Overall, there was a 25% increase in degree completion college-wide–from 9.86% of first-time students who started between Fall ’08 and Fall ’10 to 12.28% of first-time students who started between Fall ’12 and Fall ’13.

We also investigated the rates at which all first-time students became ready to transfer to four-year universities. Students are considered “transfer ready” if they complete at least 60 units of transferable coursework, have a GPA of 2.0 or higher, and have successfully completed both college-level math and college-level English with grades of C or higher. Overall, the College saw a 29% increase in transfer-readiness college-wide–from 14% of first-time students attaining this outcome before the change to 18% after the change. As shown in Figure 3, transfer-readiness rates increased for all groups.

Figure 3

A z-test of proportions was conducted to determine whether the pre- and post-test rates of transfer readiness differed significantly or fell within what might be expected to occur by chance. The z-test showed statistical significance between the pre- and post-change rates of transfer readiness for students overall, as well as for White and Latinx students. Sample sizes for the other groups may have been too small for the z-test to pick up on differences beyond what might be expected by chance. See Appendix C for these results.

As with completion of college English, students of color saw the biggest gains in transfer readiness: African American transfer-readiness increased 71%, Latinos 50%, American Indians 43%, Asians 23%, and European Americans 20%. Relative to European American students, attainment gaps for African-American and American Indian students have shrunk, and the gap between European American and Latinx students disappeared entirely.

Phase Four: Ongoing Changes in Placement

In the six years since Butte College began its experiment in placement, the landscape for placement has shifted statewide, and there is more widespread awareness of the problems that stem from using a single score on a placement exam to determine students’ educational fates. Directly acknowledging the limitations of the test in predicting college readiness, the manufacturer of COMPASS pulled their product from the market in November of 2016 (Bailey & Jaggars, 2016, p. 2). There is now a growing movement to use high school performance information to place students into college courses. High school performance data—in particular, students’ high school GPAs—have been shown to correlate more strongly with students’ actual performance in college courses than placement test scores (Bailey et al., 2016; Fagioli, 2016; Hodara & Cox, 2016; Multiple Measures Assessment Project [MMAP], 2016; Scott-Clayton & Stacey, 2015; TYCA Research Committee, 2016).

Following these trends, Butte College’s writing program decided to adopt the multiple measures “decision rules” recommended by California’s MMAP for placing students. By these rules, students with high school GPAs of 2.6 or higher or a qualifying test score will be eligible for college English, beginning with students enrolled in Fall of 2017 (MMAP Research Team, 2016). While data for students placed under this new policy were not available at the time of this writing, Michels-Ratliff (2017) predicts that, for students who can be placed using high school measures, access to college English will increase from approximately 49% to 71% of incoming students. In addition, while racial and ethnic gaps in access to college English will still exist, they will be smaller. Under the new placement rules, 76% of European American students are predicted to have access to college English, versus 70% of Asian students, 59% of American Indian/Alaskan Native students, 57% of Latinx students, and 55% of African American students (Michels-Ratliff, 2017).

Effective Fall of 2018, the Butte College writing program will also be adding a new co-requisite English course for students from the next lowest placement category. The co-requisite course is classified as a college English course and will meet the same requirements as the College’s regular college English course, but it will include more time in class with the instructor in order to help students be successful. Students will qualify for the course based on their test score or an 11th grade high school GPA of 2.3 or above. This course is predicted to increase access to college English for another 17% of recent high school graduates. If the projections are correct, 88% of incoming students will have access to a college English course after the new policies are implemented, and racial and ethnic gaps in access to college-level coursework will shrink even more. After implementation of the new course and multiple measures placement policies, 92% of European American students are predicted to have access to a college-level course in English, compared to 86% of Asian students, 79% of Native American students, 78% of Latinx students, and 74% of African American students (Michels-Ratliff, 2016).

Previous gains in completion of college English were accomplished by simply reducing the barrier and allowing more–and more diverse—students to enroll. No additional instruction was provided. Now, with the additional instruction time in the co-requisite course, evidence suggests that Butte College will see further reduction in inequality and more students completing college English. Nationally, co-requisite models have been shown to increase completion of college-level courses (Complete College America, 2016), particularly for students who score low on standardized assessments (Office of the Vice Chancellor of Academic Affairs, 2016). Co-requisite models are thought to support the contextual, non-linear way in which literacy develops. As Judith Rodby and Tom Fox (2000) concluded after the CSU, Chico English department replaced non-credit basic writing courses with a co-requisite model that allowed low-scoring students to take credit-bearing, college-level English: “1) One learns to do college writing by being in the context of college writing, not in some other context; and 2) literacy learning does not come in discrete levels” (p. 84). These principles of literacy development may explain why California colleges have seen positive results from co-requisite models and multiple measures placement policies that allow more students to begin in college-level English coursework. For example, when Solano College implemented high school grades in placement and added a co-requisite English course that allowed students from a lower placement category to enroll in college English with extra support, disparate impact in placement almost disappeared, and success rates in college English were unchanged (Henson, Hern, & Snell, 2017). These results suggest that the previous use of placement to funnel students into basic writing had underestimated students' capacity for college-level writing in English.

Discussion

Butte College’s experience demonstrates that broadening access to college English can be a powerful lever for reducing racial and ethnic gaps in the completion of college English and may help to reduce gaps in the attainment of other, longer-term outcomes. After increasing students’ access to college English in 2011, Butte College saw large, institution-wide increases in completion of the gateway college composition course, a critical early momentum point on the path to degrees and transfer to a four-year university. The data from Butte confirm other studies showing that a substantial number of students assigned to basic writing courses on the basis of standardized placement exams could, in fact, be successful if given access to a college-level course. That this problem went undetected for so many years is consistent with Scott-Clayton’s (2012) description of under-placement as “invisible to the naked eye”: “When a student is placed into a college-level course and fails there (an over-placement error), the fact that there has been a placement mistake is painfully obvious to all” (p. 35). On the other hand, Scott-Clayton (2012) writes, “Among students who do well in a remedial course, it may be difficult for an instructor (or even the student herself) to know whether they were appropriately placed or might have succeeded in the college-level course as well. In any case, when a student does well in a remedial course, it is unlikely to be perceived as a problem” (pp. 35-36). Butte College faculty’s previous lack of attention to the disparate impact of placement surely also played a role.

Some faculty might express concern about the initial modest drop in Butte’s aggregate success rates and the fact that, during the first two years of the new placement policy, lower-scoring students–the ones likely to have been placed into remediation in the past–were 5% points more likely to earn grades of W, D, or F in college English than higher-scoring students (41% vs. 36%). But given that 40% of the students in this scoring range earned grades of A or B, it would be hard to justify excluding them from the course. Further, California community college regulations protect students’ right to enroll in a course unless they are “highly unlikely” to succeed without a prerequisite (Policies for Prerequisites, Corequisites and Advisories on Recommended Preparation, 2018). Students with a 59% chance of success in college English are not “highly unlikely” to succeed in the course, making it problematic to require these students to take prerequisite coursework before being allowed to enroll in college English. And while we might still be concerned about their 59% success rate, this is a substantial improvement over the number of students who complete college English after starting out in a basic writing course. A return to the more restrictive policy of enforced pre-requisite coursework is clearly not in these students’ interest.

On the contrary, Butte’s experience reveals that increasing student access to college-level English may be a powerful lever for reducing equity gaps in both short- and longer-term outcomes. While all students saw greater completion of college English after the policy change, students of color saw the greatest gains, narrowing the gap between their completion of college English and White students’ completion. This is likely because students of color are much more likely to be classified as “underprepared” and denied access to college English based on placement tests assessing sentence-editing skills in standard English. In short, because students of color were more disadvantaged by the previous policy, they had more to gain from the change. Implementation of co-requisite English and multiple measures placement policies promises to add to these gains and further reduce disparity in completion of college English.

Longer term, students who place into college English do not seem to be less prepared for success since the policy allowing more students direct access to college English was implemented in 2011. Students who qualified for college English after the policy change are performing equally well in downstream courses when compared to students who qualified for college English before the assessment change. This suggests that allowing more students to bypass basic writing has not resulted in inferior preparation for writing in other courses. In fact, rather than harming students, the policy change may have actually allowed more students to complete longer-term outcomes. When measured across the entire first-time student population–those placing into college English and those placing into basic writing–rates of degree completion have increased modestly. Degree completion rates may also have been impacted by other efforts at the college, including first-year experience courses that emphasize the importance of associate’s degrees, and a vigorous process for ensuring that students complete the paperwork to receive a degree.

Overall, the College saw a 29% increase in transfer-readiness college-wide–from 14% of first-time students attaining this outcome before the change to 18% after the change. As with completion of college English, students of color saw the biggest gains in transfer readiness: African American transfer-readiness increased 71%, Latinos 50%, American Indians 43%, Asians 23%, and European Americans 20%. Relative to European American students, attainment gaps for African American and American Indian students have shrunk, and the gap between European American and Latinx students has disappeared entirely. These numbers are consistent with Stoup’s (2015) finding that initial placement explains a substantial portion of the inequities in completion of long-term outcomes. As Stoup’s model predicts, after reducing the inequities in students’ initial placement in English, Butte College saw a narrowing of gaps in the rates at which students of different races/ethnicities met longer-term criteria for transferring to four-year universities. While these data do not provide conclusive evidence that the assessment change is the sole or primary cause of increased transfer-readiness and degree completion rates, the assessment change is likely to have played a role in both. These findings suggest that broadening access to college English may have benefitted students longer term, particularly students of color, and that the prior policy had strong negative consequences for students’ educational progress. These consequences fell disproportionately on students of color because they were excluded from college English and required to take basic writing coursework at higher rates under the more restrictive policy.  

However, completion rates at Butte College continue to be low overall. Fewer than one in five first-time students becomes ready to transfer to a four-year university within four years of starting at the College. While there was a 29% increase on this metric after the English assessment change, math placement policies did not change, and these policies are generally the greater barrier to student completion. California’s 2017 Student Success Scorecard shows that, of first-time students who started at Butte College in 2014-2015 and completed six units after attempting any math or English in their first year, 56.3% completed a college-level course in English in their first or second year, compared to just 28.6% of the same cohort completing a college-level course in math within that same timeframe (California Community Colleges’ Chancellor’s Office, 2018). At Butte College and throughout California, access to math courses that count towards a bachelor’s degree is still highly restrictive. Even under new multiple measures placement criteria (effective Fall 2017), the majority of Butte College students are still blocked from access to math courses that count towards bachelors’ degrees, with disproportionate impact in access for Native American, Latinx, and African American students (Michels-Ratliff, 2016). This is troubling because, similar to findings in English, evidence suggests that a majority of students can be successful in college-level math–particularly, college statistics–when given access and additional support (Logue, Watanabe-Rose, & Douglass, 2016; Henson, Hern, & Snell, 2017). These findings point to the need for Butte College to consider changing policies concerning math placement and remediation in order to ensure that comprehensive reform efforts underway at the college do not continue the legacy of disparate impact for students of color.

Results for students taking English courses under Butte College’s new multiple measures placement policies were not available at the time of this writing. However, the disproportionate exclusion of students of color from college English is predicted to continue under these policies. To correct the issue, Butte College should align its math and English placement policies with California’s Title V regulation protecting students’ right to enroll in a course unless they are “highly unlikely” to succeed without taking a prerequisite. The current state-recommended MMAP placement rules adopted at Butte College do not align with this standard. Students with 11th grade high school GPAs between 1.9 and 2.6 and grades of C or higher in 11th grade English are predicted to pass college-level English at a rate of 62% (MMAP Research Team, 2016, p. 7). Yet under the current placement rules, students with GPAs below 2.3 will be excluded from a college-level English course (unless their test scores qualify them to enroll). This exclusion is a result of setting placement criteria to maximize course success for the limited number of students granted access, rather than setting placement criteria so as to maximize completion of college-level courses for all students. In developing their recommended placement rules, MMAP researchers were asked to provide placement criteria that would maintain or improve existing success rates within college-level courses, limiting college English access to just those students whose average predicted pass rate is at least 70%, and limiting co-requisite English access to students with a predicted success rate of 65%. These pass rates represent a “highly likely to succeed” standard for determining access, not the “highly unlikely to succeed” standard for barring access specified by the Title V state regulation. As a result, some Butte College students with a 62% chance of passing college English–similar to current pass rates in the course–will be required to take one or more prerequisite English courses, substantially reducing their chances of completing college English and longer-term outcomes. Most troubling, this group will disproportionately consist of students of color (e.g., 8% of European American students required to start below a college-level course vs. 26% of African American students required to start below a college-level course).

The disproportionate exclusion of students of color from college-level courses is highlighted in the Association of American Colleges and Universities’ (Witham et al., 2015) publication America’s Unmet Promise: The Imperative for Equity in Higher Education. Researchers from University of Southern California’s Center for Urban Education explain the stakes, writing that disproportionately excluding students from college-level courses “contributes to further disparities…in retention and completion rates, graduate school participation rates, and access to opportunities for deep and engaged learning throughout their postsecondary careers” (Witham et al., 2015, p. 17). Placement and remediation policies appear, on their face, to be race neutral, with a veneer of scientific accuracy provided by the processes through which colleges validate cut scores and set placement criteria. But students of color are being disproportionately excluded from college-level courses based on criteria that do not accurately reflect their ability to succeed, and this exclusion has very real and measurable consequences for their educational progress. To correct these issues, colleges should apply California’s existing standard for requiring students to take prerequisite courses. That is, students should have access to college-level courses–including ones with co-requisite support–unless a rigorous analysis of prior high school performance and other multiple measures shows that they are “highly unlikely” to succeed without a prerequisite course, particularly when there is disparate impact for underprivileged racial/ethnic groups. This standard meets the criteria for a theory of ethics in writing assessment laid out by David Slomp (2016); in particular, it holds institutions to “actionable standards of ethical practices” and has “an ecological orientation . . . that pays attention to the role assessment plays both within broader systems of education and within society as a whole.”

The intent of our policies may not have been exclusionary. But given the evidence that placement into remediation leads to worse outcomes, we need to acknowledge and address the role our assessment policies play in perpetuating stark racial and ethnic disparities in college completion. The authors of America’s Unmet Promise recognize that, given the complex roots of educational inequity, “No single reform initiative can address all of these challenges” (Witham et al., 2015, p. 3). But they urge practitioners to confront inequities within their sphere of influence. We must, they write, “be willing to disrupt the current systems of higher education and take responsibility for those aspects of inequality that are under our control” (Witham et al., 2015, p. 3).

Postscript

After this article was completed, the California legislature passed a law that aligns with the placement principles we advocate. Among other specifications, AB 705 requires community colleges to follow the “highly unlikely to succeed” standard for barring access to college-level math and English. Further, colleges must ensure that students’ initial placement in English and math gives them the best chance of completing transferable, college-level courses. Initial MMAP data show that under AB 705 criteria, all or close to all incoming students will have access to college-level English and college statistics, with or without corequisite support. Colleges must fully adhere to AB 705 by Fall 2019 (California Community Colleges’ Chancellor’s Office, 2018).

Author Note

Leslie Henson, Ph.D., is an English instructor at Butte College and a Storyteller for the California Acceleration Project (hensonle@butte.edu). Katie Hern, Ed.D., is an English Instructor at Chabot College and Executive Director of the California Acceleration Project (khern@chabotcollege.edu). The authors began collaborating in 2011, when Henson participated in the CAP Community of Practice. Special thanks to Myra Snell, Eric Hoiland, Wim McSpadden, and Emelia Michels-Ratliff for their assistance.

Earlier versions of some sections of this article appeared in a previous article –“Let Them In: Increasing Access, Completion, and Equity in College English” – published in the RP Group’s Perspectives newsletter (November 2014).

 

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Bailey, T., Bashford, J., Boatman, A., Squires, J., Weiss, M., Doyle, W., . . . Young, S. H. (2016). Strategies for postsecondary students in developmental education – A practice guide for college and university administrators, advisors, and faculty. Washington, DC: Institute of Education Sciences, What Works Clearinghouse.

Bailey, T., & Jaggars, S. (2016). When college students start behind. The Century Foundation. 1-15. Retrieved from https://tcf.org/content/report/college-students-start-behind/

Bailey, T., Jeong, D. W., & Cho, S. W. (2008) Referral, enrollment, and completion in developmental education sequences in community colleges. CCRC Working Paper No. 15 (Rev. Nov. 2009). New York: Community College Research Center, Teachers College, Columbia University. Retrieved from http://ccrc.tc.columbia.edu/publications/referral-enrollment-completion-developmental-education.html

Behm, N., & Miller, K. (2012). Challenging the frameworks of color-blind racism: Why we need a fourth wave of writing assessment scholarship. In A. B. Inoue & M. Poe (Eds.), Race and writing assessment (pp. 127–138). New York, NY: Peter Lang.

Belfield. C., & Crosta, P.M. (2012). Predicting success in college: The importance of placement tests and high school transcripts. CCRC Working Paper No. 42. New York: Community College Research Center. Retrieved from http://ccrc.tc.columbia.edu/publications/predicting-success-placement-tests-transcripts.html

California Community Colleges’ Chancellor’s Office. (2017). Student Success Scorecard. Statewide. Sacramento, CA. Retrieved from: http://scorecard.cccco.edu/scorecardrates.aspx?CollegeID=000#home

California Community Colleges’ Chancellor’s Office. (2018). What is AB 705? Sacramento, CA. Retrieved from: https://assessment.cccco.edu/ab-705/

Conference on College Composition & Communication Executive Committee. (1995). Writing assessment: A position statement. Urbana, Illinois: National Council of Teachers of English. Retrieved from http://www.csun.edu/~krowlands/Content/Academic_Resources/Composition/Responding/Writing%20Assessment%20A%20Position%20Statement.htm

Coleman, T. J., DeLong, R., DeVore, K. S., Gibney, S., & Kuhne, M. C. (2016). The risky business of engaging racial equity in writing instruction: A tragedy in five acts. Teaching English in the Two-Year College, 43(4), 347-370.

California Community Colleges Chancellor’s Office, Management Information Systems Data Mart, Basic Skills Progress Tracker. (n.d.). Butte College. Retrieved from http://datamart.cccco.edu/Outcomes/BasicSkills_Cohort_Tracker.aspx

Complete College America. (2016). Co-requisite remediation: Spanning the divide – breakthrough results fulfilling the promise of college access for underprepared students. Retrieved from http://completecollege.org/spanningthedivide/#the-bridge-builders

Cushman, E. (2016). Decolonizing validity. The Journal of Writing Assessment, 9(1). Retrieved from http://journalofwritingassessment.org/article.php?article=92

Elliot, N. (2016). A theory of ethics for writing assessment. The Journal of Writing Assessment, 9(1). Retrieved from http://journalofwritingassessment.org/article.php?article=98

Fagioli, L. (2016). Common assessment regional meeting: MMAP report. Retrieved from http://rpgroup.org/Portals/0/Documents/archive/CAI_Regional_Meeting%20_Presentation_7.29.16.pdf

Henson, L., Hern, K., & Snell, M. (2017). Up to the challenge: Community colleges expand access to college-level courses. Sacramento, CA: The California Acceleration Project.

Hern, K., & Snell, M. (2010). Exponential attrition and the promise of acceleration in developmental English and math. Perspectives. Berkeley, CA: Research and Planning Group. Retrieved from http://www.rpgroup.org/resources/accelerated-developmental-english-and-math

Hodara, M., & Cox, M. (2016). Developmental education and college readiness at the University of Alaska (REL 2016–123). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest. Retrieved from http://ies.ed.gov/ncee/edlabs

Inoue, A. (2015). Antiracist writing assessment ecologies: Teaching and assessing writing for a socially just future. Fort Collins, CO: WAC Clearinghouse.

Kelly Riley, D., & Whithaus, C. (2016). A theory of ethics for writing assessment [Special issue]. Journal of Writing Assessment, 9(1). Retrieved from http://journalofwritingassessment.org/article.php?article=99

Logue, A. W., Watanabe-Rose. M., & Douglas, D. (2016). Should students assessed as needing remedial mathematics take college-level quantitative courses instead? A randomized controlled trial. Educational Evaluation and Policy Analysis, 38(3), 578-598.. doi:10.3102/0162373716649056

Mejia, M. C., Rodriguez, O., & Johnson, H. (2016). Preparing students for success in California’s community colleges. San Francisco, CA: Public Policy Institute of California. Retrieved from http://www.ppic.org/main/publication.asp?i=1215

Michels-Ratliff, E. (2016). Understanding multiple measures placement: A retrospective analysis and look to the future [Unpublished PowerPoint]. Oroville, CA: Butte College.

Michels-Ratliff, E., & Henson, L. (2017, March). Presentation at Acceleration Across California, the first annual conference of the California Acceleration Project. Available at http://accelerationproject.org/Acceleration-Across-California

Multiple Measures Assessment Project. (2016). Presentation to the RP Group, April 8, 2016. Berkeley, CA: The Research and Planning Group for California Community Colleges. Retrieved from http://rpgroup.org/Our-Projects/All-Projects/Multiple-Measures/Presentations-and-Webinars

Multiple Measures Assessment Project Research Team. (2016). English placement models for the Multiple Measures Assessment Project — Phase 2 (Rev. November 2016). Retrieved from http://rpgroup.org/Portals/0/Documents/Projects/MultipleMeasures/DecisionRulesandAnalysisCode/English-Decision-Trees-1_11_2016.pdf

Office of the Vice Chancellor of Academic Affairs. (2016). Co-requisite remediation full implementation 2015-2016. Tennessee Board of Regents. Retrieved from http://www.aacc.nche.edu/Resources/aaccprograms/pathways/Documents/TNBoardRegents_FullImplementation2015-2016.pdf

Perry, M., Bahr, P. R., Rosin, M., & Woodward, K. M. (2010). Course-taking patterns, policies, and practices in developmental education in the California community colleges. Mountain View, CA: EdSource. Retrieved from http://edsource.org/wp-content/publications/FULL-CC-DevelopmentalCoursetaking.pdf

Poe, M. (2014). The consequences of writing assessment [Special issue]. Research in the Teaching of English, 48(3), 271-275.

Poe, M., & Cogan Jr., J. A. (2016). Civil rights and writing assessment: Using the disparate impact approach as a fairness methodology to evaluate social impact. The Journal of Writing Assessment, 9(1). Retrieved from http://journalofwritingassessment.org/article.php?article=97

Poe, M., Elliot, N., Cogan Jr., J. A., & Nurudeen Jr., T. G. (2014). The legal and the local: Using disparate-impact analysis to understand the consequences of writing assessment. College Composition and Communication, 65(4), 588-611.

Poe, M., & Inoue, A. B. (2016). Toward writing as social justice: An idea whose time has come [Special issue]. College English, 79(2), 119-126.

Policies for Prerequisites, Corequisites and Advisories on Recommended Preparation, 5 CA ADC § 55003 (2018).

Rodby, J., & Fox, T. (2000). Basic work and material acts: The ironies, discrepancies, and disjunctures of basic writing and mainstreaming. Journal of Basic Writing, 19(1), 84-99.

Scott-Clayton, J. (2012). Do high stakes placement exams predict college success? CCRC Working Paper No. 41. New York: Community College Research Center. Retrieved from http://ccrc.tc.columbia.edu/publications/high-stakes-placement-exams-predict.html

Scott-Clayton, J., & Stacey, G. W. (2015). Improving the accuracy of remedial placement. New York, NY: Columbia University, Teachers College, Community College Research Center. Retrieved from http://67.205.94.182/publications/improving-accuracy-remedial-placement.html

Slomp, D. (2016). Ethical considerations and writing assessment. The Journal of Writing Assessment, 9(1). Retrieved from http://journalofwritingassessment.org/article.php?article=94

Stoup, G. (2015, October). Using data to identify emergent inequities and the effective practices to address them. Presentation delivered at Success and Equity: Regional Conference for Collaboration. Modesto Junior College.

Two-Year College English Association Research Committee. (2016). TYCA white paper on placement reform. Urbana, IL: NCTE.

Witham, K., Malcom-Piqueux, L. E., Dowd, A. C., & Bensimon, E. M. (2015). America’s unmet promise: The imperative for equity in higher education. Washington, DC: American Association of Colleges and Universities.

 

Appendix A

 

 

Appendix B

 

Appendix C