On April 25, 2024, SFUSD began the second phase of community feedback regarding the Resource Alignment Initiative (RAI), which is the name given to the process that will lead (among other things) to proposed school closures or mergers.

In the first phase of community feedback (see slides from the first town hall in March), the district presented draft criteria under three themes of Equity, Excellence and Effective use of resources and asked the community to rate the importance of each of these criteria via a web survey. It also held in-person consultations that were more open-ended and discussion-based.

During the second town hall, the district:

  • presented the results of the first survey,
  • explained how these results were used to narrow down the criteria to 3 or 4 in each of the 3 themes,
  • presented the second survey, and
  • explained how the second survey results will be used.

The second survey itself gives a definition of the specific way each criterion will be measured and asks respondents to assign a relative importance to the criteria within each theme.

Arguably, the district should have explained the concrete definition of the criteria and how they would be used before the very first survey.

At least, now that this information is available, it is possible to use those definitions as well as the data made available publicly by the district to get an idea of what the application of these criteria would do in practice. This is the purpose of this page (which is a work in progress, currently limited to a few criteria and to the data regarding elementary schools).

Some major points of concern (tl;dr)

  • How exactly SFUSD plans to “combine” the heterogeneous metrics into a single score by school, even with weights from the community survey, remains mysterious.

  • It is not clear if all metrics can be computed for all schools (missing data, utility of geographical metric for citywide schools, etc.)

  • Grouping all particular programs/pathways into one metric means for example that special education numbers will be overshadowed by the much larger language program enrollment.

  • Metrics under school excellence, if left unadjusted, primarily reflect the racial/socio-economic composition of the school rather than anything the school is specifically doing different from other schools.

  • The amount of year-to-year variation in the outcome metrics made available to the DAC (perhaps in part due to the pandemic) means that those 3 years of data may not be sufficient for a good measure of school-to-school differences.

Overview of the criteria

One slide from town hall #2 in particular illustrates what SFUSD plans to do with the results of this round of feedback (reproduced below).

The colored balls in each bin here represent the output of the survey, how much respondents weigh the criteria in each of the three themes.

The metrics used by the district for each criteria are described here. It is not clear how those metrics can be combined in a composite score as illustrated in the slide. For example, the three criteria under Equity will measure (1) how far a school is from other schools, (2) how many of its students are in language, special education or career technical programs, and (3) how many of its students live in disadvantaged neighborhoods. Some common scale of measurement is needed before it is even possible to weigh them according to survey feedback. One possibility would be that the district would just rank all schools for the metric, then do a weighted average of the ranks for the three metrics under each theme. Perhaps the composite score for each school is then an equal-weight average of the three theme scores? This is also unclear.

The purpose of the above is really to show how the process description in the district’s presentation leaves a lot of questions, and how there are choices of how to aggregate these metrics that the district will make (in addition to their definition in the first place), which could influence the ranking as much as the community feedback itself. Especially given that the district seems to be defining/disclosing the process piecemeal as the consultation is ongoing, this means the resulting ranking cannot really be described as “produced by the community”: it will really be a construction of the district RAI team with some selected points of community feedback incorporated in.

Elementary: School Access

The district defines this metric as the “average distance between the three closest schools with the same grade span”. The map below shows the result of this calculation for the 58 elementary schools with attendance areas (those attendance areas are also outlined on the map). For each point representing a school, the color indicates if it is closer (redder points) or further (bluer points) from the nearest three other schools.

Some notes:

  • The areas with more red points seem concentrated in the northeast corner of the city and in the center-east section from Market/Castro to the Mission and Potrero Hill.

  • I did not include the four citywide elementary schools or the eight (also citywide) K-8 schools, but it is unclear if the district intends to do so for this metric. On the one hand, some of these schools are fully dedicated to language immersion (for example) so cannot act as a general education neighborhood school. On the other hand, if the district doesn’t rank all schools for each of the criteria, it becomes a problem for the process they seem to have committed to.

Elementary: Program Access

The district defines this metric as the “percentage of students in each school participating in Language programs, Special Education programs, or Career Technical Education and Pathway programs.” For elementary schools, only the first two types of programs are relevant.

The DAC materials Google Drive includes a “Data Deck” regarding special education and language pathways, which mentions the following districtwide figures:

  • Out of 48,785 students in SFUSD in 2023, 6483 (13.3%) have IEPs (Individualized Education Programs). Of those, around 23.7% are in Special Day Class or SDC (so around 3% of total enrollment) and 57.1% in Resource Specialist Programs or RSP. (However, elsewhere in the slides it is mentioned that there are 2221 students in SDC which is more than 23.7% of 6483, so one of these figures has to be incorrect.) The figures are not broken down by grade range.

  • Almost one third of enrollment capacity for elementary schools is in the language programs.

For the special education side, it is not clear whether the metric as stated above includes only students in SDC (which is a separate admission pathway and only available at some schools), also includes RSP (which is available at all schools and where students are mostly in general education classrooms) or all students with IEPs.

However, due to the far larger enrollment into language programs than special education at the elementary school level, the ranking for this metric is probably going to be dominated by language programs.

As an illustration (not an exact computation of the metric), here is a graph of the fraction of available seats for 2024-2025 kindergarten admissions in general education, language programs and SDC by school. Here I also include the citywide and K-8 schools, since many of the language programs are housed within those schools. I do not include the newcomer schools. The data comes from the “Main round 2024 transitional requests by seat” data file found here. The schools are sorted by fraction of seats in general education, and all schools with language programs (orange) occur below all schools without them.

I mention that this is an approximation, since these are the seats made available for the enrollment process for kindergarten, which may not reflect the actual enrollment patterns for the whole school. Still, the proportion of SDC seats overall (3.3%) is consistent with the district figures cited above.

In the first town hall, Program Access was defined as “the availability of educational programs in a neighborhood”. Despite this definition, specifically the mention of neighborhoods, the metric proposed by the district here doesn’t address whether those programs are well-distributed across the city.

Elementary: Historical Inequities

This metric is defined as “the average amount of neighborhood opportunity, as measured by the Opportunity Insight Lab’s upward mobility index, experienced by students in the school. A school serving a higher percentage of students from neighborhoods with lower historical opportunity levels would be less likely to be identified as a candidate for co-location, merger, or closure.”

Despite finding the Opportunity Insights website (https://opportunityinsights.org/data/), I could not find an “upward mobility index” listed. Their data page includes one section about upward mobility rates, but that data appears to be available at too coarse a level (“commuting zones”) to be useful for intra-city comparisons.

It is also unclear why the district here uses a neighborhood-level metric applied to students in each school, based on the historical socio-economic status of their neighborhood of residence, vs. using either (a) a student-level metric for the students in the school (such as the proportion of socio-economically disadvantaged students, which is already reported by the district) or (b) a measure related to the neighborhood where the school is located. To be clear, the concern is that applying historical neighborhood metrics to individuals who may or may not match the historical demographics of those neighborhoods could lead the district to miss the mark on this equity metric.

The distribution of socio-economically disadvantaged students in each school, on the other hand, has already been very well described (for example here), so it is not necessary to redo this analysis, but again this is not what the metric proposed by SFUSD is measuring.

Elementary: School Culture and Climate

This metric is defined as “the percentage of families, staff, and students responding favorably to survey questions about a sense of belonging, safety, or academic support for learning”.

The DAC materials Google Drive includes a file named ES Profiles - Data for Sharing.xlsx that describes enrollment figures and outcomes for each of the K-5 schools, which is relevant data for this metric as well as the other two under the Excellence theme (SBAC and socio-emotional development indicators).

The spreadsheet lists only one measure of School Culture and Climate per respondent group and school, which may be the average of the different elements in the survey (sense of belonging, safety, etc. as noted above). It includes 3 years of data labelled Y1, Y2 and Y3. It is not specified which years these labels represent (in general, the data is not very clearly documented). There is however a large drop in the districtwide mean culture and climate score (from 82% to 72%) between Y1 and Y2, and a further 1% decrease between Y2 and Y3, suggesting perhaps these data are from 2019-2020 (pre-pandemic) to 2021-2022. The number of surveys respondents is only available for Y3 (Y3_N column).

As the graph below shows, the number of respondents varies widely in year 3, especially for the family category, with even a few large schools having fewer than 20 responses (20 responses is the horizontal dashed line across the graph) or up to 600. Many schools also have few staff respondents. Note that I excluded the Lee Newcomer School due to its very low enrollment.

The number of student respondents is more proportional to school enrollment (the blue trend line represents approximately a 25% response rate). One large school (Taylor) has a substantially lower response rate.

Unfortunately, two schools (Rosa Parks and Malcolm X) are missing student survey data for this metric. This raises the question again of how the district will compute these metrics in every school if there is missing data for some of them.

Looking only at the student scores for school culture and climate (average of all students per school), it appears that the score has a rather strong relationship with the proportion of socio-economically disadvantaged students in the school (column SES_p in the enrollment data), as shown in the graph below. That effect in fact becomes stronger for years 2 and 3.

In fact, a simple statistical model of the data shows that 40% of the variation in the scores is due to the district-wide variation between the years, and 65% is explained when considering both year and proportion of SED students (i.e. the trend lines shown in the graph above). Adding a “school effect” (a random effect, for any statistician reading this) to represent any systematic performance of the school above or below expectations, accounting for its proportion of SED students, only increases that percentage of explained variation from 65% to 75%, with the remaining (“unexplained”) 25% representing the amount of year-to-year variability at the school level independent from the district trends.

Note: In the original Excel file from the DAC materials, some columns indicate “trends” based on this inter-annual variation over three years of data, for every subgroup of every school, even when those subgroups have as few as 1 respondent. Any statistician the district could consult would probably advise for more caution than interpreting every variation with this few data points as a “trend”.

The above is just a very simple model including one demographic variable (proportion of SED). Clearly other differences in school demographics may have a role. But at least, it shows evidence that more of the differences in the raw scores from school to school are explained by socio-economic status of the student population than by any consistent effects the schools themselves could have.

Elementary: Socio-Emotional Development

This metric is defined as “the percentage of students responding favorably to survey questions related to social awareness, self-management, growth mindset, or self-efficacy”.

The Socio-Emotional Learning (SEL) survey data for K-5 schools are found in the same dataset as described before in the School Culture and Climate section, and in fact the numbers of respondents in each school match for both surveys (with data missing for the same two schools).

Unlike for the school climate surveys, here the four indicators (social awareness, self-management, growth mindset, self-efficacy) are reported separately in the data sheet. The four indicators tend to vary together between schools and years (correlation coefficient of 0.55 to 0.86 depending on the pair of indicators considered). Therefore it seems reasonable to use the mean of the four indicators as an overall SEL score as the metric effectively proposes.

Just like school culture and climate above, we can look at the variation of SEL score between survey years and as a function of the percentage of socio-economically disadvantaged (SED) students in the school.

For this metric, the scores also decrease districtwide from Y1 to Y2 and Y3, but not as much as for the school culture and climate. 8% of the variation in the data shown above is due to the districtwide trend, 50% of the variation is explained if we also consider the percentage of SED students in the school, and the addition of school effects (independent of the percentage of SED students) further increases this percentage to 68%.

In fact, if we put aside the districtwide changes from one year to the next, the two metrics (School Culture and Climate and Socio-Emotional Learning) show a very similar pattern: the percentage of SED students alone explains about 2.5 times as much of the school-to-school variation as any consistent (across years) school effects that remain after controlling for the percentage of SED students. In other words, if the raw percentage scores are used (as the district proposes when explaining the criteria), both criteria will – redundantly – reflect the socio-economic composition of the student population much more than they would reflect any particular characteristic of the school itself.

Elementary: Academic Performance

This metric is defined as “state assessments of English Language Arts and Math performance and growth”.

The example given in the survey is: “School A has a high performance level and has maintained that level over multiple years. School B has a high performance level and has increased it over multiple years. School B shows greater growth and is less likely to be identified as a candidate for co-location, merger, or closure.”

This particular example avoids the question of what to do if one school has higher performance and lower growth or vice versa. The metric as defined is really composed of two metrics with no specification of how to weigh them. It also does not mention over how many years the growth is calculated.

Just as for the previous two metrics, the data for SBAC (California “Smarter Balanced Assessment”) proficiency level (% of students meeting or exceeding standard) is provided for English (ELA) and Math for three years (labelled Y1, Y2 and Y3) in the ES Profiles - Data for Sharing.xlsx file provided to the DAC. Here, the data is available for all elementary schools. (I still exclude Lee Newcomer School due to the small population.)

As for the previous two metrics, we can fit a simple model to see how much of the variation in SBAC scores between schools and years is explained by student demographics, in this case including not only the percentage of socio-economically disadvantaged students, but also the percentages of Black, Hispanic and Asian students.

Unlike the previous metrics, there is no systematic year-to-year variation at the district level. However, the amount of variation between schools explained by demographics alone is greater (86% for ELA and 84% for Math), with an additional 8 to 10% (94% total in both cases) explained by school effects independent of demographics. The graph below shows the actual proficiency scores compared to the expectation from school demographics alone. The dots for each school are connected by a vertical black line to provide a sense of the variation from year to year. Points corresponding exactly to model expectations would be on the diagonal dashed line.

EDIT: A previous version assumed that “growth” as defined in the metric represented the change over time of the SBAC scores for a school. I later found out (from Paul Gardiner’s SFEDup blog) that the state is developing a metric for the average growth in SBAC score for a cohort of students at each school, as described here. The state released a version of the metric for informational purposes only (to demonstrate the model) in 2021, and the first version that can be used for evaluation will be released in the Fall 2024 based on SBAC data from 2022 to 2024.

The following graph shows the growth model “non-official” results from the 2021 state model for SFUSD K-5 schools vs. the percentage of socio-economically disadvantaged students.

Even though schools with more disadvantaged students show lower growth on average, there is more variation from school to school and this metric is much less correlated with racial and socio-economic composition of the student population (the same demographic variables that explained over 80% of the mean SBAC proficiency rates explain less than 10% of the growth score).

However, the first official growth score metric will only be published after SFUSD makes recommendations for school closures. Even if they were to make recommendations based on the 2021 data (which might not be allowed), there is a risk that the one data point for each school may not represent a consistent effect, as only more years of data would tell.

Elementary: Family Choice and Demand for School / Enrollment

The metric for family choice and demand for school is defined as “the percentage of applicants ranking the school as one of their top three choices in their school application”. This information can be obtained readily from the Annual Assignments Highlights more specifically the Main choice results by round file.

The graph below shows the number of top 3 requests (rather than the percentage, but it is proportional) for 2024-2025 kindergarten enrollment by school.

The metric for student enrollment is defined as “the school’s 2023-2024 school year enrollment compared to its ideal enrollment”. I could not find information on “ideal enrollment” by school. In principle, this metric should be related to family demand since schools with high requests should be full. But it depends how ideal enrollment is defined (building capacity? staffing levels? etc.)

Before obtaining clarity on this point, we could nevertheless look at the assignment demand (here represented as the number of top-3 requests by seat available), the number of seats available and the percentage of those seats assigned by round 1. In the graph below, general education and language enrollments are combined, and SDC enrollments are excluded since they represent much smaller classrooms.

This is a somewhat complex figure, but a few notes:

  • The number of seats (\(x\) axis) is clustered around numbers corresponding to 1, 2, 3, 4 kindergarten classes, not surprisingly. Lau stands out with what looks to be 5 full classes.

  • The most requested schools relative to the number of seats can be found in all school sizes.

  • The color scale represents the percentage of seats that were assigned in round 1. It goes above 100% as some schools report more assignments than available seats. Purple, then red colored points represents increasing levels of under-assignment.

  • I want to be very clear that the color scale does not represent over/underenrollment necessarily, as these are only Round 1 assignment results.

Here is a different version that shows assignments vs. number of seats available.

Reference materials

Here are some useful links to public data made available by the district, some of which is used in this analysis.