Not enough ESL students in training data.

Schools we received data from did not have many ESL students.

Model doesn’t work well on students with accents.

Yes, we can acquire more training data from ESL students or students with accents.

If we can’t, we should disclose to schools that our model may not work well for ESL students or students with accents.

Students with the most suspensions and lowest attendance rate are Black boys.

Implicit racial bias in schools causes teachers to suspend Black students more often than white students for the same infractions.
In the schools we received data from, Black students tend to live much farther from the school and their buses often arrive late to school.

Model highlights a disproportionate number of Black students as at-risk.
If product is used to categorize students in academic tracks based on predicted outcomes, model could incorrectly assign Black students to lower academic tracks.

No, we can alter the dataset to amplify examples of Black students with no discipline record and high attendance, but this may cause some Black students to not receive the additional support they need.
The underlying bias must be disclosed and addressed in the design of the app.

Schools should be made aware that a key group of students (Black students) has disproportionately received higher rates of suspension and lower attendance rates.
Make sure you deeply understand why this is happening rather than accepting it as the norm.
Incorporate this understanding into the design of the product as a “risk” such that model predictions never dictate the academic track of a student or blindly label a student as “at risk”.

Very few data points of students with accents also get high quiz scores

Schools do not have adequate support for English language learners

Algorithm might learn that students with accents shouldn’t progress to advanced reading levels or may not make as accurate recommendations for this group of students compared to others.

No, this problem happens because of bias in the school system. Amplifying examples of high performing students with accents may work but it may also throw off the model’s accuracy for students in this group.
The underlying bias must be disclosed and addressed in the design of our product.

Draw schools’ attention to the fact that they may be underserving their students with accents.
Make sure this is acknowledged during training and implementation as an area of inquiry when teachers and students use your product and consider this fact in the design of your product. For example, if you know that students with accents aren’t progressing as fast as the rest of the class, don’t automatically place students on a lower academic track. Instead suggest that students receive additional support in specific language areas.

Less data for Black and Brown students at advanced reading levels

Black and Brown students were historically placed in less rigorous reading classrooms.

Model may learn that this group of students should progress more slowly than others.

No, this problem happens because of bias in the school system. Amplifying examples of Black and Brown students at advanced reading levels may work but it may also hurt Black and Brown students who need additional reading support.
The underlying bias must be disclosed and address in the design of our product.

Make sure schools aware of the fact that they may be underserving Black and Brown students in literacy.
Make sure this is acknowledged during training and implementation as an area of inquiry when teachers and students use your product and consider this fact in the design of your product. For example, if you know that Black and Brown students are traditionally placed in less rigorous reading classrooms, don’t recommend placing students on lower academic tracks as a result of your model. Instead suggest that students receive additional support or be assigned more engaging types of reading content.