Reflection 3 — Measurement and Fairness

Measurement and Fairness, Abigail Z. Jacobs, Hanna Wallach, FAccT 21, https://arxiv.org/abs/1912.05511

The paper proposes “measurement modeling from the quantitative social sciences as a framework for understanding fairness in computational systems.” More specifically, the authors introduce the unavoidable process of making assumptions in computational systems that might introduce construct-operationalization mismatches and finally lead to fairness issues, and propose measurement modeling as well as concepts of construct reliability and construct validity as a way to visualize these mismatches and prove trustworthiness.

I doubt that this process of hamming fairness into computational systems is practical, for the following reasons:

1. Not everyone will agree with the measurement proposed. In fact, although claimed to be “objective” in this paper, measurements are by nature subjective and contain biases from their designers.
2. Everyone’s definition of fairness is different. For example, when you allocate food to people, do you allocate more food to adults/males than child/females as the former eats more for physical and biological reasons, or do you allocate food equally to everyone? Which way is fairer? I believe different people would have different opinions. The topic of fairness has too many ways of perceptions and I think it cannot be clearly measured.
3. In most people’s conceptions, math and science are exact and accurate. This is especially true when these people are scientists or researchers. The fact that their well-designed scientific models might contain some falseness would be hard to believe, not to mention that this falseness comes from the seemingly immeasurable ethics.
4. Even when the researchers believe the fact that they need to change their model to enforce those humanity norms, it would be hard for them to even start it. Computer science researchers are usually more used to concrete computations than thinking in “fairness” or “equality” like social scientists.
5. Even when the researchers know how to enforce fairness in their models, there are not enough incentives for them to do so. The process is time- and money-consuming, but it doesn’t give much return in terms of publicity (number of citations).