Double Binds and Double Blinds: Evaluation Tactics in Critically Oriented HCI
DOI:
https://doi.org/10.7146/aahcc.v1i1.21266Keywords:
Critical technical practice, critically oriented HCI, evaluation,Abstract
Critically oriented researchers within Human-Computer Interaction (HCI) have fruitfully intersected design and critical analysis to engage users and designers in reflection on underlying values, assumptions and dominant practices in technology. To successfully integrate this work within the HCI community, critically oriented researchers have tactically engaged with dominant practices within HCI in the design and evaluation of their work. This paper draws attention to the ways that tactical engagement with aspects of HCI evaluation methodology shapes and bears consequences for critically oriented research. We reflect on three of our own experiences evaluating critically oriented designs and trace challenges that we faced to the ways that sensibilities about generalizable knowledge are manifested in HCI evaluation methodology. Drawing from our own experiences, as well as other influential critically oriented design projects in HCI, we articulate some of the trade-offs involved in consciously adopting or not adopting certain normative aspects of HCI evaluation. We argue that some forms of this engagement can hamstring researchers from pursuing their intended research goals and have consequences beyond specific research projects to affect the normative discourse in the field as a whole.
References
Agre, P. Computation and human Experience. Cambridge University Press, 1997.
Agre, P., Chapman, D. 1987. Pengi: an implementation of a theory of activity. In Proceedings of the sixth National conference on Artificial intelligence - Volume 1 (AAAI'87), Vol. 1. AAAI Press 268-272.
Agre, P. "Toward a critical technical practice: Lessons learned in trying to reform AI." Bridging the Great Divide: Social Science, Technical Systems, and Cooperative Work, Mahwah, NJ: Erlbaum (1997): 131- 157.
Aoki, P. M., Honicky, R. J., Mainwaring, A., Myers, C., Paulos, E., Subramanian, S., & Woodruff, A. "A vehicle for research: using street sweepers to explore the landscape of environmental community action." In Proc. CHI ‘09
Bardzell, S., Bardzell, J., Forlizzi, J., Zimmerman, J., and Antanitis, J. "Critical design and critical theory: the challenge of designing for provocation." In Proc. DIS ‘12
Bardzell, J., Bardzell, S, and Stolterman, E. 2014. Reading critical designs: supporting reasoned interpretations of critical design. In Proc. CHI ‘14
Bardzell, J., and Bardzell, S. 2013. What is "critical" about critical design?. In Proc. CHI ‘13
Baumer, E. P. S., Cipriani, C., Davis, M., He, G., Kang, J., Jeffrey-Wilensky, J., Lee, J., Zupnick, J., and Gay, G. K. (2014). Broadening Exposure, Questioning Opinions, and Reading Patterns with Reflext: a Computational Support for Frame Reflection. Journal of Information Technology & Politics, 11(1), 45–63
Baumer, E. P. S., Halpern, M., Khovanskaya, V., & Gay, G. Probing the Market: Using Cultural Probes to Inform Design for Sustainable Food Practices at a Farmers’ Market. In J. H. Choi, M. Foth & G. Hearns (Eds.), Eat, Cook, Grow: Human-Computer Interaction with Human- Food Interaction. MIT Press, 2014.
Blythe, M., Overbeeke, K., Monk, A., Wright, P. (eds). Funology: From Usability to Enjoyment. Kluwer Academic Publishers, 2003.
Boehner, K., Vertesi, J., Sengers, P., & Dourish, P "How HCI interprets the probes." In Proc. CHI ‘07
Boehner, K., Sengers, P., and Warner, S. 2008. Interfaces with the ineffable: Meeting aesthetic experience on its own terms. ACM Interact. 15, 3, Article 12 (December 2008), 29 pages.
Chong, D., & Druckman, J. N. (2007). Framing Theory. Annual Review of Political Science, 10(1), 103–126.
Cohn, M., Sim, S., and Dourish, P. 2010. Design methods as discourse on practice. In Proc. GROUP ‘10
DiSalvo, C. Adversarial Design. The MIT Press, 2012. 16.Dourish, P. Where the action is: the foundations of
embodied interaction. MIT press, 2004.
Entman, R. M. (1993). Framing: Toward Clarification of a Fractured Paradigm. Journal of Communication, 43(4), 51–58
Gaver, W. W., Boucher, A., Pennington, S., & Walker, B. (2004). Cultural probes and the value of uncertainty. interactions, 11(5), 53-56.
Gaver, William W., Jacob Beaver, and Steve Benford. "Ambiguity as a resource for design." In Proc. CHI ‘03.
Gaver, W., Bowers, J., Kerridge, T., Boucher, A., & Jarvis, N. "Anatomy of a failure: how we knew when our design went wrong, and what we learned from it." In Proc. CHI ‘09.
Gaver, W., Boucher, A., Law, A., Pennington, S., Bowers, J., Beaver, J., Humble, J., Kerridge, T., Villar, N., and Wilkie, A. “Threshold devices: looking out from the home”. In Proc. CHI ‘08
Goffman, E. (1974). Frame Analysis. Cambridge, MA: Harvard University Press.
Hirsch, T. 2010. Water wars: designing a civic game about water scarcity. In Proc. DIS ‘10
Khovanskaya, V., Baumer, E.P.S., Cosley, D., Voida, S. & Gay, G. “Everybody knows what you’re doing: A critical design approach to personal informatics.” In Proc. CHI ‘13
Kuznetsov, S., Hudson, S., and Paulos, E. "A low-tech sensing system for particulate pollution." In Proc. TEI ‘14
Kuznetsov, S., Davis, G. N., Paulos, E., Gross, M. D., & Cheung, J. C. 2011. Red balloon, green balloon, sensors in the sky. In Proc. UBICOMP ‘11
Kuznetsov, S., Davis, G., Cheung, J., & Paulos, E. “Ceci n'est pas une pipe bombe: authoring urban landscapes with air quality sensors.” In Proc. CHI ‘11
Kuznetsov, S., Odom, W., Moulder, V., DiSalvo, C., Hirsch, T., Wakkary, R., & Paulos, E. HCI, politics and the city: engaging with urban grassroots movements for reflection and action. In CHI EA '11.
Lieberman, H. The Tyranny of Evaluation, http://web.media.mit.edu/~lieber/Misc/Tyranny- Evaluation.html
Löwgren, J. 2013. Annotated portfolios and other forms of intermediate-level knowledge. interactions 20, 1 (January 2013), 30-34.
Meyerson, D., and Scully, M. "Crossroads tempered radicalism and the politics of ambivalence and change." Organization Science 6.5 (1995): 585-600.
Michael, M. (2012). “What are we busy doing?” Engaging the idiot. Science, Technology & Human Values, 37(5), 528-554.
Olson, J., and Kellogg, W. Ways of Knowing in HCI. Springer, New York, NY, 2014.
Orne, M. T. (1962). On the social psychology of the psychological experiment: with particular reference to demand characteristics and their implications. American Psychologist, 17(11), 776–783.
Pierce, J. 2012. Undesigning technology: considering the negation of design by design. In Proc. CHI ‘12
Pierce, J. and Paulos, E. “Counterfunctional things: exploring possibilities in designing digital limitations.” In Proc. DIS ‘14
Polletta, F., Pierski, N., Baumer, E. P. S., Celaya, C., & Gay, G. (2014). A “Peopled” Strategy of Frame Reflection. In Annual Meeting of the American Sociological Association (ASA). San Francisco.
Preece, J., and D. Maloney-Krichmar. "The human- computer interaction handbook." (2003): 596-620.
Schön, D. A., & Rein, M. (1994). Frame Reflection: Toward the Resolution of Intractable Policy Controversies. New York: Basic Books.
Sengers, P., Boehner, K., Mateas M., and Geri Gay. 2008. The disenchantment of affect. Personal Ubiquitous Comput. 12, 5 (June 2008), 347-358.
Sengers, P., Boehner, K., David, S., & Kaye, J. J. "Reflective design." In Proc. CC ‘05
Sengers, P., and Gaver, B. “Staying open to interpretation: engaging multiple meanings in design and evaluation.” In Proc. DIS ‘06
Silberman, M., Blevis, E., Huang, E., Nardi, B. A., Nathan, L. P., Busse, D. Preist, C., and Mann, S."What have we learned?: a SIGCHI HCI & sustainability community workshop." In CHI EA'14.
Suchman, L. Human-machine reconfigurations: Plans and situated actions. Cambridge University Press, 200
