It is a cliché that the Internet has opened up new ways for HCI researchers to conduct human subjects research. Yet remote, Internet-based experimentation is not yet part of the standard research toolkit because of the concerns about data validity, feasibility of recruiting representative participant samples, and perceived technical barriers. We are developing and validating tools and methods to enable large-scale empirical research with human subjects. Our core goals are to enable a much faster theory-to-experiment cycle, to facilitate access to larger and more diverse participant populations, and to enable larger scale experimentation (in terms of numbers of conditions and experiments) than what is feasible with lab-based methods.
In one project, we developed a system for collecting lab-quality measurements of a person’s motor performance by unobtrusively monitoring his or her natural usage of input devices in situ. Our system uses a rare machine learning technique (learning from positive and unlabelled examples) to automatically filter all of user's mouse pointer trajectories for those representing deliberate, targeted movements that can be used to compute standard measures of motor performance. Our results show that, on four distinct metrics, the data collected in-situ and filtered with our system closely matches the data obtained from the same participants in a formal experiment.
In another project, we developed LabintheWild.org---a platform for conducting behavioral experiments with unpaid online volunteers. Volunteers from all over the world to participate in LabintheWild studies in exchange for interesting personalized feedback. Over the past three years, LabintheWild has attracted over 2.7 million distinct visitors from over 200 countries and resulted in approximately 1 million completed experimental sessions. We have validated this platform by demonstrating that results obtained on LabintheWild match those obtained in traditional laboratory settings. LabintheWild has made it possible for us to conduct research that would not have been feasible with traditional methods. I will summarize the findings from several experiments conducted on LabintheWild and I will synthesize the emerging set of best practices for designing studies that attract intrinsically motivated participants and for ensuring validity of the data.
Krzysztof Gajos is an associate professor of Computer Science at the Harvard Paulson School of Engineering and Applied Sciences. Krzysztof is broadly interested in intelligent interactive systems, a research area that bridges artificial intelligence and human-computer interaction. Recent projects pursued by his group contributed to diverse areas such as personalized adaptive user interfaces, systems for supporting collective creativity, organic crowdsourcing, large-scale experimentation in the wild, and learning technologies.
Prior to arriving at Harvard, Krzysztof was a postdoctoral researcher at Microsoft Research. He received his Ph.D. from the University of Washington and his M.Eng. and B.Sc. degrees from MIT. In the Fall of 2005, he was visiting faculty at the Ashesi University in Accra, Ghana, where he taught Introduction to Artificial Intelligence. Krzysztof is a coeditor-in-chief of the ACM Transactions on Interactive Intelligent Systems. He is a recipient of a Sloan Research Fellowship.