Matters of ethics, trust, and potential liability for autonomous systems

The objective of this panel was to discuss issues related to the development and use of autonomous systems, with specific focus on the overriding themes of ethical considerations and potential liability for Human Factors and Ergonomics (HF/E) professionals who are involved in their development. Chris Brill provided opening remarks to frame the discussion and introduce the panelists. James Bliss discussed legal implications related to our collective penchant for developing conservative, false-alarm prone automation. Peter Hancock advocated for human-centered constraints on autonomous systems, as they may, one day, pose an existential threat to humanity. Dietrich Manzey discussed ethical considerations for autonomous systems, including how design can encourage ethical user behavior. Joachim Meyer argued that HF/E professionals have an obligation to help designers understand the ethical implications of poor design, particularly in the context of autonomous systems. Lastly, Alison Vredenburgh provided thoughts on potential liability for HF/E professionals, particularly in light of the relative newness of autonomous systems. The panel then turned to facilitated discussion with panelists and audience members. Specific themes included the boundaries of our responsibilities as HF/E professionals for ill-conceived or morally-objectionable systems, potential implications of manipulating user trust through design, cross-cultural perspectives on public acceptance and legal peril, and how concerns might differ by domain (e.g., medical vs.combat vs. manufacturing). The session concluded with panelists summarizing how ethics influence design and recommendations for how HF/E professionals can potentially protect themselves from legal liability for mishaps involving autonomous systems they helped develop.