Designer, Researcher, & Facilitator
2020
Sketch, UserZoom, Abstract
Before takeoff, our team had various involvements in UPMC Health Plan's chatbot initiative, including prior research and designing the company's digital future vision. We were chosen by leaders aiming to accelerate the project.
Based on prior knowledge, we identified foundational UX elements for chatbots: Clear disclosure they are not human, concise response, familiar messaging app-style interface, and an option to contact a real person.
Given the widespread use of chatbots in healthcare and other industries, we sought commonalities among experiences. We examined various companies, including Woebot, USAA, Amazon, Bank of America, Geico, Lyft, Uber, Starbucks, and TurboTax. Interestingly, some like Zappos found chatbots insufficiently customer-centric and chose not to adopt them.
Our research revealed that the key factors distinguishing great from mediocre chatbot experiences are trustworthiness, actionability, and usability. Failure to meet these criteria results in low user engagement and adoption.
We knew the next step before we dove into testing was getting a plan together. We wanted to rapidly iterate and make changes for the following day of testing. This process involved four tests with an average of 5.25 participants per usability study.
Following the research plan, we recognized the necessity of a prototype for user testing. We devised scenarios covering situations where users might seek info about their deductible or locate a doctor.
We conducted testing with five actual UPMC members, aiming for a realistic experience. To achieve this, five Sketch prototypes were made, incorporating user names and profile pictures.
Due to the project's tight timeline, we conducted evening tests, synthesized findings in the morning, and redesigned in the afternoon. We maintained round-the-clock focus testing the right things with the right users.
We followed this iterative design process for five days, expediting data collection using UserZoom. The final test involved in-person sessions with real UPMC Health Plan members, who provided valuable feedback. One member expressed a preference for using the chatbot — exclaiming, "If it was this quick and easy to get information, I would."
We presented our findings to nearly 20 leaders from various teams invested in chatbot development. Despite limitations, we had strong evidence that users trust and prefer chatbots for their convenience.
Potential research areas include chat onboarding, language usage, chatbot-to-human transitions, and testing with older Medicare members. We've laid a solid foundation for future designers or researchers, and there's ample room for further chatbot testing and development at UPMC.
Our chatbot research led to the exploration of future native app functionalities, enabling users to seek immediate assistance through chats with either a chatbot or a live person. This initiative involved the entire organization in uncovering the future of Telehealth and healthcare — Becoming substantially valuable for UPMC with the Covid-19 pandemic a few months later.
UPMC • Find Care