About the Research
It’s clear that the use of digital products supports learning, but there is also a sense of frustration when applications are hard to use. We conducted original research with students at Penn State University to better understand their experiences with EdTech.
Our study investigated five specific EdTech applications used at Penn State: Canvas, Starfish, Top Hat, LionPATH: Course Enrollment, and LionPATH: Schedule Builder. A secondary goal was to understand student and professor perspectives on the use of AI in higher education. Qualitative and quantitative data was gathered through 8 in-depth interviews and 70 survey responses.
Canvas is a centralized platform for accessing and managing online course materials and communication. Students use it daily to stay updated on assignments, interact with instructors, and monitor their academic progress.
Starfish is a platform that enables students to directly contact professors and schedule appointments with academic advisors. Depending on the student, this platform is used anywhere from once per semester to once a month.
Top Hat is a web-based student response system that instructors use for polling, discussion prompts, lecture presentations, and attendance tracking. Students use this platform weekly, based on their professors’ preferences.
LionPATH’s Schedule Builder allows students to input courses and time preferences, generating several schedule options. This tool is typically used once per semester, as students design their personalized schedules for the upcoming term.
LionPATH’s Course Search and Enrollment features enable students to filter and explore available courses, adding them to an “Enrollment Shopping Cart” for later selection. Students utilize these functions once per semester to schedule their upcoming courses.
Our goal was to understand student’s experience with each application. We focused our questions and probes on:
- How students rated the usability of each application
- What students liked about the application
- What students disliked about the application
Foxgroove is a user research and product design agency that specializes in crafting usable experiences. Contact us to learn more.
Usability Ratings
Students were asked to rate each application on a scale from 1 (Very Bad) to 7 (Very Good). Students answering 6 or 7 (Good in the graph below) are net promoters. Scores of 5 and lower (Not Good in the graph below) equate to detractors. Canvas was rated highest with 70.5% Good, while Course Enrollment rated lowest with only 29.7% Good. The gap between the two applications is 40.8% suggesting that students can differentiate between good and bad experiences.
Usability Ratings
Each application we studied has room for improvement. Even though Canvas had the highest positive rating of 70.5%, a relevant percentage of students (29.5%) felt it needed improvement in certain areas. With over 73,000 students enrolled in 2024 that equates to ~21,500 who want a better experience from the top-rated application.
Foxgroove can conduct a usability evaluation to find, prioritize, and fix your application’s specific usability issues. Contact us to learn more.
Student Usability Likes & Dislikes
Open-ended, qualitative questions and probes were used gather student likes and dislikes. We used an AI platform called Looppanel to transcribe, analyze, and tag the data. The Foxgroove research team reviewed the AI synthesis and manually read through each transcript for additional insights. The manual intervention was performed to ensure comprehensive data analysis and integrity.
Our analysis found a number of important takeaways for each application. Two key positive and negative insights are presented below along with supporting quotes.
Canvas
28 mentions
Class assignments and modules are in a single calendar view
“I can locate my assignments all in one place and that they present the assignments the day they are due, so you know what you need to get done.”
9 mentions
Incorrect grade calculations
“Sometimes it grades online assignments incorrectly.”
“Canvas said I had an A- for the class, but it was actually an A.”
46 mentions
Ease of use and organization of content
“It’s very organized and easy to navigate.”
“Its easy to use interface.”
9 mentions
Too many notifications
“Sometimes I’ll get 4 notifications instead of just one.”
“…the notifications are really bothering me”
Starfish
33 mentions
Ease of use
“Easy to use.”
“It gets the job done.”
“It’s very clear.”
12 mentions
Meeting logistics
“Sometimes it’s not clear where the appointments are or if professors or staff note something, it’s not always visible which leads to missed meetings.”
“It’s hard to tell if appointments are in person or online.”
23 mentions
Scheduling appointments
“Easy and quick to make appointments. Gives variety of options. Remembers people you frequently contact so you can find them faster.”
7 mentions
Ease of scheduling an appointment
“Sometimes it’s hard to find who you’re looking for.”
“Can be hard to schedule an appointment.”
Schedule Builder
24 mentions
Ease of use
“I like how easy it is to build a schedule.”
“Easy to use.”
25 mentions
Schedule options
“Some classes are available and then they really aren’t for you.”
“They’d give you 500 similar schedules that you could possibly have, and you’re obviously not gonna go through all of them.”
32 mentions
Schedule options and visualization
“It makes it easy to build a schedule and comes up with countless options.”
“Helps visualize the schedule.”
7 mentions
Too many notifications
“Sometimes it’s overwhelming and the blocks are confusing.”
“It’s unorganized.“
Top Hat
26 mentions
Ease of use
“Easily accessible from my phone as well as my computer.”
“The app is easy to use.”
26 mentions
Technical issues and lagging
“Logging in takes too long, there is also some buffering that may occur which can negatively impact your grade.”
“It sometimes lags out and makes you miss questions…”
15 mentions
Class engagement
“The live in-class questions keep me engaged.”
“Keeps me paying attention.”
9 mentions
Time limits
“Can take off points because an answer was incorrect if you didn’t get enough time to finish it.”
Course Enrollment
17 mentions
Searching for courses
“You can use course search to find very niche classes and sections to make your schedule easier.”
“Super easy to find a class.”
25 mentions
Onboarding & Poor usability
“I guess there’s not really any instructions when you first are learning how to use it. So you’re kind of just thrown in there and you kind of have to figure it out all by yourself. So if there was like more videos or any instructions how to use it better, that would be really helpful for new students.”
6 mentions
Navigation
“Easy to navigate.”
“It’s easy to navigate.”
4 mentions
Navigation
“Yeah. And the backs arrow thing is really weird to me.”
“…hard to navigate to what you want to find.”
Understanding how your application helps or hinders students is critical in designing future EdTech experiences. Contact us to learn more.
Implications for product teams and educational institutions
We found a wide usability gap between our study applications and across all of EdTech. During COVID there was a huge need for EdTech capabilities. Functionality was prioritized to meet the fast emerging needs of educational institutions. Students, teachers, and administrators began to realize that functionality on its own cannot win the day. Usability is the critical thread that must be woven into functionality and pedagogy.
Product teams can win and maintain business when their applications are usable. Usability is becoming an important evaluation criteria for EdTech purchases. Educational institutions are beginning to rely on certifications like the ISTE Seal. Usability is on their radar and it needs to become a valued driver in product planning.
Educational institutions now understand the impact and role usability plays in the educational experience. Elevating the importance of usability in RFP criteria helps product teams understand the role it plays in EdTech decision-making. Involving usability consultants to participate in RFP evaluations will help ensure applications with the highest level of usability are selected.
The use and power of AI has increased dramatically over the past 2 years and it shows no signs of slowing down. It’s relatively cheap and even free to use. Businesses love it for the productivity increases and expect their workforce to have AI skills. We learned a lot about how K-12 teachers and administrators felt about AI use at the CoSN 2024 conference in Miami (see our CoSN 2024 Highlights blog article), and wanted to focus some of our research in the higher education space.
The student interviews gave us an opportunity to discuss various aspects of AI usage at Penn State. Four key takeaways are presented below along with supporting quotes.
Students use AI as a resource rather than fully relying on it
“I use AI to verify answers when uncertain but rely on my own thoughts and opinions, and never use AI for writing tasks.”
“There’s definitely a lot of really good learning and teaching skills that can come from AI.”
Students have an overall positive experience with AI
“Sometimes when I don’t think an answer makes sense, it does a very good job of explaining why it’s right and why it is that answer.”
“It’s pretty amazing just how quickly I can get accurate responses. It’s very useful.”
Students have several concerns and frustrations with AI
“I don’t like when teachers say you can use it because I feel like people abuse it, which they definitely do.”
“If I give it a question to a problem on an assignment, it gives me answers that are sometimes wrong. So I don’t really. I don’t usually follow it.”
Teachers generally allow the use of AI for guidance but not for direct usage.
“I would say our professors probably despise chat GPT and think that it makes it easier for students to cheat.”
“Some teachers are open to students working with AI as long as they acknowledge it’s not entirely their own work.”
Research Kudos
Our intrepid summer intern Elyse Terzian played a critical role in this research project. She participated in the planning, recruiting, moderation, and analysis of the research. She was a quick learner and brought a level of context and insight to this project that helped us grow as researchers. Elyse is currently a Human-Centered Design and Development major at Penn State University.
Product Design 
Usability Analysis 
