Initial usability interviews were conducted with 15 teachers (9 at Carver and 6 at Schafer Park) between 2nd-6th grade to understand their first impressions with Beereaders’s platform. The central research question for these interviews was “Does the training protocol allow teachers to independently navigate the Beereaders platform and successfully find and use all its features? Why or why not?”
Research Team
Dr. Trisha Callella
Dr. Erin Huebert
Schools Team
Andrea Cook
Karnissa Caldwell
Jorge Holgiun
Graphic Design
Alexander Sheppard
School Goals, Challenges, and Expectations for Beereaders
What are teachers’ biggest challenges in Spanish reading comprehension that they want to target with Beereaders?
Vocabulary development
“Many of the students don’t know how to use context clues to understand words.”
Identification of main ideas and/or author’s purpose
“They’re able to read the words but they can’t comprehend.”
Discerning supporting details
“The kids struggle with picking out details and linking them back to main idea.”
What are your top three instructional goals related to reading comprehension in Spanish?
Get students on grade-level reading comprehension
“I hope that this system will help the students become better readers and grow in their reading levels.”
“I have never seen such a large group of second grade students who don’t know how to read.”
Build and increase vocabulary
Build background knowledge around words and pictures
“We’re finding big gaps in background knowledge, especially without pictures.”
Build confidence with reading and comprehension
“I want them to feel like they’re growing [...] to see that they’re growing.”
What value do you expect and hope Beereaders to add to your classroom and/or to student learning?
Access to a variety of authentic books from all Latin American countries
“We don’t get text from those areas, so getting to look at language from different places is key in my classroom.”
“We can’t always find good literature by ourselves or from our district.”
“It’s a great library of authentic literature that these kids can use during their independent reading. It’s not always easy for them to have access to authentic Spanish literature.”
“I’m hoping that it exposes them to different types of texts in my classroom. The students are only exposed to Imagine Español and one other text source, many of which they have already read, so having a larger selection of texts will be beneficial.”
Increase engagement with Spanish literature
“I will learn more about language differences from the literature.”
“I’m hoping that Beereaders books will be more engaging [...] they will want to understand and be engaged.”
“I want them to identify with their culture and where they’re from.”
“I hope Beereaders gives students the motivation to learn more about the countries they’re from.”
New resource for independent reading and during centers
“I expect to use Beereaders during independent reading time.”
“I plan to use Beereaders as a center for the students during my reading workshop — it will be a resource for them to explore during independent time.”
New data resource for assessing differentiation of student learning
“I hope it will allow me to discern who is struggling — some kids don’t act like they are having a hard time, so having stats and reports to back this up will help me to better discern who is having a hard time and needs more support.“
“The data will be helpful in terms of making sure the children are moving along as they are supposed to and help teachers make sure they are meeting certain goals and providing the right kinds of support.”
Other Resources that Teachers Currently Use for Spanish Reading Comprehension:
Learning A-Z (Reading A to Z)
i-Ready
Imagine Learning (Imagine Espanol)
Raz-Kids
Epic Books
Lexia
Usability Results
“Does the training protocol allow teachers to independently navigate the Beereaders platform and successfully find and use all its features? Why or why not?”
Task Analysis
At the beginning of the study, teachers were asked to complete a series of tasks on the Beereaders platform. At this point, the only exposure that teachers had to the Beereaders platform was during the training. These initial usability interviews were conducted virtually. The principal researcher led the teachers through these tasks during a 45 minute interview, while the teachers shared their screen and a research assistant took notes.
TASK
COMPLETION RATE
Find a book for a 10-year-old about adventure.
………………………………………………………………………………………………………………………………………
Easy to find and complete.
100%
Completed without help
Find a book from Colombia.
………………………………………………………………………………………………………………………………………
Easy to find and complete.
100%
Completed without help
Assign that book to one of your classes for the dates October 1-20. ………………………………………………………………………………………………………………………………………
Easy to find and complete.
100%
Completed without help
Assign that book to one of your classes for the dates October 1-20.
………………………………………………………………………………………………………………………………………
Easy to find and complete.
100%
Completed without help
Change the time period for that book to November 15-30.
………………………………………………………………………………………………………………………………………
Easy to find and complete.
100%
Completed without help
Delete that book you just assigned.
………………………………………………………………………………………………………………………………………
Easy to find and complete.
100%
Completed without help
Find an activity for an 11-year-old and show me how you would download it.
………………………………………………………………………………………………………………………………………
Once teachers found the activity section in the Resources Room, it was easy to find and download an activity. About half of the teachers had to, however, poke around a little before finding the activity section but did, fairly quickly, find it without help.
Note: the activity buttons do not load very quickly and some don't load at all until one clicks on preview.
100%
Completed without help
Download a report for one of your classes for the month of October.
………………………………………………………………………………………………………………………………………
About half the teachers searched first in the Quick Stats and/or Skill and Reading Level sections before finding reports in the Resource Room. Those that did find reports on the first try did so because they were already in the Resource Room and said their intuition would have been to first look in the Quick Stats section.
Additionally, about half the teachers first clicked on "Reports and Analytics" and the other half first clicked on "Monthly Report."
All teachers could easily create and download a report from either location.
Before creating/downloading a report in either section, only one teacher completely understood what the difference would be between a monthly report and a report from the "reports and analytics" section that covered the same date range.
56%
Completed without help
Download or create a student report for the last two weeks.
………………………………………………………………………………………………………………………………………
All teachers could easily create and execute a student-level report.
It is worth noting, however, that they were already in the relevant Reports and Analytics section when this question was asked and to find it may not have been as easy if they had started elsewhere.
100%
Completed without help
Send a message to all of the students in one of your classes.
………………………………………………………………………………………………………………………………………
About half the teachers first clicked on the envelope icon at the top of the Beereaders page. After poking around a little, they were all able to pretty quickly find it in the Message Center. Those that found it on the first try were already in the Resource Room. All teachers mentioned that the Message Center was not intuitively in the right place.
All teachers faced a major barrier when trying to send a message to the whole group - all teachers clicked in the "To groups" line and clicked on their relevant class. They then all clicked in the "To students" line and started to click each student (which was frustrating to them).
They also didn't notice the error message they received back in the "To groups" line saying they "must select a grade and a class." When they went back to fix it, they didn't understand the difference between "grade" and "class" since those terms mean the same thing to them.
Only half scrolled to the bottom of the list in the "To groups" line and saw the three "Tier" options. They then selected all three options but all teachers said that "doesn't make sense."
0%
Completed without help
Send a personal message to one student in one of your classes
………………………………………………………………………………………………………………………………………
All but one teacher first clicked in the "To groups" line again, and then selected just one student. The teachers were concerned and unclear, however, if having a class selected in the "To groups" line would send a message to the whole class even if just one student was selected.
11%
Completed without help
Show me where you would go if you needed technical support/help on Beereaders?
………………………………………………………………………………………………………………………………………
About half the teachers poked around for awhile before eventually heading into Tutorials and Webinars. Most teachers first clicked on the envelope at the top thinking it would email or contact customer support. When teachers did arrive in the Tutorials and Webinars, they said they "didn't really think of that as a 'help' section" but more like topics for continued tips. All teachers wanted a FAQ section and/or area for "trouble-shooting" along with a way to contact a live agent for customer support.
44%
Completed without help
Usability Recommendations
How can teachers use features more easily?
The usability of all of Beereaders features is easy and intuitive, especially the features within the Library. The only feature that was difficult for teachers to understand was the student messaging feature, as it required teachers to select two categories within one drop down line (“To groups”), which was not clear or intuitive.
Student Messaging
It is, therefore, recommended that there be three separate “To lines:” Class, Tier, and Student. Each line should be optional, so that teachers only need to select the relevant category. For example, if a teacher wants to send a message to the whole class, he/she should only choose their class from the class line while leaving the other two lines blank. Or if a teacher wanted to send a message to just a few students, he/she should be able to select a few students from a list in the Student line.
How can teachers FIND features more easily?
There were three features that were hard or not as intuitive to find.
Quick Stats / Reports Section
Teachers recommended that reports be included in the Quick Stats or Skill Level section since that is where the data is being displayed. As is, it is also not clear to teachers the difference between the “Reports and Analytics” and “Monthly Report” section, so it is recommended that a brief description of the available data be included at the top of each section’s page.
“The reports were difficult to find.”
Envelope Icon / Messaging
Teachers thought the envelope on the top of the Beereaders page would take them to Student Messaging, so it is recommended that the envelope icon take teachers to the Messaging section rather than just to the inbox. The Messaging section itself could be its own landing page rather than a part of the Resource Room where it was not intuitive and, therefore, difficult to find.
“The Resource Room is weird, having the message center there.”
“The Message Center was difficult to find at first. The Message Center and the envelope [icon] should be better integrated.”
Help Button / Help Section
It is recommended that there be a help button on the top of the main screen that would direct teachers to the Tutorials and Webinars section. Teachers also preferred a FAQ section related to technical/usability issues as well as potential trouble-shooting issues. While some of the videos were informative, most teachers said they wouldn’t want to take the time to watch a video and would, instead, read a FAQ section. Additionally, teachers suggested having a step-by-step tutorial in the Help section that would serve as a good supplement to the personal training.
“The tech support was hard to find.”
“When I found the tutorials/webinars tab, it didn’t exactly fit my definition of tech support.”
“I would appreciate also having step-by-step instructions to return to after the training.”
“A brief tutorial on how to find and use features would be great as it would serve as a good reminder of the training.”
“A 30 minute walkthrough video where the trainer walks you through the program and how to use it would be great.”
How can the training be improved?
Strengths
The trainer (Sebastian) was very engaging, informative and clear.
“It was great that the presenter was able to split between English and Spanish.”
“He [Sebastian] was clear and he could answer all our questions quickly.”
It was helpful to have access to both the teacher and student versions of the Beereaders platform to be able to follow along during the training.
“Getting to see the perspective of a student before you give it out to your students is very helpful.”
“The demo account is super great because it gives me a chance to explore the student side and also provides an opportunity for me to walk my students through the login process together in the class.”
Recommendations
Most teachers felt like 75-90 minutes of training was needed to cover both the teacher and student platform, but teachers got tired and it felt too long to have in one sitting. Thus, teachers suggested the training occur in two separate sessions or having a couple breaks in one session.
“ [There was] no weakness, except time. It took a long time.”
“It is a lot of information to take in at one time.”
Teachers acknowledged that they ran out of time but did want to have more time to go over the student-facing platform.
“I want to be sure that I know how to help my students.”
Most teachers (especially those that identified as “technologically confident”) felt like there could be less time spent going over how to use the features (i.e. library, highlighter) since it was very intuitive.
Instead, more time could have been spent going over how to read, use, and interpret the data. There was also a desire to have actual data available to see, even if it was fake data because it would have made following along with and recalling the data part of the training easier.
“I wish that we had been able to see during the training what the reports look like side by side to learn how they [monthly reports vs. reports and analytics] are different.”
“Have a pre-made teacher website to see data.”
“When you’re trained on these reports and they don’t mean anything yet it’s harder to stick.”
Most importantly, teachers wished that their full student rosters were set up and ready to be seen during the training because it would have made it easier to follow along with and remember later the data features. It also delayed teachers from being able to immediately use Beereaders after the training which they had hoped to do.
“It’s difficult that the teachers were not able to practice on their own with their student rosters set up.”
“The hardest part for teachers is wanting to go try something but it’s not set up yet.”
“We will have more questions when the rosters are set up.”
Logistically, it is better that teachers each log-in to the zoom meeting from their own devices - rather than one zoom video be projected onto a big screen - as it is hard to see what the presenter is doing on the big screen and harder to type in the links rather than being able to simply click on the links in the chat.
Is a personal training necessary?
89% of teachers said yes.
“Yes. It's interactive & engaging.”
“Yes. It's helpful for learning alongside each other.”
“Yes. If there was an actual help center within Beereaders, then no. But because there is no real in-app tech support, it helps to have training in person with someone to answer those questions. The appeal of in-person training would be to have an expert on Beereaders walk teachers through the application and work through any technical difficulties or questions we may have.”
“Yes. A personal training allows us to assist each other.”