“The best time to invest in research is before you start your company.
The second best time is today.”
In 2020’s transition to remote learning, schools were inundated with edtech tools. Teachers were hacking together solutions on a class-by-class basis, attempting to keep students engaged and on-track with learning goals.
Schools worked with a necessary urgency to get tech tools implemented and paid for; however, this left little time for tools to go through the normal procurement and vetting process. Now, as ESSER funds expire, schools are facing a glut of underutilized and ineffective edtech tools and they finally have time to start evaluating what truly works and which ones are worth keeping.
As schools make long-term decisions about procurement, they’ll be looking for ones that work for students and teachers AND have the evidence to back up their claims. In order for edtech companies to prepare for long-term market trends, it’s more important than ever that they start conducting research and collecting evidence.
What does evidence-based mean?
The phrase “evidence-based” should sound familiar if you’ve been working in this space for any length of time. It’s a phrase with a very specific definition that was established in the Every Student Succeeds Act.
Specifically, “evidence-based” refers to the four tiers of recognized evidence “that demonstrate a statistically significant effect on improving student outcomes or other relevant outcomes based on--”
(TIER I) strong evidence from at least 1 well-designed and well-implemented experimental study;
(TIER II) moderate evidence from at least 1 well-designed and well-implemented quasi-experimental study;
(TIER III) promising evidence from at least 1 well designed and well-implemented correlational study with statistical controls for selection bias;
(TIER IV) demonstrates a rationale based on high-quality research findings or positive evaluation that such activity, strategy, or intervention is likely to improve student outcomes or other relevant outcomes.
Tier I is the most rigorous tier of evidence, though not necessarily the “best.” For many companies--regardless of stage--deeper, qualitative research can be the most beneficial option when you’re trying to understand the “why” behind the success or failure of an edtech intervention.
The biggest difference between Tiers I, II, and III is the type of study: correlational, quasi-experimental, and experimental. These terms, basically, delineate how much a researcher is able to control for variables within the study.
In an experimental study (Tier I), researchers assign subjects to groups at random and are, therefore, able to speak with more confidence that the intervention causes the studied outcome.
A quasi-experimental study (Tier II) has the same aim as an experimental study but researchers aren’t able to assign subjects randomly, instead they use pre-existing groups. They use statistical matching methods that still allow them to speak with confidence about the causation of an intervention.
With a correlational study (Tier III), researchers cannot randomly assign groups and can only study participants using the intervention. Therefore, they are able to look at relationships between two or more variables but cannot assess causation of an intervention.
Common to all these types of studies is that they are well-designed and implemented and they seek statistically significant results. The terms “well-designed and implemented” are explained in What Works Clearinghouse’s 100+ page handbook. Most State Education Agencies define “statistically significant” as a result with a p-value that is less than or equal to .05; or to put it simpler, a result that only has a 5% probability of being random.
Tier IV is a little different, though. It’s the minimum level that education interventions need to meet in order to be considered evidence-based. To qualify for TIER IV, your edtech tool needs to have a logic model that’s based on high-quality research. This level of evidence doesn’t require you to actually demonstrate that your product improves student outcomes, only that it is “likely to” because it’s based on a well-founded research in learning sciences.
In addition to the logic model, there must be an effort to study the edtech tool that is either planned or already happening. Given this low barrier to entry, there’s no practical reason that every edtech tool in the market today shouldn’t qualify for TIER IV evidence.
There are two things that are important to note here.
First, we talk to early-stage edtech companies all the time who ask us if we can just go ahead and jump to conducting Tier I research on their behalf. And while that thinking totally makes sense (any edtech founder worth their salt absolutely wants to get the highest tier of research right away), that’s not quite how it works. The research tiers are about the journey of your product, and emerging edtech companies can get more fruitful and actionable data when they start by researching the foundational elements of their product then moving to impact-oriented research.
Second, and perhaps most important, now more than ever, schools need to find edtech tools that work for their students and can truly address learning loss. Collecting evidence about your edtech product is not a “nice to have;” it’s a requirement for any school leader looking to make a purchase.
How much does research cost?
As few as five years ago, the prospect of collecting evidence for early- and mid-stage edtech companies was daunting. Anything besides Tier IV evidence seemed out of reach except for industry incumbents with the cash reserves needed to fund a multi-year, experimental study. TIER I evidence can still cost a company anywhere from $250,000 - $3,000,000, according to NSVF.
Today, However, collecting TIER II and III evidence is more approachable. NSVF estimates the cost of TIER II and III at $45,000-$250,000. This entry-point of $45,000 can still be hard to justify when you’ve only started to generate revenue.
Given that edtech startups have a notoriously high customer acquisition cost and low user engagement, an implementation or correlational study can begin to seem like a good value proposition to shorten sales cycles and increase the lifetime value of a customer.
How do you collect evidence?
So, given that you’re at a stage where you’re ready to conduct research and understand how important collecting evidence is; the next step is figuring out how to make sure your product is evidence-based.
One way we’ve seen companies collect evidence is by hiring an in-house researcher or research team. This can be a big decision for companies that are focused on building out their sales and development teams. So, it can make sense to hire a third-party given your company’s stage. Aside from being a sometimes cheaper option, third-party evaluation also has the added benefit of giving more credibility to the research findings.
As you probably guessed by now, Leanlab offers a few different research options. We’d love for you to apply to the program, but we wanted to lay out some more options. Whether you’re collecting evidence with Leanlab, in-house, or with another third party, we believe that research and development that’s inclusive of the people that use your product is paramount to your organization’s continued success.
Below is a list of a few resources for edtech companies that want to get started on their research journey.
Leanlab Education offers concept and product validation, logic model creation, rapid cycle evaluation, usability, feasibility, implementation, and correlational research to companies that are addressing the needs of our partner schools. We also work with companies to find them school partners.
WestED offers usability, feasibility, and fidelity of implementation research
Companies can use resources from the Institute of Education Sciences to develop a logic model.
The federal government offers SBIR grants to small businesses to establish the technical merit, feasibility, and commercial potential of the product. Read our FAQ section here to find out how we can help with SBIR grants.
Whether you're just starting to consider research or you want to level up your research efforts, there are more and more options for edtech companies to truly embrace research as an ongoing part of their development.
