Investing in Research and Development Infrastructure through Edtech Testbeds
Summary
There continues to be a disconnect between what teachers and students need and what we know about the impact of education technology.
There is currently momentum behind increased R&D for edtech, yet we should consider why edtech R&D is so hard to do well.
We look to available funding for R&D, misaligned incentives (companies trying to stay afloat, researchers driven by research grants, schools seeking impact), and how conducting research in schools has historically been a cumbersome, inefficient and expensive process.
We posit a new way:
Influence the flow of capital toward research happening in school systems—agile, applied and formative research on technology that is occurring in real school settings.
Build a clusters of well-resourced testbed school systems in key go-to-market geographies (TX, CA, NY, etc) that effectively bend the market toward producing high-quality, evidence-based solutions that work to reduce educational inequities through the co-development and evaluation of tools in authentic and diverse learning settings.
January - I’m sitting in the living room of my great-aunt’s house in Puerto Rico watching my 15-year-old cousin try to memorize his multiplication tables. Since 2017, he’s experienced Hurricane Maria-related school closures, earthquake-related school closures, political upheaval-related closures, and Covid-related school closures. His older sister is quizzing him with pen and paper. His family is worried his basic skills have suffered, and—per most parents of 15-year-olds—they’re also concerned he spends too much time on his phone.
February - I’m sitting in the back of a KC public school library during a codesign session between school teachers, administrators and product developers. There is urgency in the comments made by the educators. They want solutions that not only work but accelerate learning. One teacher cites the concern of ACT scores hovering around 14, there’s just not enough time; how do we get students where they need to be?
March - I’m sitting in a conference room outside of Rome, Italy with some of the world’s leaders in education technology research, education startups, government and philanthropy trying to define what it will take to build a global community that supports the development and evaluation of education technology tools in authentic learning environments (known as “testbeds”).
April- I’m sitting at a private dinner with venture capitalists and entrepreneurs discussing the impending consolidation of edtech companies/tools, and, of course, discussing how we can verify practical uses of AI within the context of the classroom.
What do we take away from this? Sure, my life is charmed. And it’s also filled with complex paradoxes as I find myself straddling the space between student/teacher realities and the rooms where policy and funding decisions are made.
The reality is that despite the proliferation of edtech, we seem to still be missing the mark; my cousin isn’t leveraging his phone to remediate basic skills, and our educators are still desperate to understand what works and how. If we dig deeper, we reveal that this is a systemic issue; of the 100 most used edtech tools, analyzed by Learn Platform, only 26% had published studies aligned to the the ESSA tiers of evidence, and roughly 39% had published any studies at all. In a sector that has championed top-down reform initiatives that favor standardized testing and data driven instruction, the lack of insight into how technology tools impact learning is curious. The question we should be asking ourselves is why do we know so little about the impact of edtech?
Broken Markets
One theory is that US education is suffering from a “broken market.” In a nation of more than 13,000 individual public school districts, the solution providers (“supply”) operate in a fractured environment where the size of the contract is dictated by the student population, and where school systems (“demand”) have historically made purchasing decisions based on relationships and social proof rather than evidence. In this environment, solution providers prioritize geographies with large student populations and strong purchasing power. With limited budgets and strong pressures to increase market share, growing edtech companies are often faced with the decision to invest in relationship building with administrators at the expense of conducting ongoing third party R&D.
In a post-Covid landscape, however, we are beginning to see these breaks in the market narrow to fissures. District consumers are becoming savvier, and are making more thoughtful procurement decisions, in contrast to the frantic rate of adoption experienced during the pandemic. Navigating a crowded market and growing regulatory pressure, administrators are looking for products that are evidence-based and will be frequently used.
Who Pays for R&D?
Among the ways edtech companies can fund their research—government grants, philanthropy, venture capital, and self-funding—federal funding remains the largest and most viable avenue. The Institute for Education Sciences—with an annual budget of $808 million—currently limits their investment in edtech to efficacy studies and Small Business Innovation Research grants. Importantly, the new ARPA initiative allocates an additional $40 million for research and development, including funds earmarked for “quick-turnaround, high-reward scalable solutions.”
Meanwhile, an unprecedented amount of venture capital and private equity has entered into K12 education since the pandemic, though we’ve seen a resetting back to pre-pandemic levels. In the United States alone, $25.5 billion has been invested in edtech since 2010 while the market size for instructional materials has grown beyond $8 billion annually. These factors have flooded the market with new edtech solutions, yet the amount allocated to research from these investments remains to be seen. While some impact-oriented investors have begun to see value in research for edtech, for many VCs, this research is only valuable inasmuch as it contributes to a company's growth.
In contrast, big names in philanthropy are beginning to increase funding availability for R&D in an attempt to accelerate learning outcomes, particularly for Black and brown student populations. Together, this suggests that there is a considerable increase in resources available to engage in R&D. However, this also assumes that money is the answer—have companies been hesitant to engage R&D just because of the expense?
Or is the Pace of R&D the Problem?
On a conference call with venture-backed companies, founders discussed the felt experience of conducting R&D. The perception is that it’s time-consuming and arduous. It can take months to negotiate a data share agreement with a system, recruit participants, and navigate IRB processes. Given this protracted pace of research projects, when findings are revealed, they can feel out of date—the pace of technology has won, the product has already evolved, the findings, perhaps, irrelevant.
Still, for efficacy research to be conducted with validity, edtech companies must be able to demonstrate that there is fidelity of product usage, a big barrier for many (if not most) edtech companies, with the vast majority of edtech licenses purchased being underutilized. That said, the bias from companies and school administrators is toward large ‘n’ size studies that demonstrate that the product “works.” However, the reality is that it takes many cycles of experimentation and evaluation—and often fundraising—to get to studies that can demonstrate statistically positive results (for example, out of 11 studies Dreambox submitted to What Works Clearinghouse , only one “met standards”).
These challenges are compounded with disruptive, emergent technologies, like AI; as virtually all tech companies are now racing to implement different forms of artificial intelligence, how will we measure the impact or potential of these rapidly evolving technologies? How will we hold the companies accountable?
Look to the End Users: School Districts & Educators
One often cited barrier, but overlooked opportunity, to conducting agile applied research on technology in school settings is the capacity of schools to participate in research. In order to do the kind of product focused, agile research that technology demands with increased efficiency and accuracy, we need networks of school systems that have the resources, capacity, training and interest to generate meaningful insights quickly and seamlessly.
But historically, minimal resources have been allocated for applied and formative research—or research conducted in school settings that has practical applications—that can influence digital product improvement and/or instructional practice. Even fewer resources have been allocated to the learning environments and schools themselves to activate them as essential partners in the R&D process.
Imagine a world where school systems are equipped with staff that have the time and capacity to engage in professional learning about best practices in R&D and product development; data share agreements have been pre-negotiated; staff have the interest and capacity to engage in meaningful conversation with researchers and developers during school sanctioned PD time. Take this design a step further and ensure that historically-marginalized communities are well represented—that school systems serving Black, brown, indigenous, rural, and students of varying abilities—are adequately resourced to engage meaningfully in such R&D.
This reality is, in fact, within reach—and not without precedent. Imagine a lab school, but for the 21st century, or a research hospital. Recently, the term testbed has gained renewed momentum. Defined as, “The authentic learning environment where the EdTech Test takes place,” the concept of testbeds leans on more than a decade of work across the globe to understand how school environments can contribute to edtech development and research. (We understand the term testbed has negative connotations in the United States, and conversations to rename are ongoing).
But, what’s at stake if we don’t act?
Increased inequities. Just last week the New York Times published on post-pandemic recovery: “By 2022, the typical student in the poorest districts had lost three-quarters of a year in math, more than double the decline of students in the richest districts.” This paints a dire picture of an emerging generation of students where gaps in access to effective learning opportunities along lines of race and class has widened.
If technology is to live up to its promise of democratized access to learning, we must create systems that guide technology to be developed where it's needed most. When R&D occurs divorced from the context of authentic learning environments and the unique contexts of historically-marginalized communities, the chance of finding breakthroughs in the field that could radically impact all learners is diminished. It’s the equivalent of trying to bring a drug to market without an adequate lab, with no clinical trials, with no accounting for who had access.
Bending the Market : A Case for Testbeds
Herein lies the opportunity: In an environment where top-down federal regulation demanding rigorous evidence of edtech is unlikely, we must work to leverage market forces for good. To do so, we need to work to find the middle ground between solution providers' natural go-to-market inclinations and revenue obligations and R&D; where do those two intersect? School Systems and Educators.
We hypothesize that when we equip a diverse set of school systems and educators with the resources, capacity, training, and community necessary to engage in ongoing R&D, we will quicken the pace of R&D activities. If we cluster these school districts in key go-to-market states attractive to edtech developers, we can equip the field with the agile lab and clinical trial conditions necessary to accelerate R&D. If we create clusters of school systems that are truly diverse and representative of the diversity of the United States—with an explicit focus on historically-marginalized communities—we create the conditions to bring to bear a new generation of tools that work for all learners.
This means we must do two things
Influence the flow of capital toward research happening in school systems—agile, applied and formative research on technology that is occurring in real school settings.
Build a clusters of well-resourced testbed school systems in key go-to-market geographies (TX, CA, NY, etc) that effectively bend the market toward producing high-quality, evidence-based solutions that work to reduce educational inequities through the co-development and evaluation of tools in authentic and diverse learning settings.
Special thank you to Lora Kaiser of Center of Education Market Dynamics and Erin Mote of InnovateEDU for contributing to early versions of this post.
This post originally appeared in Groundshifts: A Newsletter on Public Education. Subscribe for early access to future posts.
