Most Read

Humans in the Loop: Insights from Codesigning AI in Real Classrooms

With the explosion of AI in education since ChatGPT’s release in November 2022, the edtech market has seen an influx of tools promising revolutionary changes to teaching and learning. But are these tools meeting real classroom needs? Leanlab Education set out to answer this question by working with five edtech companies, each developing AI-powered solutions, through codesign research involving over 40 teachers and their students across the U.S.

The Great Divide: Teachers' Expectations vs. AI Reality

Teachers bring three core expectations to AI-powered edtech. They expect:

  1. Tools that save time.

  2. Support for differentiated, engaging instruction.

  3. Assistance with instructional design, delivery, and feedback.

These expectations aligned closely with the intentions of the edtech companies developing AI-powered tools involved in our studies. Their logic models and theories of change emphasized four primary goals:

  • Personalize and differentiate learning to meet individual student needs.

  • Enhance student engagement and increase agency in learning.

  • Improve student outcomes by supporting educators and addressing diverse classroom needs.

  • Support educators in designing, delivering, and providing feedback on instruction.

While teachers' expectations and edtech company intentions are well-aligned, Leanlab’s studies revealed significant gaps in practical application.

Across four Leanlab studies with teachers and students, several issues consistently impacted AI tools’ effectiveness:

  • Technical issues like slow load times disrupted learning and wasted valuable classroom time.

  • Differentiation shortcomings meant tools lacked flexibility to meet diverse student abilities, hindering personalized support.

  • Trust in AI-generated content was low, with teachers questioning accuracy and reliability.

  • Integration challenges limited how seamlessly tools fit within existing classroom workflows.

  • Teacher familiarity and confidence with technology varied, affecting both adoption and effectiveness.

These barriers highlighted a core issue: while AI-powered tools aim to save time and enhance learning, their actual usability often falls short, leaving educators questioning the value of incorporating these tools into their classrooms.

Codesign Research: Bridging the Gap with "Humans in the Loop"

Leanlab’s codesign approach—facilitating collaboration between educators and developers—proved instrumental in addressing these gaps. Teachers provided ongoing feedback that informed real-time product changes, fostering improvements in both functionality and teacher trust. For instance, when teachers identified differentiation limitations, Leanlab recommended adding scaffolding to better support learners at various levels. This iterative process not only enhanced tool utility but also empowered teachers, who felt heard and valued in the product development process.

Building Trust and Usability in AI Edtech

Leanlab’s research demonstrates that embedding “humans in the loop” through codesign is essential to creating trustworthy, effective AI tools. This method helps companies validate that products address real classroom needs while evolving based on educator input. By collaborating with a diverse group of teachers and students early on, companies can ensure their tools are equitably designed and accessible across various educational contexts.

At Leanlab, we believe that codesign research is crucial for closing the gap between AI-powered edtech’s promise and reality in education. Interested in exploring how codesign research can shape your product? Discover Leanlab’s approach to creating trustworthy tools that align with teachers’ needs and ultimately transform education.


Codesign Product Research

Curious how codesign research can help bridge the gap for your AI-powered product? Learn more about Leanlab’s approach to creating trustworthy and effective tools by diving into Codesign Product Research. Let’s shape the future of AI in education, together.


In the Age of AI, Edtech Needs Inclusive Innovation More Than Ever

This post originally appeared in Edtech Insiders.

The post-pandemic era has led to learning loss and an increased dependence on edtech tools—however,  the majority of edtech tools still lack credible evidence supporting their efficacy. One 2023 report noted that only 39% of the most-accessed learning solutions support their usage with published research, while only 26% support their usage with studies aligned to federal standards. This is true despite the implementation of the 2015 Every Student Succeeds Act (ESSA), which clearly outlined four tiers of evidence to support edtech adoption in schools

Why have so few edtech products engaged in efficacy research, and especially few in objective third-party research & development or evaluation studies?

  • Some attribute the lack of evidence to the lack of demand for evidence among school purchasers. Historically, many districts have relied on peer recommendations rather than a tools’ evidence base to make purchasing decisions.

  • Others note the noisy credentialing environment, in which organizations with no regulatory authority badge and credential products, makes it hard for decision makers to know which products are truly evidence-based.

  • Still others point to the cost of research—the painful, expensive, and time-intensive third-party evaluation processes that often don’t keep pace with the realities of a fast-moving tech sector. 

However, as a flood of new AI-enabled edtech enters the market at the same moment that schools face the ESSER funding cliff, buyers are starting to cast a more discerning eye on which tools are truly moving the needle on outcomes.

Raising the Bar for Edtech Adoption

This June at ISTE Live, one group of education leaders sought to support the field by outlining some top considerations purchasers should consider when adopting new edtech solutions:

  • Does the tool have a credible evidence base?

  • Will the tool help us address a critical learning need?

  • Is tool adoption and implementation within our budget? 

  • How much time will staff need to commit?

  • How will instructional time be impacted?

  • What data will be collected?

  • Is the tool safe for use with our students (e.g., will student data be collected)?

  • Is the tool just one more thing/initiative for our teachers to deal with (AKA: does the cost of getting buy-in outweigh the reward)?

  • What are the edtech company's intentions?

Simultaneously, a coalition of the sector’s leading edtech arbiters—1EdTech, Cast, CoSN, Digital Promise, InnovateEDU, ISTE, and SETDA— launched five Edtech Quality Indicators to bring more cohesion to the field. Consistent with education leaders' needs, the five indicators state that quality tools should be safe, evidence-based, inclusive, usable, and interoperable

These parallel developments point to a potential shift in the market; nonprofit organizations are engaging in collective impact work to send cohesive and consistent signals into the market, while purchasers are seeking safe, vetted, streamlined, and cost-effective solutions. 

Inclusive Innovation: Preventive Medicine for Edtech

While there is a need for more evidence measuring the potential of edtech, the evidence that does exist is promising… at least for some populations. Laurence Holt’s remarkable EdTech Next article “The Five Percent Problem” cites studies demonstrating considerable student learning gains from leading products Khan Academy, Dreambox, and IXL. The problem he points out is that these gains were realized only by “students who used the product as intended.” How many of the students “use the product as intended”? Often, as few as five percent of students studied, and often those with additional benefits. The obvious question: what do we do about the other 95%?

Enter inclusive innovation, a R&D approach that aims to increase the usability and effectiveness of emerging solutions by focusing on the process by which products are created and evolve. Inclusive innovation refers to a systematic process that elevates the insights of users representative of the diverse backgrounds, needs, and abilities in American classrooms. 

The theory underlying inclusive innovation is that when products are:

  1. Developed in close proximity to diverse, representative users

  2. Designed to incorporate the feedback of diverse users

  3. Built on a pre-existing foundation of evidence-based best practices

… the resultant products are more likely to be implemented as intended, and thus more likely to demonstrate impact on student outcomes. 

We see inclusive innovation as ‘preventative medicine’ for edtech companies. By incorporating early touch points and feedback loops with diverse student populations while products are being developed, companies can expect reliable usage that drives long-term results. 

Various organizations in the field describe such inclusive innovation practices, as ”inclusive R&D”, “co-design”, “co-creation”, “co-development”,” inclusive design”, and “participatory methods”. In practice, all variants of inclusive innovation involve engaging teachers and learners early and often during product development. Whether it’s the creation of user surveys, the development of research questions for focus groups, or the work to match an edtech company with a school to co-design an efficacy study that addresses school-identified goals, inclusive innovation practitioners are working directly with learner communities. 

The Importance of Inclusive Innovation in AI-Powered Edtech 

While AI has the potential to revolutionize education through personalized learning experiences and increased teacher efficiency, it also has the well-documented potential to perpetuate bias, threatening to exacerbate existing inequities in education.

The speed at which AI technologies are developing is rendering traditional research methods (which can take multiple years) unhelpful or obsolete. So, how might R&D help guide AI-powered edtech toward its full potential; to advance educational equity rather than threaten it? 

Rather than eschewing research altogether, we believe that inclusive innovation practices—like rapid-cycle evaluations and iterative co-design—can be leveraged to produce timely and actionable findings for companies. But what exactly are these approaches and how do companies access them?

Many organizations are engaging in this work, often through collaborative efforts. Here are a few examples from the field.  

Leanlab Education: Building Communities and Co-Designing Innovations

Leanlab Education is a nonprofit with a mission “to study and grow transformational education innovations co-designed with school communities”. Leanlab acts as the intermediary between edtech companies and school environments, matching partners on projects and leveraging its in-house research team to facilitate product feedback and research studies aligned to the ESSA tiers of evidence. By involving educators and students in the research design and product development process, emerging learning innovations address real-world challenges and meet the needs of diverse learners—intentionally designing for the margins. 

By pairing enthusiastic education partners trained in inclusive innovation methods with companies seeking third-party research, Leanlab has increased access to inclusive R&D; to date, Leanlab has engaged over 30,000 students in nearly 80 studies. This approach ensures that emerging innovations are equitable, user-friendly, and a value-add to school communities, and thus more likely to demonstrate significant impact.

Central to this work is the newly-launched American Group of Innovative Learning Environments (AGILE) Network, a cohort of public schools, charter schools, and nontraditional learning communities ready to partner in R&D through a centralized (and co-designed) infrastructure. While managed by Leanlab, the AGILE Network is a collective impact initiative built on findings from the Global Edtech Testbed Network and supported by a growing number of partners—including founding partner AERDF, Digital Promise, and ISTE. 

ISTE: Edtech Index 

ISTE (along with acquired non-profit ASCD) is a global education community working to accelerate innovation in education through the smart use of technology. Well-known for their marquee conference, ISTE Live, that convenes more than 17,000 educators annually, ISTE is now using its reach to elevate the importance of evidence-based edtech and inclusive innovation. 

ISTE’s Edtech Index consolidates information and validations on over 1,500 edtech products in one accessible platform, as well as a free teacher ready evaluation tool that helps educators determine the usability of edtech products for their contexts. The Edtech Index measures products aligned to the five domains of quality identified by the partnership of groups in the field: safety, evidence-base, inclusivity, usability, and interoperability. 

Digital Promise: AI, Equity, and Inclusive Networks

Digital Promise’s efforts to advance equitable education systems through R&D takes various forms, including AI literacy and digital equity initiatives, work through their Center for Inclusive Innovation, and the Learner Variability Project. By conducting rigorous human-centered research, making findings accessible, and providing resources for educators, Digital Promise helps create an environment where all students can benefit from edtech advancements. 

The Center for Inclusive Innovation reimagines education R&D by building the capacity for districts and communities across the country to engage in R&D, with a focus on supporting students furthest from opportunity. After their network partners co-design and engage in R&D, project details and solutions are shared with the larger field, highlighting R&D practices in diverse contexts. Digital Promise has also been an early leader in AI in education policy guidance, creating an AI Literacy Framework, leading workshops for school leaders, and advocating with companies, government, and schools for keeping humans in the loop.

AERDF: Finding Bold and Equitable Solutions

The Advanced Education Research & Development Fund (AERDF) maintains an extensive portfolio of inclusive R&D projects that incorporate learner and educator perspectives from start to finish. Inspired by federal Advanced Research Projects Agency (ARPA) initiatives, AERDF’s focus on equity-driven R&D ensures that breakthroughs are developed that address systemic inequalities in education, with a focus on Black and Latino students and students experiencing poverty.

AERDF’s advanced R&D projects involve sustained collaborations with educators, researchers, and communities to co-create solutions that are effective and inclusive. Their work creates bridges across the ecosystem, tackling challenges like assessments, math outcomes, and literacy, and showcases their findings for the larger field. 

Chan Zuckerberg Initiative: Partnering to Co-Build

As a funder and collaborator in the edtech field, the Chan Zuckerberg Initiative (CZI) supports the development of evidence-based, co-created personalized learning experiences through the use of technology. CZI partners with organizations conducting and promoting inclusive R&D–including Leanlab Education, Digital Promise, and ISTE–as well as educators and students to co-build tools and reimagine education.

CZI engages in co-building through their edtech innovation studio, Render. With their collaborative product development approach, Render is creating AI-customized math materials, tools focused on elementary literacy, and administrative applications for classroom grouping. Reflecting their inclusive approach, Render’s co-building opportunities are posted online and matched to educator interests.

The Path Forward: Scaling Access to Inclusive Innovation 

As AI continues to reshape the edtech landscape and schools look for breakthrough solutions that improve outcomes for all students,, there is an opportunity to establish inclusive innovation as an expected norm. Stakeholders across the edtech ecosystem—educators, researchers, solution providers, funders, and policymakers—can all engage in inclusive innovation and guide the transformation of K-12 education. 

Through collective action, we can ensure the future of edtech, particularly AI-powered edtech, is not only innovative but also inclusive. 

Here’s how each group can get started:

  • Educators, district leaders, and students can get involved in R&D projects, with opportunities aligned to their needs, interests, and capacity. Leanlab Education, In Tandem, Transcend, and Digital Promise are only a few examples of robust innovation communities. 

  • Schools, when seeking out new edtech, can check validations in the Edtech Index, use Evidence for ESSA or other reviewing bodies, or include inclusive design criteria in RFPs. 

  • Solution providers can co-build promising edtech with the communities they aim to serve, tapping third-party research organizations that use inclusive R&D methods or design internal R&D efforts on co-design principles

  • Funders, policy makers, and solution providers can leverage open research, guidelines, and other resources to support inclusive R&D efforts, improve developing edtech, and collaborate more effectively. 


AGILE Network Launches to Accelerate Inclusive Education R&D

The American Group of Innovative Learning Environments (AGILE) Network offers a centralized infrastructure for research, development, and evaluation.

Research nonprofit Leanlab Education launched the American Group of Innovative Learning Environments (AGILE) Network at ISTELive24 with members from learning environments across the country, including school systems, teachers, and non-traditional learning environments. Comprised of education communities on the cutting edge of innovation, the AGILE Network works to accelerate the pace of education research and development (R&D), while elevating historically underrepresented educator and student perspectives. 

By leveraging a centralized infrastructure that facilitates “matching” between R&D opportunities—development of emerging technologies and research studies—and participating schools and educators, the AGILE Network aims to become a critical piece of American education R&D infrastructure, designed to keep pace with rapid development cycles in the private sector. 

Member schools and educators influence the development and evaluation of emerging education solutions, including curriculum and technology—addressing a well-documented evidence shortage in the market. Studies in the 2024-2025 school year will cover topics including generative AI, literacy, assessment, and mathematical reasoning. 

The AGILE Network represents the diversity of learning experiences in the United States, welcoming public districts, charter schools, independent schools, non-traditional environments like micro-schools, and individual educators. To combat existing inequities in education R&D, members have been intentionally assembled to prioritize historically marginalized learner communities. Twenty learning environments are joining the 2024-2025 cohort, including:

  • Academy for Integrated Arts (Missouri)

  • Allegiance STEAM Academy (California)

  • Astra Nova Schools (National)

  • Catalyst Schools (Illinois)

  • Distinctive Schools (Illinois)

  • Ednovate (California)

  • Limestone Community School (Kansas)

  • Milpitas Unified School District (California)

  • Oak Ridge Schools (Tennessee)

  • Prisma Online School (National)

Of the new network, Leanlab Education Founder and CEO Katie Boody Adorno shared, “The AGILE Network will reduce barriers to engage school communities in the co-development and co-design of emerging technologies with researchers. By systematically elevating school community perspectives, we believe we will accelerate the pace of breakthroughs in the field of teaching and learning.”

Following the kickoff convening, Kevin Nham from Ednovate remarked, “I’m looking forward to partnering with edtech companies to develop tools that will truly impact student achievement.” Catalyst Schools’ Nick Tomasso added, “I’m excited about opportunities to grow in my own practice by collaborating with like-minded educators.”

The AGILE Network is a collective action initiative and builds on a decade of Leanlab Education’s leadership bringing together learning communities and education technology companies to co-design, co-develop, and study education innovations. The AGILE Network framework was developed through a planning grant with InnovateEDU and the Center for Education Market Dynamics (CEMD) and is based on the guiding Tenets and Principles published by the Global Edtech Trialing Network. 

Advanced Education Research and Development Fund (AERDF), the world’s first Advanced Inclusive R&D organization dedicated to scientific discovery and invention for PreK-12 education, is a founding partner, contributing essential R&D infrastructure expertise, resources, and advanced R&D projects currently seeking breakthroughs in reading, executive function and math, and assessment. “The AGILE Network serves a critical need to enable education R&D to actually be inclusive and done with educators,” said Chris Liang-Vergara, Chief Learning Officer at AERDF.

Digital Promise and ISTE are contributing network partners, nominating qualifying educators and school districts, and supporting inclusive innovation convenings. Initial funding was provided by the Siegel Family Endowment and Walton Family Foundation, among others. Education, nonprofit, and supporting partners are expected to increase following the pilot year of the network. 

Grief, Pressure & Connection: A Primer on Education in a Post-Pandemic World

While private schools saw limited interruptions, some stable, public school districts have used the catalyst of the Covid-19 pandemic to progress personalized learning initiatives and implement more experimental curricula. However, increasingly more public schools are struggling with bare necessities—extreme shortages of bus drivers, food service delivery, and substitute teachers. These operational obstacles are now bleeding over to disrupt instruction.

2021: An Imperative to Innovate

bima-rahmanda-4mPo9OVEg7g-unsplash.jpg

In education, we’ve been standing at a crossroads for the last 10 months. The decisions we make now will dramatically shift the education sector--perhaps permanently.  I know you know this. Like me, you’ve been ruminating on this intuition.  For those of us working day-to-day in this sector, the weight of these decisions is real. Palpable. The tenuous footing we stood on a year ago has become even more unsteady; our entire sector is undeniably in flux. Is this what disruption feels like?

I’ve felt this with my five-year-old nephew and my sister-in-law. He’s a kindergartner at a Kansas City charter school and she’s a working mom.  He accesses the only public school he’s ever known through a screen, from a dining room chair with his Abuela’s steady support. 

Our family dinner conversations have shifted. We talk about learning management systems instead of playground gossip. Everyone wants to know, “why is it so hard to get all these apps and platforms to play nice together? My sister-in-law  talks excitedly about her ideas for product modifications to her son’s learning management system. She has no background in technology development, but still she envisions improvements. She longs for more streamlined integrations, more information about how to customize and advance her son’s love for math, and guidance on how to nurture the areas where he needs more support (reading).  

My conversations with teachers have shifted. During one-off Saturday mornings or stolen lunch breaks, teachers tell me earnestly, that for some children, the virtual environment is working better! Working remotely, teachers  have better systems to differentiate instruction and provide more personalized support. However, they also have students who are struggling; students who have fallen off the radar, are rarely logging on, or have challenging home learning environments.

”I find that I’m actually a more successful educator this year. I’m more focused, can give more targeted attention,” one teacher says, while also noting, “but if we do go back to business as usual, and the status quo... I’m not sure that I can.”

I make time to listen to teachers first-hand experiences, but I also observe their actions. Teacher unions are understandably advocating for delayed returns to school until they’re satisfied their environments are safe. They want fully-executed vaccination and testing plans, resources for proper social distancing and sanitization measures, revised distance learning plans, and enough substitutes on hand for support (CTU reopening demands).

The prolonged delays to returning full-force to the classroom may seem especially protracted, but I suspect they also speak to a deeper longing from our collective unconscious… a desire  for a paradigm shift that would affect the entire industry, then  ripple outward to all professions… a declaration that we’re not going back… not back to the way things were… we can’t.

Am I advocating for a shift to an edtech-driven universe, the demise of brick and mortar classrooms? No. Our research from the last year illuminates persistent inequities and early evidence suggests learning gaps have been exacerbated by this pandemic (PACE). There’s no question that children need safe, nurturing, developmentally appropriate places to socialize and learn, but now we need to grapple with what a more dynamic, customized and hybrid environment should look like for the long-term.  

We’re not going back… not back to the way things were… we can’t.

And yet here we are--in a new calendar year, either staring down the barrel or standing on the precipice. Do we return to slow, incremental change--to structures and models that have historically done little to solve for the persistent inequities, painfully illuminated by this pandemic? Or do we acknowledge where we’ve been wrong, commit to making a change, and begin placing new bets? 


What we’re musing

Can accountability be shifted away from classrooms, students and teachers, and onto systems, leaders and edtech tools?
We aren’t holding doctors and nurses single-handedly accountable for the reduction in spread of Covid-19. Instead, we are equipping them with vetted  vaccines. After months of research, clinical trials and testing to ensure the vaccines were sound, we then understood under what conditions the vaccines were most effective (they have temperature requirements, expiration dates, etc). It was then that elected leaders were responsible for leading distribution.

So, I wonder, can we transfer this idea to the education sector? 

  • What would a world look like if we had a structured system to vet and test education tools, ensuring they were based on foundational learning science and were easy to use across a variety of learning contexts (grandma’s dining room, the classroom, the community center)? 

  • What if we had a clear, evidenced-based guide to understand what conditions these solutions achieve their intended outcomes? 

  • Could we then provide system leaders with the information they need to buy best-fit solutions, while equipping educators with implementation roadmaps that tell them  how to modify instruction toward optimal outcomes (i.e. “for best results…”)? 

  • That’s what I want to test.  

I envision a  world where educators spend less time on arduous content creation. They are no longer beholden to unimaginative, unproven and clunky tools, or constrained by the heaviness of standardized tests. Instead, they are freed up to invest their time on building meaningful relationships with students and their families, developing more customized and targeted interventions and designing experiences that bring learning to life.

In this world, rather than relying on overly cumbersome and high-stakes assessments, teachers leverage assessments to inspire agile, data-driven student interventions. The accountability then, shifts. Imagine a world where we hold our state elected and appointed officials and school system leadership accountable to the expedient and equitable distribution of evidence-based, trialed solutions.

Can this be the year where we begin trying something different? We’ve loosened restrictions on standardized testing for the last year. Why not re-envision education all-together?


Conclusion

The days ahead will not be  easy. I’m reminded of the Maya Angelou quote, “I did then what I knew how to do. Now that I know better, I do better.” We have arrived at this moment. We now know better.  

Am I suggesting that edtech is a panacea, a one-shot vaccine? No. But I am suggesting that we are in a new age of rapidly-evolving technology. We owe it to our students and to future generations to begin developing new tools, new methods, new accountability practices and more dynamic learning environments to reach learners wherever they might be.  Now, let’s do more, better. 


How Do We Learn if Ed Innovations Really Work?

How Do We Learn if Ed Innovations Really Work?

How do you learn whether music lessons can increase social-emotional learning; if inquiry-based learning can increase student engagement; or if an app can increase a student's STEM awareness? The entrepreneurs in our program worked diligently last year in concert with educators at our pilot sites to find out.