The Hard Things About Ed Tech

Or, at least, some of them

Part 1, where I describe a rosy, tech-enabled future, is here.

My favorite part of sharing writing publicly is the feedback and conversations that result. People say, "You should really talk to X," or out of the blue will say "I liked your post, but I'm not sure about Y." There were enough of these follow-up conversations I thought it would be worth another post to share what I've learned, after talking to teachers, founders, and other folks working in ed tech.

Education is close to all of our hearts. We've all gone to school, felt the shortcomings, and felt that we as a society can do better. The potential benefits of improved education are limitless - who wouldn't want to provide better opportunities and a love of learning to all? Many are tirelessly working towards this goal, from teachers to administrators to tech folks.

This excitement is tempered with the cold, hard realities of actually innovating in this space. It seems that the more time people have spent trying to innovate, the more pragmatic and/or pessimistic they become. Here are some of the key reasons:

  • The stakes are high with a child’s education, so it’s risky for administrators to experiment with new approaches.

  • The payer (school boards) is generally separated from those actually using the software (the teachers and the students), and budgets are tight.

  • Incentives are attached to improving standardized test scores, which ignores many aspects of learning and is antithetical to personalized learning.

  • Iteration speed, and thus improvement, is slow when one “experiment” happens per semester. In contrast, Google runs experiments and updates its search algorithm multiple times per day.

It's an industry that doesn't yield to the dreamer who is determined to stop at nothing to create change. Many dead ends must be avoided, and short-term compromises must be accepted, to even have a shot at making a small dent in the problem. 

Markets

K-12

An initial niche within K-12 is test prep and tutoring services to high school students. It's a fairly big market (slated to grow by over $8B by 2023), is quite fragmented, and the shift to remote learning should provide fertile ground for new digital learning tools. Some Chinese startups have been demonstrating some exciting models that could work in other markets.

However, you likely won't want to stay in this market forever. Test prep works for students that have the support and intrinsic motivation to hit the books, but you’ll generally be helping children from wealthy families save some study time. This is valuable, and it would be great to increase access to these services as well, but there are many kids who don't even have the support structures or motivation to pursue extracurricular tutoring.

The real goal in K-12 is to integrate with existing school systems. They have the most resources and leverage for changing the way students learn. However, school administrators err on the conservative side; money is always tight, and they're (justifiably) skeptical about a shiny new app actually improving their learning outcomes.

Photo by CDC on Unsplash

Selling to school systems often means making some sort of Learning Management System (LMS), which helps with administrative work associated with teaching like attendance, grade tracking, lesson planning, online distribution to students, and so on. These are focused on saving a teacher time so they can focus more on what matters. With the pandemic, there has been a flurry of activity in this space, essentially combining admin tools with a classroom adapted for video conferencing. There could be a niche for digital learning for certain subjects - the Khan Lab School does this for mathematics - but this doesn’t seem to be very common today.

An important question is, if a school system has an "extra" chunk of money, $50/student, what's the best way to spend it? Saving teachers time with better tools is helpful, but the benefits must be weighed against funding a new sports program, repairing a leaky roof, and giving the teachers a raise (which is sorely needed). The opportunity cost of money in education is very high, as are switching costs in a large and slow-moving system, so you’ll likely need to prove that your LMS or learning tool is 10x better to have a real shot. And that’s all assuming the school even has the ability to sign a new contract at that time, which could easily not be true.

Universities

Next we consider university students. Especially in the US, the costs are incredibly high, and the quality of the product isn't very good. People keep attending four year colleges because society tells them to, you still need the credential in most industries, and it provides a unique social experience and network that’s hard to get elsewhere. However, the edifice is cracking, especially during the pandemic. Why sign up for a lifetime of debt to learn course materials that are already available for free when most end up with dismal job prospects anyways? If you could provide a subset of these services at a lower time or monetary cost, and/or with better learning or career outcomes, you would have a great business.

University students also have higher freedom, agency and sophistication to seek out and adopt new educational and productivity tools, which could help drive initial adoption without dealing with the bureaucracies above.

Job Skills

We can also consider adults switching careers; coding boot camps are the prime example. People going through these programs are earning more, typically seeing salary increases of 50% or more, and in many cases finding more fulfilling work as well. There are many variations on this model, from intensive and high-touch ($10k+ for 3 months) to Udemy classes that are more self-directed ($10 for the same course material).

There is also a large market in training employees or users, whether it's agents in call centers to use their internal tools and embody the company voice, to knowledge workers looking to improve their management skills or observe their unconscious bias. The incentives here are a bit different - learning outcomes are key, but the company likely wants employees to learn the content more than the actual employee does. This could even be extended to training people outside of your company to use your products, from API documentation to educating customers to use your online portal.

For the rest of the post, I'll focus primarily on K-12, since changes there have the most compounding interest for the individual, and the curricula are generally focused and consistent.

Students

To rehash a Bezos-ism, we should stay Student Obsessed. Student needs and problems are quite heterogeneous. In K-12 education, there's a big split between those who are motivated to learn, have great support structured at home, and are generally wealthier, and those who lack support structures at home and generally struggle in school. The first group can definitely benefit from digital tools; they're already studying hard to get into a good school, and everyone wants better ROI on their time. Note that this still isn't "easy" - it's a competitive market, and parents are naturally skeptical of shiny new unproven tools. The app has to actually improve outcomes. For the latter group, the tool won't do very much if the students are stressed, have issues at home, etc. So the kids that are already on the right track do better, but the kids that are struggling continue to be left behind.

Scaling Up

This leads to a very tricky path to scale within the K-12 segment. You can focus on building that test prep tool, try to prove efficacy, and try to sell to a school board to make it the new standard for teaching high school math. But this has a few key problems:

  1. Growing a user base in education tends to be slow. Customers are conservative, and it takes time to refine these products and demonstrate efficacy. It's rare for software to go viral in education.

  2. If you take VC money, they're going to get impatient, and will likely push you to do suboptimal things to juice growth

  3. If you don't take VC money, progress and growth will be even slower, and you could be bootstrapping for many years (perhaps 8+ years).

  4. Once you're ready to do a pilot with an actual school district, you might discover you don't even improve their outcomes. The needs of the average student is different from your initial market, and you still aren't solving some of their core problems around support, safety and guidance.

It sounds like there should be ways around this. Is it guaranteed that education products grow slowly? Not necessarily. We should continue to question and challenge, but many smart, dedicated people have been unsuccessful at cracking this nut.

This probably indicates that we need deeper solutions than videos and assessments for students. We could tackle problems around better student structure and emotional support head on, perhaps with coaching, mentoring, better communities, tools that free up a significant amount of time and energy for teachers, or even coaching for parents. If we can solve these problems, it's going to be an even longer road than pure software.

This is why many believe that real change will come from policy. Increasing school funding will bring much needed relief to teachers and allow them to do more of the hard, emotional, supportive work that's needed. UBI or better earning prospects for low income families will help all kids get the support they need at home. None of these are massively scalable through technology, but maybe technology won't be ready to take on these burdens for years to come. Do we need superintelligent robots like Mother in I Am Mother, or can we make headway before? And when?

Scene from the Netflix movie I Am Mother, where an AI robot provides instruction and support throughout childhood.

These are some of the reasons why the folks I talked to were on the skeptical side for a massive, tech-enabled transformation in education - even the techno-optimists like myself.

Can Tech Really Improve Learning?

With all of that said, there is hope that technology can help with learning. There was general agreement that the following characteristics were desirable in a personalized tutoring system, and would be possible to implement using today’s technology:

  • Break up classes and concepts into small, atomic pieces of content and assessment

  • Teach these pieces at a pace that's appropriate for the learner

  • Use assessments to identify gaps in understanding, and revisit previous content to fill in these gaps (i.e. mastery-based learning)

  • Use spaced repetition to consolidate learning over time

  • Customize content to be as relevant and enjoyable for the student as possible

One interesting dimension I missed was how self-contained the field and background knowledge was. In mathematics, you can enumerate every concept, have full explanations, lessons and assessments for each, and keep surfacing the right content at the right time to get a full understanding. For biology, this is harder, since there are more concepts, and they overlap with other topics and real-world examples in myriad ways. So the complexity of generating a mastery-based adaptive curriculum for these subjects can be much higher.

Another crucial concept I missed was considering different levels of learning and understanding. Teaching facts is very different from creating and assessing a creative project to synthesize disparate concepts, and ideally we’d want our dream educational system to address both. Today, the former could likely be taught by an AI, but the latter might require much higher intelligence on theoahmac part of the tutor. It's also true that group settings (e.g. a seminar of 10 people) can be better for higher-level learning, while solo is probably just fine for facts.

Bloom's Taxonomy breaks learning down to two dimensions: Cognitive Processes and Knowledge. This is used to help instructors at Columbia create better courses and curricula. This can help the creator of an AI teacher focus on the right level of learning, while providing a roadmap to work up the stack over time.

Closing Thoughts

This is a more pessimistic post than Part 1, but it's important to confront challenges with open eyes. And the more we understand the problem, the more real solutions reveal themselves.

Many thanks to those who helped provide feedback and input to this piece, including Arjun, Milica, Ahmed, Jeremy, and Rishi. Much of this content is from their ideas and prompts. Thanks for reading!