David Liu
David Liu, Knewton COO

In our recent series of interviews with adaptive learning powerhouse, Knewton, a number of interesting points were raised by our readers, which we thought were worth exploring further. So, we compiled some of the more challenging questions and David Liu, Knewton’s Chief Operating Officer, sent us his responses.


Is adaptive learning actually desirable?

Gavin Dudeney commented that he is unconvinced by adaptive learning:

“CAI is now back as “adaptive learning systems.” Some of the old programs have been repurposed with more interactivity. McRae states it as “adaptive learning systems still promote the notion of the isolated individual, in front of a technology platform, being delivered concrete and sequential content for mastery. However, the re-branding is that of personalization (individual), flexible and customized (technology platform) delivering 21st century competencies (content).” “


David’s response:

Classrooms are filled with students that learn at different rates; come from different educational backgrounds; and have different needs, attention spans, and interests.

Data analysis in education enables us to learn more about which content works best for which students. We can provide continuously updated predictive analytics that empower teachers and help them differentiate instruction — in and out of class — for individual learners. Educators can determine when an individual student is off-track, and identify gaps in knowledge at a granular level.

By helping identify particular concepts students have trouble understanding, we can provide more targeted interventions. Adaptive learning technology empowers teachers with information so they can offer hints to help students persevere through challenges.

While leveraging adaptive learning technology for language learning can be challenging, it is one of the areas of education most exciting for the promise of adaptivity. In ELT, students come from very different levels of proficiency and exposure, making adaptive learning particularly valuable.


Can language learning really be quantified and tracked?

Julie Moore wondered just how quantifiable and trackable language learning can (and should) be. Her experience of students who have been through education systems focussed on box-ticking and testing is that they develop no real feeling for the language and have poor communication skills. Could too much emphasis on quantifiable elements leave no room for the more ‘touchy feely’ elements of learning?

David’s response:

Yes. Language learning is certainly tricky because it is a multi-dimensional domain but for any lesson where there is a rubric we can leverage adaptive learning technology to analyze student progress. There is a misconception that adaptive learning technology only works in subjects like math. Knewton can power courses in a variety of subjects — including ELT, reading, maths, chemistry, biology, physics, anatomy and physiology, accounting, and finance. In general, in order to benefit from Knewton, learning experiences must be at least partially online and there must be generally agreed upon “correct” and “incorrect” answers.

Today, there are many organizations — from universities to MOOCs to hardware manufacturers — working to increase distribution of digital learning materials to students around the globe. Adaptive learning technology allows publishers (including ELT publishers) to personalize these digital materials for students. This allows students and instructors to better understand what they know, what they should work on next, and how they learn best.

Over the next ten years, many more ELT courses will be online and blended. We’re already seeing the shift from brick-and-mortar language classes to online options, including incorporating the matching of live real people to communicate and learn from each other. Many innovative language learning products are blended, either combining independent computer-based learning with a traditional classroom and teacher, or with online social media. Ultimately, as long as what the student is learning can be organized around learning goals, and these learning goals build upon each other over time, then adaptive learning technology can help personalize learning materials.


Can we truly measure ‘efficacy’ in language learning?

Julie Moore commented that “it’s very difficult to measure what’s effective in terms of language teaching and learning. There are some things that are relatively easy to teach and you know that students will pick up quite quickly, such as some vocab sets (which will show up as effective when tested). But other concepts, like more complex grammar, are slow burners, you don’t expect students to grasp them straight away. Does that mean that we just end up pushing out more of the ‘easy’ stuff in order to get better ‘results’?”

Michael Butler added that ‘machine learning’ has a place in that it can “ independently and tirelessly test students over a long period of time on what they have already learned … to make sure it sticks”, but asks whether there’s a risk of turning learning into a set of mechanical problems.

David’s response:

We don’t expect adaptive learning tools to replace teachers or humans in the language learning process. That’s never been the goal.  We understand the importance of interaction and conversation with a teacher or other learners.

However, there are many online tools to help the language learning process and analyze student progress, to help instructors and students understand what a student knows and what they need to work on. A Knewton-powered ELT course, for example, could pinpoint which types of grammatical structures students need to study, and provide targeted follow-up exercises to address those goals.

Achieving fluency in a given language is a subjective goal. As such it is difficult, and not necessarily productive, to quantify the efficacy of every effort toward this goal. However, there are smaller targets for which efficacy can be measured objectively — a student’s vocabulary in a certain subject area, understanding of the past tense, use of pronouns, the ability to converse about certain topics in certain situations, the ability to recognize the appropriateness of a greeting, the ability to distinguish between two close-sounding words, and more. As a student progresses through a digital ELT course or through supplemental online ELT materials, a proficiency-based adaptive learning system can estimate what that student knows and what they should work on next to best meet these discrete learning goals.


How can adaptive language learning work in a classroom situation?

Thomas Ewens felt that types of content described would be great for self-study, but wouldn’t work in the classroom, since the students’ work would not be in sync.

Michael Butler agreed that Knewton’s approach can’t be valuable unless the teaching process is standardized: “If teacher 1 spends 10 minutes on task A in a class of 20 and teacher B spends 20 minutes on the same task in a class of 6 how can the results be productively compared? They can only be compared after you standardize the time spent teaching, what is introduced in the class, how it is introduced, how it is practiced and indeed at which point the assessment test is given.”

David’s response:

Teachers for decades have understood the importance of “differentiated instruction” — the process of tailoring instruction to meet individual learners’ needs. Knewton helps educators evaluate what course content resonates well, with which students. We can also help educators quickly identify difficult concepts so that they can tailor lessons accordingly, or spend more time focusing on concepts that the majority of the class is struggling with.

A Knewton-powered course gives teachers a real-time snapshot of student achievement and concept-level proficiency. Instructors don’t have to wait until they grade the next test to identify weak spots. When a student struggles with a concept in a Knewton-powered course, he or she is guided to work on prerequisite skills. These skills are prioritized based on the strength of their relationship to the topic at hand, the student’s strengths and weaknesses, and learning goals set by the instructor (ex. preparation for an upcoming quiz). This frees teachers up from needing to pull together personalized prerequisite materials for each student. Teachers, armed with the knowledge of what might be causing frustration, have more time to sit down with individual students, spark group discussions, and orchestrate classroom activities.


What does adaptive learning mean for the role of the language teacher?

Thomas Ewans commented: “David Liu doesn’t mention teachers much in this interview, where does Knewton see itself in relation to supporting teachers?”

Michael Butler echoed the same concern, commenting that adaptive learning would be presented to teachers as ‘progress’, when it fact it would eventually result in teachers losing their jobs, even if “it probably will result in better learning in many instances”. He added that “the end result will probably be systems that benefit big business at the expense of teachers and small schools. In the end this is a system that will tend to concentrate wealth – distributing it from the many to the few in the name of efficiency.”

Michel also raised the question of how to ensure that publisher, student and teacher benefit from the data-driven approach.

David’s response:

Knewton doesn’t work without teachers.  Especially in language learning, there are technical nuances and cultural information that teachers must convey. Knewton aims to empower educators and give them more time to do what they do best — teach and inspire students. The data teachers receive from an adaptive learning system improves their ability to help students break down complex ideas, encourage critical thinking, point out relevant examples that will resonate with that individual student, and more.

Knewton provides analytics and tools to help educators tailor their lessons or differentiate instruction depending on what the class or a particular student needs. This helps teachers monitor performance and reduce administrative work. In a flipped classroom setting, language students could work on personalized lessons at home and then spend in-class time practicing the language with the teacher, going over the most difficult concepts from the night before, or engaging in any other activities the teacher feels will help a student succeed.


How does Knewton support low-stakes, non-assessed learning?

Paul Ricketts commented that “learners – and their teachers, the publishers’ direct clients, may have many different types of desired outcomes, many of which are outside Knewton’s traditional comfort zone of high stakes learning and clear goalposts like the CEF. What is clear, though, is that instructional design should benefit enormously from intelligent, nuanced use of Knewton’s analytical tools.”

David’s response:

New education technology is emerging that enables teachers, students, parents, and administrators to accurately and continuously gauge how well a student is doing at any given moment. Soon, it will be possible to measure students’ competency in a given subject — demonstrating exactly what they know, how quickly they learned it, and how well they retained it.

Today we can measure more than just answers inputted from an assessment. We can measure a student’s engagement with lessons, understand a student’s ability to persist through challenging material, and more. These types of analytics support the student but also help education content authors create better courses and content.

This technology will reduce the need for high-stakes standardized tests to try to estimate students’ mastery of a curriculum. Teachers and students will be able to spend less time stressing about test prep and more time exploring, problem-solving, creating, and developing a more engaging learning experience.

Is Knewton open to sharing the insights into language that its technology could potentially reveal?

Thomas Ewens commented: “The idea of analyzing data from thousands of students around the world is very, very interesting and could, potentially, give us new insights into how languages are learned much in the same way as how corpora have given us new insights into how language is used.

Does Knewton plan on disseminating this information publicly?

I would imagine,  for example, that Knewton would find patterns caused by L1 interference, i.e that Russian L1 speakers would learn certain grammar or vocabulary quicker than Japanese students, because that grammar or vocabulary also exists in their language, and vice versa. But it may well come up with insights that language teachers had never thought of before.”

David’s response:

We haven’t announced any specific plans, but we’re looking forward to uncovering and sharing insights over time. In addition to insights into language learning, Knewton can also uncover information about publishers’ content, which can help inform the creation of more effective ELT learning materials. With this information, publishers can create better courses, which further benefits students. Our mission is to help more students reach their full potential. To the extent that any generalized discoveries about student learning and ELT content can help in this regard, we’d be happy to share our findings.


Is ELT really ready for this?

Thomas Ewens pointed to the conservatism of educational institutions, and the fact that print is still far from dead. Are publishers taking a big risk in investing heavily in Knewton-powered digital courses when the bulk of the market still favours printed textbooks?

David’s response:

Yes. Adaptive learning requires that learning materials and assessments be at least partially digital. Until relatively recently the ELT industry was dominated by print courses, with add-on print and digital components. Now more and more courses are moving online, either fully or in blended formats. Publishers around the world are already experimenting with adaptivity to meet the needs of their tremendously varied and increasingly savvy audience, an audience that wants more personalized learning experiences.

This shift to digital, personalized learning materials is pushing ELT publishers to evaluate technical aspects of their product and understand how to deliver more targeted experiences. It also involves careful consideration of the pedagogy of a course, how it can be expressed in an adaptive product, and how that product fits in with the complete student experience. Publishers must consider how each product functions and why. For example, is the product only online or does it have a classroom component? What can be taught online and what should take place in the classroom?

We’re seeing rapidly increasing interest from top publishers around the globe, across many subject areas and grade levels, that want to leverage Knewton’s infrastructure platform to make adaptive learning products. Macmillan is creating next-generation ELT products, using Knewton technology to provide personalized recommendations and analytics. In addition, Cambridge University Press, among the world’s foremost ELT publishers, will use Knewton adaptive learning technology to power a new generation of digital ELT products.

Personalization is most effective via one large platform that analyzes vast amounts of data from classrooms around the globe. Building a self-contained adaptive learning app is expensive and unscalable. We’ve built a strong team of data scientists and learning experts that handle personalization so publishers can focus on creating quality content. Every application Knewton powers brings its own core competencies while outsourcing the heavy machinery of its personalization infrastructure to Knewton.

It’s beneficial for publishers to begin early, both to meet customers’ demands for personalized materials as well also to begin measuring how their content is doing — so they can make informed decisions about where best to focus their resources. Knewton is proud to power these next-generation adaptive ELT products.


  1. I’ve taken an ‘adaptive learning’ course online partly for the experience. I liked it and it worked well for me. Stayed motivated, challenged and enjoyed seeing the little graph fill up as I solved harder and harder problems more and more quickly. There’s a bit of that sort of learning in language learning. I would say the adaptive course was an improved version of a workbook or self-study program. That description in itself probably a reasonable way to sell this sort of thing to ELT professionals. Plenty of things that it can’t do that a good classroom teacher can (just like a workbook or exercise book).

    Knewton should hire someone who knows how to talk with ELT teachers, though. Painful to read David’s words.

    I’ll try it for him: ‘English teachers, here’s something that’s a little better than the old exercise book for homework. You might even look at the data it collects and judge whether it adds anything to what you’ve learned about your students from your classroom interaction.’ Very like an improved version of a workbook, which by the way never threatened to take a teacher’s job (nor was ever marketed to us teachers with such a lot pretentious nonsense talk).

  2. Reading that again, I’ve got it a bit wrong. But even if there’s 1 mention of efficacy in a 6000 word text, that’s easily more that 6.7 per million. It will add up to around 160 per million.


    1. Thanks, Thomas, you’re right. My maths was way off. The main focus of my comment was that the BNC wasn’t the best choice for a reference corpus, being so out of date.
      Keyword lists are great and Scott’s is particularly useful here. But I think it’s best not to focus on the figures too much. After all, they assume that if Liu had continued these interviews beyond 6000 words until he reached 10 million words (!!) the frequency of his use of ‘efficacy’ would have remained the same. Comparing with a much smaller corpus from a comparable domain – say, business or technology, would, I think, have yielded less dramatic figures. Comparing with a more similar-sized corpus of teachers discussing teaching would be very interesting.

      1. Laurie, we overlapped, and it looks like you’ve said what I was trying to say much more succinctly. I seem to have taken things a bit off-topic. Sorry for that.

        Jam Corpus. I plan to keep building this as it’s still very small (130,000 words) in corpus terms and I want to compile an appropriate reference corpus too. I built it because I’m fascinated by what we talk about and the language we use when we talk about edtech – especially educators. I’m happy to discuss more but I don’t want to take this off-topic any more than I have. You’re wrong about #2 btw – but because I don’t consider edtech a proper noun. Think companies and their products 😉

More comments

Leave a Reply

Your email address will not be published. Required fields are marked *


Other related posts

See all

New decade, new Jam!

My English learning experience – 6 lessons from a millennial learner

Deconstructing the Duolingo English Test (DET)