One of the promises of digital publishing is the ability to improve and update a product on a continual basis. We recently posted about iterative publishing. In order for that to work, you need to know which parts of your course are working well and which are failing to meet their objectives. In part 3 of our interview series with Knewton, David Liu (Chief Operating Officer) and Sally Searby (Partnership Manager) explain how they believe their data analysis can provide the insights needed to refine and improve ELT courses, going beyond what even the most knowledgeable and experienced author/editor team could do.

Do you want to say a bit more about how [Knewton’s data analysis] will improve ELT products on an iterative basis?

With the data, each of the publishers gets analytics. And there are analytics at different levels. There are teacher analytics, which of course help the customer of the publisher: they can group students into optimal study groups, they can really do deep dives on students, and really understand what’s happening with each student. So that’s a very visible, marketable benefit.

We help publishers and content creators to evaluate content efficacy at the concept level and create more effective learning materials

But for the publishers themselves, we have the ability to provide content efficacy. We help publishers and content creators to evaluate content efficacy at the concept level and create more effective learning materials.

A good example is when we look at the knowledge graph of our partners, which is a map of how concepts relate to other concepts and prerequisites within their product. There may be two or three prerequisites identified in a knowledge graph that a student needs to learn in order to understand a next concept. And when we have hundreds of thousands of students progressing through a course, we begin to understand the efficacy of those said prerequisites, which quite frankly were made by an author or set of authors. In most cases they’re quite good because these authors are actually good in what they do. But in a lot of cases we may find that one of those prerequisites actually is not necessary, and not proven to be useful in achieving true learning or understanding of the current concept that you’re trying to learn. This is interesting information that can be brought back to the publisher as they do revisions, as they actually begin to look at the content as a whole. Is there a return on your investment in producing some of this content? It costs money to build the textbook, and even in a digital world it costs a lot of money. So, if we can provide some content efficacy back to the publisher on how not only that book, but how certain constructs and chapters are working, publishers can be smarter about how they invest in those areas.

We’re beginning to value both the bundled content, which is traditionally textbooks and even online textbooks, but also how to unbundle certain content to be able to provide subscriptions to certain areas. And, if we can do that, then you need to be more efficient with your investment.

So those are a couple of benefits. But I think overall, analytics and understanding what is actually happening with you customer, with students and teachers, is important. You can begin to provide value to schools and help teachers differentiate instruction or understand what concepts a class struggles with, or even help administrators at a more macro level.

And the ELT publishers are all talking about how to get closer to their customers through digital content and delivery and analytics – through measurable data in order to improve the learning experience through better content.

better insight changes the role of the publisher

Yeah, I think generally better insight changes the role of the publisher. You’re providing awesome content obviously, you’re providing a great user experience, but now there’s insight into what’s happening, and I think that, we hope, provides a bit of an evolution in the business model. Because it’s not about binding and printing and sales and distribution only.

I think one of the interesting challenges for publishers is actually going to be using those insights. How quickly and how effectively will they be able to take the insights that they can gather from this kind of data, and then put them into practice in the next iteration of the product? I think that’s going to be an interesting thing for publishers to grapple with once the data starts coming in.

Absolutely. In ELT I find it to be a fascinating issue, because it’s so big, but also what we’re learning in some cases is that ELT is very successful in what it does in an offline mode. As the world in education is moving online, how does ELT transition? We’re seeing a lot of different interesting models being tested out around that transition.

There’s a lot of opportunity to provide the same great content or even better, if you can show some efficacy and put the best content for the individual student in front of them. Again that’s another huge mindset change. It’s not just the best book overall – what does that mean? The best book for the average of a million people may not be the best book for me, or Sally, or Molly.
And so it’s not even about the book, it’s about that particular concept or that chapter or that module that you’re working on. I think it’s an incredible opportunity to keep the best of what has been working in ELT and publishing, and education in general, but then leverage these new tools and technology.

And I think the notion of programs lasting 15 weeks or 10 weeks – the time-based kind of paradigm that we’ve been in, or session-based paradigm, gets blown up. What happens then? What you know becomes more important and students can move at their own pace. We’re seeing a shift even in physical universities in the United States and abroad. The University of Southern New Hampshire and other schools are moving to competency-based programs.

It’s just an exciting time to be in education. And with ELT specifically, it’s such a global business that I think they can leverage adaptive learning technology more in some respects than others, because you’re talking about students learning similar things around the world. That’s a lot of data. The learnings that you can gain are tremendous if you’re an ELT provider.

Knewton office


Knewton interviews

Part 1 – Big data and adaptive learning in ELT

Part 2 – Sharing data and competitive advantage

Part 3 – Powering iterative publishing in ELT

Part 4 – Can adaptive learning really work for languages?



  1. @ Thomas Ewens re: your point one.

    I think publishers are moving to digital publishing for financial reasons and competitive reasons not because they suddenly became convinced it is the best thing for teachers or students. If that were the case you probably would have gradually seen better (easier to use) digital solutions long ago. It took ages for publishers to move to MP3s vs. CDs or tapes. Many still don’t offer MP3s. They are choosing these solutions to protect their bottom line and I don’t particularly blame them.

    I think that they are scared to death of open-source solutions and they see digital publishing as an end-around solution to “paper-based” free and low-cost self publishing.. I think they see digital publishing as a way to reassert their dominance over the educational distribution chain. The reason: for now this kind of publishing, if done well, is high-cost and capital intensive.

    I think publishers will soon be in a race to pry people away from the very materials that they thought so highly of 5 years ago.

  2. I don’t see how any large data sets can be valuable unless the teaching process is completely standardized. If teacher 1 spends 10 minutes on task A in a class of 20 and teacher B spends 20 minutes on the same task in a class of 6 how can the results be productively compared. They can only be compared after you standardize the time spent teaching, what is introduced in the class, how it is introduced, how it is practiced and indeed at which point the assessment test is given.

    I am sure the publishers realize this. Given the same book and two different teachers you most likely will produce two different outcomes and many of the outcomes may not be apparent on your test.

    It seems to me that this model leads to one outcome– a more rigorous control of the learning process. It probably leads to some kind of machine based instruction and testing in the end. I’m not sure I entirely mind this. What I do mind is that the people who are advocating this system say that big data is in the teacher’s best interest. They have a long way to go to convince me of this. The need to build some real protections in that insure that publisher, student AND TEACHER benefit from this system of open sharing.

More comments

Leave a Reply

Your email address will not be published. Required fields are marked *


Other related posts

See all

Am I a Content Creator or a Writer?

Deconstructing the Duolingo English Test (DET)

My English learning experience – 6 lessons from a millennial learner