One of the promises of digital publishing is the ability to improve and update a product on a continual basis. We recently posted about iterative publishing. In order for that to work, you need to know which parts of your course are working well and which are failing to meet their objectives. In part 3 of our interview series with Knewton, David Liu (Chief Operating Officer) and Sally Searby (Partnership Manager) explain how they believe their data analysis can provide the insights needed to refine and improve ELT courses, going beyond what even the most knowledgeable and experienced author/editor team could do.
Sally:
Do you want to say a bit more about how [Knewton’s data analysis] will improve ELT products on an iterative basis?
David:
With the data, each of the publishers gets analytics. And there are analytics at different levels. There are teacher analytics, which of course help the customer of the publisher: they can group students into optimal study groups, they can really do deep dives on students, and really understand what’s happening with each student. So that’s a very visible, marketable benefit.
We help publishers and content creators to evaluate content efficacy at the concept level and create more effective learning materials
But for the publishers themselves, we have the ability to provide content efficacy. We help publishers and content creators to evaluate content efficacy at the concept level and create more effective learning materials.
A good example is when we look at the knowledge graph of our partners, which is a map of how concepts relate to other concepts and prerequisites within their product. There may be two or three prerequisites identified in a knowledge graph that a student needs to learn in order to understand a next concept. And when we have hundreds of thousands of students progressing through a course, we begin to understand the efficacy of those said prerequisites, which quite frankly were made by an author or set of authors. In most cases they’re quite good because these authors are actually good in what they do. But in a lot of cases we may find that one of those prerequisites actually is not necessary, and not proven to be useful in achieving true learning or understanding of the current concept that you’re trying to learn. This is interesting information that can be brought back to the publisher as they do revisions, as they actually begin to look at the content as a whole. Is there a return on your investment in producing some of this content? It costs money to build the textbook, and even in a digital world it costs a lot of money. So, if we can provide some content efficacy back to the publisher on how not only that book, but how certain constructs and chapters are working, publishers can be smarter about how they invest in those areas.
We’re beginning to value both the bundled content, which is traditionally textbooks and even online textbooks, but also how to unbundle certain content to be able to provide subscriptions to certain areas. And, if we can do that, then you need to be more efficient with your investment.
So those are a couple of benefits. But I think overall, analytics and understanding what is actually happening with you customer, with students and teachers, is important. You can begin to provide value to schools and help teachers differentiate instruction or understand what concepts a class struggles with, or even help administrators at a more macro level.
Sally:
And the ELT publishers are all talking about how to get closer to their customers through digital content and delivery and analytics – through measurable data in order to improve the learning experience through better content.
better insight changes the role of the publisher
David:
Yeah, I think generally better insight changes the role of the publisher. You’re providing awesome content obviously, you’re providing a great user experience, but now there’s insight into what’s happening, and I think that, we hope, provides a bit of an evolution in the business model. Because it’s not about binding and printing and sales and distribution only.
eltjam:
I think one of the interesting challenges for publishers is actually going to be using those insights. How quickly and how effectively will they be able to take the insights that they can gather from this kind of data, and then put them into practice in the next iteration of the product? I think that’s going to be an interesting thing for publishers to grapple with once the data starts coming in.
David:
Absolutely. In ELT I find it to be a fascinating issue, because it’s so big, but also what we’re learning in some cases is that ELT is very successful in what it does in an offline mode. As the world in education is moving online, how does ELT transition? We’re seeing a lot of different interesting models being tested out around that transition.
There’s a lot of opportunity to provide the same great content or even better, if you can show some efficacy and put the best content for the individual student in front of them. Again that’s another huge mindset change. It’s not just the best book overall – what does that mean? The best book for the average of a million people may not be the best book for me, or Sally, or Molly.
And so it’s not even about the book, it’s about that particular concept or that chapter or that module that you’re working on. I think it’s an incredible opportunity to keep the best of what has been working in ELT and publishing, and education in general, but then leverage these new tools and technology.
And I think the notion of programs lasting 15 weeks or 10 weeks – the time-based kind of paradigm that we’ve been in, or session-based paradigm, gets blown up. What happens then? What you know becomes more important and students can move at their own pace. We’re seeing a shift even in physical universities in the United States and abroad. The University of Southern New Hampshire and other schools are moving to competency-based programs.
It’s just an exciting time to be in education. And with ELT specifically, it’s such a global business that I think they can leverage adaptive learning technology more in some respects than others, because you’re talking about students learning similar things around the world. That’s a lot of data. The learnings that you can gain are tremendous if you’re an ELT provider.
Knewton interviews
Part 1 – Big data and adaptive learning in ELT
Part 2 – Sharing data and competitive advantage
Part 3 – Powering iterative publishing in ELT
Part 4 – Can adaptive learning really work for languages?
A very interesting series of articles here, thank you. I have a few comments.
1. David Liu says above that the world of education is moving online. I think he may be getting a bit ahead of himself there. I’d argue it’s more accurate to say that educational publishers are moving online and are hoping that students and teachers will follow. This isn’t certain though, remember when everyone said that MOOCs were the future of education? And looks what’s happened now.
2. The idea of analyzing data from thousands of students around the world is very, very interesting and could, potentially, give us new insights into how languages are learned much in the same way as how corpora have given us new insights into how language is used.
Does Knewton plan on disseminating this information publicly?
I would imagine , for example, that Knewton would find patterns caused by L1 interference, I.e that Russian L1 speakers would learn certain grammar or vocabulary quicker than Japanese students, because that grammar or vocabulary also exists in their language, and vice versa. But it may well come up with insights that language teachers had never thought of before.
3. I imagine that the kind of materials described would be great for self-study, but just wouldn’t work in the classroom. If each student in the room was doing a different task on their iPad at the same time there would be no classroom dynamic.
More worryingly, the teacher would have no agency over the process. David Liu doesn’t mention teachers much in this interview, where does Knewton see itself in relation to supporting teachers?
@ Thomas Ewens re: your point one.
I think publishers are moving to digital publishing for financial reasons and competitive reasons not because they suddenly became convinced it is the best thing for teachers or students. If that were the case you probably would have gradually seen better (easier to use) digital solutions long ago. It took ages for publishers to move to MP3s vs. CDs or tapes. Many still don’t offer MP3s. They are choosing these solutions to protect their bottom line and I don’t particularly blame them.
I think that they are scared to death of open-source solutions and they see digital publishing as an end-around solution to “paper-based” free and low-cost self publishing.. I think they see digital publishing as a way to reassert their dominance over the educational distribution chain. The reason: for now this kind of publishing, if done well, is high-cost and capital intensive.
I think publishers will soon be in a race to pry people away from the very materials that they thought so highly of 5 years ago.
Michael
In my own personal experience educators and educational institutions are a very conservative bunch. In large parts of the world, I would suggest, schools, universities and teachers are no way ready enough to ‘go digital’ in the foreseeable future. I think that publishers are greatly underestimating the demand for traditional printed coursebooks. A printed coursebook has many advantages over a digital one. It can be kept on the teacher’s room bookshelf and picked up to be used as an extra resource in class, it can be photocopied (whether publishers like it or not, this happens, often). It easily be borrowed for short periods of time. It can be re-used by students who buy it off their friend who has just finished the same course they are starting (I’ve seen this happen too). It is cheap and you don’t need any hardware to use it.
I can also imagine that many educational institutions just won’t be able to see the long-term benefits in investing in the hardware needed to ‘go digital’ and will continue to go for the cheap, printed coursebook option.
I can understand why publishers would want to ‘cover their backsides’ by having a digital portfolio just in case ed-tech really takes off. But reducing their print portfolios to just 20%? I think that’s very risky.
Thomas,
Thanks for expanding on that Thomas.
Lots of interesting stuff again. As an ELT writer, I do like the idea of getting feedback direct from students and teachers, rather than channeled through sales reps or school owners, etc.
I’m a bit concerned about the constant use of the word ‘efficacy’ though because I think it’s very difficult to measure what’s effective in terms of language teaching and learning. There are some things that are relatively easy to teach and you know that students will pick up quite quickly, such as some vocab sets (which will show up as effective when tested). But other concepts, like more complex grammar, are slow burners, you don’t expect students to grasp them straight away. Does that mean that we just end up pushing out more of the ‘easy’ stuff in order to get better ‘results’?
Good points Ms. Moore. You point out a real problem- which is things never get learned once and then remembered forever. Some things take a very long time to be well understood. But, this is an reasonable argument for machine learning as far as I am concerned. A machine that can independently and tirelessly test students over a long period of time on what they have already learned (ahem…studied) to make sure it sticks.
But will all this efficiency turn learning into a mechanical set of problem? Will a machine ever learn to tell a student to ignore (for the present) a certain grammar pattern in favor of something that is easier? One day perhaps….
I don’t see how any large data sets can be valuable unless the teaching process is completely standardized. If teacher 1 spends 10 minutes on task A in a class of 20 and teacher B spends 20 minutes on the same task in a class of 6 how can the results be productively compared. They can only be compared after you standardize the time spent teaching, what is introduced in the class, how it is introduced, how it is practiced and indeed at which point the assessment test is given.
I am sure the publishers realize this. Given the same book and two different teachers you most likely will produce two different outcomes and many of the outcomes may not be apparent on your test.
It seems to me that this model leads to one outcome– a more rigorous control of the learning process. It probably leads to some kind of machine based instruction and testing in the end. I’m not sure I entirely mind this. What I do mind is that the people who are advocating this system say that big data is in the teacher’s best interest. They have a long way to go to convince me of this. The need to build some real protections in that insure that publisher, student AND TEACHER benefit from this system of open sharing.
@ Thomas Ewens re: your point one.
I think publishers are moving to digital publishing for financial reasons and competitive reasons not because they suddenly became convinced it is the best thing for teachers or students. If that were the case you probably would have gradually seen better (easier to use) digital solutions long ago. It took ages for publishers to move to MP3s vs. CDs or tapes. Many still don’t offer MP3s. They are choosing these solutions to protect their bottom line and I don’t particularly blame them.
I think that they are scared to death of open-source solutions and they see digital publishing as an end-around solution to “paper-based” free and low-cost self publishing.. I think they see digital publishing as a way to reassert their dominance over the educational distribution chain. The reason: for now this kind of publishing, if done well, is high-cost and capital intensive.
I think publishers will soon be in a race to pry people away from the very materials that they thought so highly of 5 years ago.