In our recent series of interviews with adaptive learning powerhouse, Knewton, a number of interesting points were raised by our readers, which we thought were worth exploring further. So, we compiled some of the more challenging questions and David Liu, Knewton’s Chief Operating Officer, sent us his responses.
Is adaptive learning actually desirable?
Gavin Dudeney commented that he is unconvinced by adaptive learning:
“CAI is now back as “adaptive learning systems.” Some of the old programs have been repurposed with more interactivity. McRae states it as “adaptive learning systems still promote the notion of the isolated individual, in front of a technology platform, being delivered concrete and sequential content for mastery. However, the re-branding is that of personalization (individual), flexible and customized (technology platform) delivering 21st century competencies (content).” “
http://barbarabray.net/2013/12/30/this-time-its-personal-and-dangerous/
David’s response:
Classrooms are filled with students that learn at different rates; come from different educational backgrounds; and have different needs, attention spans, and interests.
Data analysis in education enables us to learn more about which content works best for which students. We can provide continuously updated predictive analytics that empower teachers and help them differentiate instruction — in and out of class — for individual learners. Educators can determine when an individual student is off-track, and identify gaps in knowledge at a granular level.
By helping identify particular concepts students have trouble understanding, we can provide more targeted interventions. Adaptive learning technology empowers teachers with information so they can offer hints to help students persevere through challenges.
While leveraging adaptive learning technology for language learning can be challenging, it is one of the areas of education most exciting for the promise of adaptivity. In ELT, students come from very different levels of proficiency and exposure, making adaptive learning particularly valuable.
Can language learning really be quantified and tracked?
Julie Moore wondered just how quantifiable and trackable language learning can (and should) be. Her experience of students who have been through education systems focussed on box-ticking and testing is that they develop no real feeling for the language and have poor communication skills. Could too much emphasis on quantifiable elements leave no room for the more ‘touchy feely’ elements of learning?
David’s response:
Yes. Language learning is certainly tricky because it is a multi-dimensional domain but for any lesson where there is a rubric we can leverage adaptive learning technology to analyze student progress. There is a misconception that adaptive learning technology only works in subjects like math. Knewton can power courses in a variety of subjects — including ELT, reading, maths, chemistry, biology, physics, anatomy and physiology, accounting, and finance. In general, in order to benefit from Knewton, learning experiences must be at least partially online and there must be generally agreed upon “correct” and “incorrect” answers.
Today, there are many organizations — from universities to MOOCs to hardware manufacturers — working to increase distribution of digital learning materials to students around the globe. Adaptive learning technology allows publishers (including ELT publishers) to personalize these digital materials for students. This allows students and instructors to better understand what they know, what they should work on next, and how they learn best.
Over the next ten years, many more ELT courses will be online and blended. We’re already seeing the shift from brick-and-mortar language classes to online options, including incorporating the matching of live real people to communicate and learn from each other. Many innovative language learning products are blended, either combining independent computer-based learning with a traditional classroom and teacher, or with online social media. Ultimately, as long as what the student is learning can be organized around learning goals, and these learning goals build upon each other over time, then adaptive learning technology can help personalize learning materials.
Can we truly measure ‘efficacy’ in language learning?
Julie Moore commented that “it’s very difficult to measure what’s effective in terms of language teaching and learning. There are some things that are relatively easy to teach and you know that students will pick up quite quickly, such as some vocab sets (which will show up as effective when tested). But other concepts, like more complex grammar, are slow burners, you don’t expect students to grasp them straight away. Does that mean that we just end up pushing out more of the ‘easy’ stuff in order to get better ‘results’?”
Michael Butler added that ‘machine learning’ has a place in that it can “ independently and tirelessly test students over a long period of time on what they have already learned … to make sure it sticks”, but asks whether there’s a risk of turning learning into a set of mechanical problems.
David’s response:
We don’t expect adaptive learning tools to replace teachers or humans in the language learning process. That’s never been the goal. We understand the importance of interaction and conversation with a teacher or other learners.
However, there are many online tools to help the language learning process and analyze student progress, to help instructors and students understand what a student knows and what they need to work on. A Knewton-powered ELT course, for example, could pinpoint which types of grammatical structures students need to study, and provide targeted follow-up exercises to address those goals.
Achieving fluency in a given language is a subjective goal. As such it is difficult, and not necessarily productive, to quantify the efficacy of every effort toward this goal. However, there are smaller targets for which efficacy can be measured objectively — a student’s vocabulary in a certain subject area, understanding of the past tense, use of pronouns, the ability to converse about certain topics in certain situations, the ability to recognize the appropriateness of a greeting, the ability to distinguish between two close-sounding words, and more. As a student progresses through a digital ELT course or through supplemental online ELT materials, a proficiency-based adaptive learning system can estimate what that student knows and what they should work on next to best meet these discrete learning goals.
How can adaptive language learning work in a classroom situation?
Thomas Ewens felt that types of content described would be great for self-study, but wouldn’t work in the classroom, since the students’ work would not be in sync.
Michael Butler agreed that Knewton’s approach can’t be valuable unless the teaching process is standardized: “If teacher 1 spends 10 minutes on task A in a class of 20 and teacher B spends 20 minutes on the same task in a class of 6 how can the results be productively compared? They can only be compared after you standardize the time spent teaching, what is introduced in the class, how it is introduced, how it is practiced and indeed at which point the assessment test is given.”
David’s response:
Teachers for decades have understood the importance of “differentiated instruction” — the process of tailoring instruction to meet individual learners’ needs. Knewton helps educators evaluate what course content resonates well, with which students. We can also help educators quickly identify difficult concepts so that they can tailor lessons accordingly, or spend more time focusing on concepts that the majority of the class is struggling with.
A Knewton-powered course gives teachers a real-time snapshot of student achievement and concept-level proficiency. Instructors don’t have to wait until they grade the next test to identify weak spots. When a student struggles with a concept in a Knewton-powered course, he or she is guided to work on prerequisite skills. These skills are prioritized based on the strength of their relationship to the topic at hand, the student’s strengths and weaknesses, and learning goals set by the instructor (ex. preparation for an upcoming quiz). This frees teachers up from needing to pull together personalized prerequisite materials for each student. Teachers, armed with the knowledge of what might be causing frustration, have more time to sit down with individual students, spark group discussions, and orchestrate classroom activities.
What does adaptive learning mean for the role of the language teacher?
Thomas Ewans commented: “David Liu doesn’t mention teachers much in this interview, where does Knewton see itself in relation to supporting teachers?”
Michael Butler echoed the same concern, commenting that adaptive learning would be presented to teachers as ‘progress’, when it fact it would eventually result in teachers losing their jobs, even if “it probably will result in better learning in many instances”. He added that “the end result will probably be systems that benefit big business at the expense of teachers and small schools. In the end this is a system that will tend to concentrate wealth – distributing it from the many to the few in the name of efficiency.”
Michel also raised the question of how to ensure that publisher, student and teacher benefit from the data-driven approach.
David’s response:
Knewton doesn’t work without teachers. Especially in language learning, there are technical nuances and cultural information that teachers must convey. Knewton aims to empower educators and give them more time to do what they do best — teach and inspire students. The data teachers receive from an adaptive learning system improves their ability to help students break down complex ideas, encourage critical thinking, point out relevant examples that will resonate with that individual student, and more.
Knewton provides analytics and tools to help educators tailor their lessons or differentiate instruction depending on what the class or a particular student needs. This helps teachers monitor performance and reduce administrative work. In a flipped classroom setting, language students could work on personalized lessons at home and then spend in-class time practicing the language with the teacher, going over the most difficult concepts from the night before, or engaging in any other activities the teacher feels will help a student succeed.
How does Knewton support low-stakes, non-assessed learning?
Paul Ricketts commented that “learners – and their teachers, the publishers’ direct clients, may have many different types of desired outcomes, many of which are outside Knewton’s traditional comfort zone of high stakes learning and clear goalposts like the CEF. What is clear, though, is that instructional design should benefit enormously from intelligent, nuanced use of Knewton’s analytical tools.”
David’s response:
New education technology is emerging that enables teachers, students, parents, and administrators to accurately and continuously gauge how well a student is doing at any given moment. Soon, it will be possible to measure students’ competency in a given subject — demonstrating exactly what they know, how quickly they learned it, and how well they retained it.
Today we can measure more than just answers inputted from an assessment. We can measure a student’s engagement with lessons, understand a student’s ability to persist through challenging material, and more. These types of analytics support the student but also help education content authors create better courses and content.
This technology will reduce the need for high-stakes standardized tests to try to estimate students’ mastery of a curriculum. Teachers and students will be able to spend less time stressing about test prep and more time exploring, problem-solving, creating, and developing a more engaging learning experience.
Is Knewton open to sharing the insights into language that its technology could potentially reveal?
Thomas Ewens commented: “The idea of analyzing data from thousands of students around the world is very, very interesting and could, potentially, give us new insights into how languages are learned much in the same way as how corpora have given us new insights into how language is used.
Does Knewton plan on disseminating this information publicly?
I would imagine, for example, that Knewton would find patterns caused by L1 interference, i.e that Russian L1 speakers would learn certain grammar or vocabulary quicker than Japanese students, because that grammar or vocabulary also exists in their language, and vice versa. But it may well come up with insights that language teachers had never thought of before.”
David’s response:
We haven’t announced any specific plans, but we’re looking forward to uncovering and sharing insights over time. In addition to insights into language learning, Knewton can also uncover information about publishers’ content, which can help inform the creation of more effective ELT learning materials. With this information, publishers can create better courses, which further benefits students. Our mission is to help more students reach their full potential. To the extent that any generalized discoveries about student learning and ELT content can help in this regard, we’d be happy to share our findings.
Is ELT really ready for this?
Thomas Ewens pointed to the conservatism of educational institutions, and the fact that print is still far from dead. Are publishers taking a big risk in investing heavily in Knewton-powered digital courses when the bulk of the market still favours printed textbooks?
David’s response:
Yes. Adaptive learning requires that learning materials and assessments be at least partially digital. Until relatively recently the ELT industry was dominated by print courses, with add-on print and digital components. Now more and more courses are moving online, either fully or in blended formats. Publishers around the world are already experimenting with adaptivity to meet the needs of their tremendously varied and increasingly savvy audience, an audience that wants more personalized learning experiences.
This shift to digital, personalized learning materials is pushing ELT publishers to evaluate technical aspects of their product and understand how to deliver more targeted experiences. It also involves careful consideration of the pedagogy of a course, how it can be expressed in an adaptive product, and how that product fits in with the complete student experience. Publishers must consider how each product functions and why. For example, is the product only online or does it have a classroom component? What can be taught online and what should take place in the classroom?
We’re seeing rapidly increasing interest from top publishers around the globe, across many subject areas and grade levels, that want to leverage Knewton’s infrastructure platform to make adaptive learning products. Macmillan is creating next-generation ELT products, using Knewton technology to provide personalized recommendations and analytics. In addition, Cambridge University Press, among the world’s foremost ELT publishers, will use Knewton adaptive learning technology to power a new generation of digital ELT products.
Personalization is most effective via one large platform that analyzes vast amounts of data from classrooms around the globe. Building a self-contained adaptive learning app is expensive and unscalable. We’ve built a strong team of data scientists and learning experts that handle personalization so publishers can focus on creating quality content. Every application Knewton powers brings its own core competencies while outsourcing the heavy machinery of its personalization infrastructure to Knewton.
It’s beneficial for publishers to begin early, both to meet customers’ demands for personalized materials as well also to begin measuring how their content is doing — so they can make informed decisions about where best to focus their resources. Knewton is proud to power these next-generation adaptive ELT products.
David’s response to the question about efficacy is very revealing. He acknowledges that not every learning activity can or should be quantified, but adds that ‘there are smaller targets for which efficacy can be measured objectively — a student’s vocabulary in a certain subject area, understanding of the past tense, use of pronouns, the ability to converse about certain topics in certain situations, the ability to recognize the appropriateness of a greeting, the ability to distinguish between two close-sounding words, and more.’
Of the examples he gives, ‘the ability to converse about certain topics in certain situations’ is extremely vague. Which topics in which situations and at which level? Of the rest, we have grammar at the mcnugget level, vocabulary (presumably single item lexis), segmental phonology (phonemes), and appropriacy of functional language (i.e. a combination of lexis and intonation – although it is unlikely that the software will be any good at recognising intonation patterns). In other words, we are talking about language knowledge, not language use.
It is both possible and likely that data analysis could help in the delivery of this part of the English language syllabus. And it makes a lot of sense for the presentation of language knowledge to be done outside the classroom (whether with books of software) in order to free up classroom time for opportunities for language use. But David’s description of what would take place in the classroom (practicing the language with the teacher, going over the most difficult concepts from the night before, or engaging in any other activities the teacher feels will help a student succeed) is both airily vague and shockingly ignorant of what language learning entails. He confuses concepts and skills and seems to assume that learning a language is centrally concerned with ‘knowing’, as opposed to ‘doing’ something. He seems unaware of the learning / acquisition distinction, and unaware that the main problem in language acquisition is moving from language knowledge (which Knewton’s software might help with) to appropriate language use.
Adaptive software might improve ‘English Grammar in Use’ style study, but learners don’t need more of this, or a better version of this – they need less of it. As a profession, we have spent years and years trying to move away from an obsession with teaching individual bits of language and towards a more communicatively oriented approach.
It would seem that Knewton is desperately in need of someone who knows something about language acquisition. Spin about helping students reach their full potential, inspiring students or encouraging critical thinking cannot mask a fundamental lack of understanding about language curricula and language learning processes.
Maybe I’m not a very good teacher…(!) but “practicing the language with the teacher, going over the most difficult concepts from the night before, or engaging in any other activities the teacher feels will help a student succeed” is exactly how I do most of my classes with private students where I can tailor the course more to their needs. “Any other activities” like film, TV, stories, creative writing, debates, games etc. I don’t do grammar exercises, vocab tests etc and that’s why I retain private students for years – we do the other, fun stuff! Although this is it, isn’t it? “for years” mostly with no visible progress from the students because only the ones who self study manage to progress. Why exactly do we favour a model that seems to yield mostly plateauing B1s and quitting A2’s?
To me the really big question here is about the absence of one-to-one cause/effect relationships in successful language learning, which is what, as I understand it, adaptive learning needs to be able to analyse. We already have good tools for analysing discrete aspects of language mechanics, such as whether a learner knows certain vocabulary items or specific tenses, relevant mostly only at lower levels. But language performance is about so much more than this – just because a user makes a simple grammatical error doesn’t necessarily impede understanding, and we wouldn’t want to assess a piece of writing or a longer conversational exchange just on the basis of the things that can easily be analysed (that would be like evaluating an essay-type answer solely on the basis of the grammar and vocabulary used). Nor do we want to assess reading ability just on the basis of having understood a concept – we might ask a learner to outline and describe the process they’ve just heard or read about, but we would be interested in their use of the language to do this and their overall comprehension – we wouldn’t want to test their grasp of the concept before they could move on. Learning concepts isn’t what language learning is about. I’d love to see some more holistic analysis able to reflect the simultaneously required multiple aspects of language use – I suspect the answer will be in the big data area, similar to the ways machine marking of extensive written compositions now takes place.
“Education is not the transmission of information or ideas. Education is the training needed to make use of information and ideas”. (Hieronymi, P. (2012) Don’t Confuse Technology With College Teaching, The Chronicle of Higher Education, Aug 13, 2012).
If I were David’s teacher I would make him write this out 100 times. 😉
Firstly, thank you to Laurie and David for taking the time to respond to my comments and those of others.
“In a flipped classroom setting, language students could work on personalized lessons at home and then spend in-class time practicing the language with the teacher, going over the most difficult concepts from the night before, or engaging in any other activities the teacher feels will help a student succeed.”
This seems like a very reasonable idea, to me. As a practising teacher, I can certainly go along with this.
Scott Thornbury and others, dating back to Postman in the 1980s, have written extensively of their distrust of much of the hyperbole surrounding Ed tech, and I think that their distrust is often well- founded. But, if technology companies talk more about contributing to education and ‘helping’ teachers, as David Liu does above, rather than ‘disrupting’, they appear a lot more credible.
Leaving aside questions about the views of language and language learning which underpin adaptive learning products, surely it’s how teachers and learners use them which really counts?
On further reflection, there, of course, is a role in language learning for grammar and vocabulary self-study, declarative knowledge, etc etc. If Knewton want to corner the market for that then ELT professionals, materials writers and teachers don’t really have a right to complain.
What we do have a right to complain about is any attempt to disintermediate English language teachers from the process. Therefore, I think that David Liu’s willingness to engage, and to outline where he thinks his products can help teachers and where he thinks teachers should just be left to do their jobs, is very positive.
I’m not sure I’m complaining, but I certainly have a right, perhaps even a duty, to criticise the marketing (and, let’s be clear, David’s piece here is a marketing exercise) of a product which is a distraction from more important issues. Adaptive learning is intimately connected with attempts to disintermediate teachers. Check out the links between Knewton and the Foundation for Excellence in Education, and then check out what the Foundation for Excellence in Education has to say about teachers’ tenure, salaries and so on.
I’m not sure I’m complaining, but I certainly have a right, perhaps even a duty, to criticise the marketing (and, let’s be clear, David’s piece here is a marketing exercise) of a product which is a distraction from more important issues. Adaptive learning is intimately connected with attempts to disintermediate teachers. Check out the links between Knewton and the Foundation for Excellence in Education, and then check out what the Foundation for Excellence in Education has to say about teachers’ tenure, salaries and so on.
It is too bad that this back and forth never rose to the level of a real conversation. Obviously we are far from understanding what David is trying to do, but it seems to me that David really never addressed our concerns head-on either. His two direct Yes answers were vague at best and for Heaven’s sake, where were the No answers. Don’t spare our feelings David. If you disagree with us let us know so. Language teachers are a disagreeable lot and we don’t mind being told we are “wrong” or off-track” or “barking up the wrong tree”. I think this was a monumental waste in terms of mutual communication; instead it was more like the skirmish before a battle with each side preparing its ammunition.
I am left with the feeling that you would turn language learning into a mechanistic experience. You make it sound like language learning is entirely an additive experience whereas I see it as a transformational experience. How can you expect us to believe that you are going to care about our interests when you stick to the language of incremental learning while we all (many of us?) believe that language learning cannot be explained solely as a slow accumulation of stuff? Is there anyone on your staff willing to accept that and engage us mid-way? Because if there isn’t there should be. You should have someone on staff that argues against seeing language learning as the mere accumulation of stuff. That person would be the natural person to communicate with teachers as we believe we do more than simply help students accumulate facts one on top of another. We believe in transformation, we believe in emotion, we believe in desire, we believe in leadership. Can you imagine what kind of a partnership we could have if you believed in these things as well?
As a start David, now is a good opportunity to declare exactly what you are willing to share with the community. It would be a wonderful gesture of goodwill on your part.
It is too bad that this back and forth never rose to the level of a real conversation. Obviously we are far from understanding what David is trying to do, but it seems to me that David really never addressed our concerns head-on either. His two direct Yes answers were vague at best and for Heaven’s sake, where were the No answers. Don’t spare our feelings David. If you disagree with us let us know so. Language teachers are a disagreeable lot and we don’t mind being told we are “wrong” or off-track” or “barking up the wrong tree”. I think this was a monumental waste in terms of mutual communication; instead it was more like the skirmish before a battle with each side preparing its ammunition.
I am left with the feeling that you would turn language learning into a mechanistic experience. You make it sound like language learning is entirely an additive experience whereas I see it as a transformational experience. How can you expect us to believe that you are going to care about our interests when you stick to the language of incremental learning while we all (many of us?) believe that language learning cannot be explained solely as a slow accumulation of stuff? Is there anyone on your staff willing to accept that and engage us mid-way? Because if there isn’t there should be. You should have someone on staff that argues against seeing language learning as the mere accumulation of stuff. That person would be the natural person to communicate with teachers as we believe we do more than simply help students accumulate facts one on top of another. We believe in transformation, we believe in emotion, we believe in desire, we believe in leadership. Can you imagine what kind of a partnership we could have if you believed in these things as well?
As a start David, now is a good opportunity to declare exactly what you are willing to share with the community. It would be a wonderful gesture of goodwill on your part.
Maybe I’m not a very good teacher…(!) but “practicing the language with the teacher, going over the most difficult concepts from the night before, or engaging in any other activities the teacher feels will help a student succeed” is exactly how I do most of my classes with private students where I can tailor the course more to their needs. “Any other activities” like film, TV, stories, creative writing, debates, games etc. I don’t do grammar exercises, vocab tests etc and that’s why I retain private students for years – we do the other, fun stuff! Although this is it, isn’t it? “for years” mostly with no visible progress from the students because only the ones who self study manage to progress. Why exactly do we favour a model that seems to yield mostly plateauing B1s and quitting A2’s?
I think that what you do, successfully, with small groups of private students is not really comparable to more typical situations with groups of a minimum of 15+. Knewton’s product, in partnership with big ELT publishers, is targeted at large institutional adoptions, not the world of private students (for whom a product like duolingo or babble may be more affordable and more appropriate). The key selling point of Knewton’s adaptive software is that it promises differentiated, personalized learning paths. In their marketing, they made a lot out of the need to move away from a ‘one-size-fits-all’ approach.
But, in a group of 15+. students will move at very different rates through different material – or, at least, they will if the software is personalized and differentiated in any meaningful sense. In this context, I fail to see how a teacher could easily spend class time ‘going over the most difficult concepts from the night before’ since everybody will have been doing something different. And, if they haven’t been doing something different. where’s the personalized, differentiated learning path?
Out of curiosity, I submitted the 5 Knewton interviews on @eltjam to a Keyword search (using Compleat Lexical Tutor). These are the 30 words that occur significantly frequently in ‘Newtonspeak’:
(1) 8155.00 rubric
(2) 5301.00 efficacy
(3) 2446.50 resonate
(4) 2038.88 requisite
(5) 1631.00 iterate
(6) 1191.92 personalise
(7) 1030.16 online
(8) 978.64 digital
(9) 679.58 proficient
(10) 652.40 comprehend
(11) 543.67 optimise
(12) 407.75 domain
(13) 313.65 globe
(14) 310.67 textbook
(15) 295.56 adapt
(16) 242.54 content
(17) 226.24 student
(18) 203.88 blend
(19) 184.72 publish
(20) 171.68 lever
(21) 140.60 infrastructure
(22) 127.42 insight
(23) 122.33 empower
(24) 116.24 data
(25) 115.13 author
(26) 104.11 platform
(27) 101.94 administer
(28) 97.86 maximise
(29) 94.10 tailor
(30) 89.99 outcome
Interesting. I never would have thought to do this. Does this mean that efficacy appears about 45 times for every once empower does?
No, Michael, it means that – compared to a reference corpus (here a 10 million word sample from the British National Corpus) – the occurrences of ‘efficacy’ are unusually high – especially given the fact that ‘efficacy’ is (unlike ’empower’) a relatively rare word). Here’s how the Lexical Tutor site explains it:
The number accompanying each word below represents the number of times more frequent that word is in your text than in bnc_speech_fams_per10mill, proportionally. For example, the first item in the output 8155.00 rubric means that rubric has 1 natural occurrences in 10000000 words, but 5 occurrences in your 6131-word text, which would work out to (5/6131) x 10000000 = 8155 occurrences if your text were the same size as the corpus. The word is thus 8155 / 1 = 8155.00 times more frequent in your text than it is in the reference corpus. This probably means that the word plays quite an important (or ‘key’) role in your text.
Hi Scott,
I think it’s important to bear in mind that compilation of the BNC stopped in 1993, so you’re lacking 20 years of instances of ‘efficacy’, a word which, I gather from a brief explore, has (re-)gained a lot of currency in recent years. It would have been better to use a more up-to-date and comparable reference corpus such as, e.g., UKWac. UKWac has 6.7 instances of efficacy per million, beating the Knewton interviews.
Having said that, I was pleased to see someone applying some linguistic analysis to these interviews. Very interesting!
Incidentally, back in December, I created a corpus of all eltjam posts and comments from its launch to late December 2013. My 130,000 word ‘Jam Corpus’, which includes only two of the Knewton interviews, already showed then that, with a whopping 54 mentions, Knewton was by far the most frequent proper noun on the eltjam site. Far ahead of the next contender (any guesses?), with 39 mentions. I wonder what the frequency would be now?
What was it that Oscar Wilde said about being talked about?
Very interesting to see this analysis – although it’s meaningless without taking context into account. All we learn is that someone writing on the topic of measuring students’ progress against defined objectives is much more likely to use words like ‘efficacy’ or ‘rubric’ than a cross-section of people talking about every imaginable subject. What would be really interesting would be to see this adjusted for context. How does David’s interview compare with other people talking on the same topic? That might show us whether such a thing as ‘Knewton speak’ actually exists.
And Diane, can we see the full results of the ‘Jam Corpus’? Amazed and fascinated that you’ve done such a thing! My guess for word #2 is ‘edtech’…
Thanks Scott!
Hi Scott,
I think it’s important to bear in mind that compilation of the BNC stopped in 1993, so you’re lacking 20 years of instances of ‘efficacy’, a word which, I gather from a brief explore, has (re-)gained a lot of currency in recent years. It would have been better to use a more up-to-date and comparable reference corpus such as, e.g., UKWac. UKWac has 6.7 instances of efficacy per million, beating the Knewton interviews.
Having said that, I was pleased to see someone applying some linguistic analysis to these interviews. Very interesting!
Incidentally, back in December, I created a corpus of all eltjam posts and comments from its launch to late December 2013. My 130,000 word ‘Jam Corpus’, which includes only two of the Knewton interviews, already showed then that, with a whopping 54 mentions, Knewton was by far the most frequent proper noun on the eltjam site. Far ahead of the next contender (any guesses?), with 39 mentions. I wonder what the frequency would be now?
What was it that Oscar Wilde said about being talked about?
Regarding grammar mcnuggets, I came across these ‘grammar snacks’ on the British Council website.
http://learnenglishteens.britishcouncil.org/grammar-vocabulary/grammar-videos
Regarding efficacy, as I understand it Scott’s analysis of the interviews showed an equivalent frequency of 5301 per 10 million, or 530 per million, easily beating your UKWak corpus, Diane.
Reading that again, I’ve got it a bit wrong. But even if there’s 1 mention of efficacy in a 6000 word text, that’s easily more that 6.7 per million. It will add up to around 160 per million.
I
Thanks, Thomas, you’re right. My maths was way off. The main focus of my comment was that the BNC wasn’t the best choice for a reference corpus, being so out of date.
Keyword lists are great and Scott’s is particularly useful here. But I think it’s best not to focus on the figures too much. After all, they assume that if Liu had continued these interviews beyond 6000 words until he reached 10 million words (!!) the frequency of his use of ‘efficacy’ would have remained the same. Comparing with a much smaller corpus from a comparable domain – say, business or technology, would, I think, have yielded less dramatic figures. Comparing with a more similar-sized corpus of teachers discussing teaching would be very interesting.
Laurie, we overlapped, and it looks like you’ve said what I was trying to say much more succinctly. I seem to have taken things a bit off-topic. Sorry for that.
Jam Corpus. I plan to keep building this as it’s still very small (130,000 words) in corpus terms and I want to compile an appropriate reference corpus too. I built it because I’m fascinated by what we talk about and the language we use when we talk about edtech – especially educators. I’m happy to discuss more but I don’t want to take this off-topic any more than I have. You’re wrong about #2 btw – but because I don’t consider edtech a proper noun. Think companies and their products 😉
Ah yes, I didn’t register that it was only proper nouns – doh! OK, I’m now going to waste a chunk of my morning skimming through our posts to try and work this one out!
Pearson?
I’ve taken an ‘adaptive learning’ course online partly for the experience. I liked it and it worked well for me. Stayed motivated, challenged and enjoyed seeing the little graph fill up as I solved harder and harder problems more and more quickly. There’s a bit of that sort of learning in language learning. I would say the adaptive course was an improved version of a workbook or self-study program. That description in itself probably a reasonable way to sell this sort of thing to ELT professionals. Plenty of things that it can’t do that a good classroom teacher can (just like a workbook or exercise book).
Knewton should hire someone who knows how to talk with ELT teachers, though. Painful to read David’s words.
I’ll try it for him: ‘English teachers, here’s something that’s a little better than the old exercise book for homework. You might even look at the data it collects and judge whether it adds anything to what you’ve learned about your students from your classroom interaction.’ Very like an improved version of a workbook, which by the way never threatened to take a teacher’s job (nor was ever marketed to us teachers with such a lot pretentious nonsense talk).