IMG_1668As the sun set over Hackney  on Friday, the week was winding down as usual for the ELTjam London team. In-boxes were being zeroed; our weekly newsletter was being compiled; tasks were being moved to ‘done’ on the company Kanban board.

The team often ends the week with a bottle of craft ale or two, but the box of 24 beers from Hackney Wick’s Crate Brewery and a 3-litre box of wine suggested that this was to be a different kind of Friday night. At 6pm, we turned off our computers and gathered around the table for the start of our first ever hackathon. It would be the start of a process that, in less than 24 hours, would lead us to create Amé, an ELT ‘bot’.

Meet Amé

Screen Shot 2016-03-22 at 10.21.08

Amé is the user-facing persona of Ask Me English, a free service we’ve built that lets Spanish-speaking people ask any question they like about learning or using English via Whatsapp. They ask their question, and Amé responds; it’s as simple as that.

I first had the idea for Ask Me English over a year ago, and the purpose of the hackathon was to see if it actually had legs. For those who are unaware, during a hackathon a team comes together under extremely tight time and resource constraints to see if they can build a shippable product. For the purposes of the Ask Me English hackathon, we set a time constraint of 48 hours and a stipulation that the only cost to ELTjam should be food and drink. There was to be no budget for design and engineering. This was to be a zero-cost MVP (minimum viable product) – a way of testing a set of hypotheses around a product or business idea as quickly and cheaply as possible.

Three things had already factored into our thinking around Ask Me English. Firstly, people seem to love asking questions on the internet. From Reddit’s AMA, to Quora, to the developers’ treasuretrove Stack Overflow, everywhere you looked were sites designed to facilitate the answering and asking of questions. What these sites do is go beyond search, providing not just information but insight and a deeper level of understanding.

Secondly, there is the growing trend towards ‘chat as app’ – that is, apps that use the chat interface as a way of doing something more than just messaging. There’s Slack, the messaging platform that’s become the central information hub of thousands of companies around the world (whilst at the same time becoming the fastest growing piece of business software of all time). There’s Quartz, an app that delivers news stories via an interactive chat interface. And there’s Text Rex – my favourite and one of the key inspirations behind Ask Me English – a service for New Yorkers that allows them to text a phone number and get personalised, real-time restaurant recommendations.

Finally, there’s the rise of the Bots. Bots and Artificial Intelligence (AI) are everywhere right now. From Apple’s Siri to Microsoft’s Cortana to Facebook’s forthcoming ‘M’, the trend is towards combining complex natural language processing and machine learning to create human-like personal assistants that live in your phone.

Three interesting trends, and one potentially interesting product idea that drew from all of them.

With the Ask Me English MVP, we set out to test the following hypotheses in one weekend:

  1. Do non-native speakers want to ask questions about learning and using English?
  2. If so, do they want to ask them via text message?
  3. What do they want to ask?
  4. Can we answer?
  5. Can we answer at scale?
  6. Can we turn this service into a viable business?

In less than 48 hours, we managed to answer questions 1–5. Here’s what happened …

Friday night

IMG_1672The hackathon was to be run as an accelerated version of our nascent LXD (Learner Experience Design) framework. We’d aim to identify a problem; figure out a useful, usable and delightful solution; and build it. The 48-hour time constraint meant that we’d need to conflate those three stages into one: actually build something to see if it validated a solution to a problem we thought existed.

We began with a Lean Canvas, a way of quickly capturing some high-level, essential information about your product or business idea. We started by thinking about what market segments we wanted to target with Ask Me English and quickly settled on Spanish-speaking, non-native English speakers. This made sense for a number of reasons. The Spanish-speaking world is a huge ELT market and the ELTjam team present that night luckily comprised Spanish speakers from B1 to native-speaker level. We then started to analyse what problems our target market might have. We soon realised that we had to segment again, choosing between Spanish-speakers who live in English-speaking environments and those who don’t. As we were in London, we opted for the former, figuring we’d be more likely to be able to get feedback from actual users.

We began analysing the kinds of problems that a Spanish person in an English-speaking country might encounter. We drew on our own experiences of traveling and living abroad as well as the experiences of non-English-speaking friends that live in the UK. For example, Berta, our Marketing and Events Manager, commented that she had dislocated her knee in Morocco once and had struggled to explain to the doctor that this had happened before and that she knew exactly what the problem was. On a much more mundane note, I commented that I’d recently had to go to a barber’s shop in Spain and had found myself completely incapable of explaining in Spanish how I wanted my hair cut, despite being a solid C1-level Spanish speaker. Other use cases included succeeding at online dating, deciphering complicated street signs, and dealing with plumbing emergencies.

We soon hit upon an interesting insight: what most language teaching does is prepare learners for ‘just in case’. We’re told to learn a thing, and then when the time comes that you need the thing, you hope you can recall the thing (for example, checking into a hotel or ordering a meal). But what happens if either a) you don’t remember how to say the thing or b) you find yourself in a situation that you’ve not been prepared for (like, for example, a dislocated rotula)? In those situations, the ‘just in case’ language you’ve been taught doesn’t help you. What you need is ‘just in time’ language.

When people find themselves dealing with unexpected situations in a foreign language, they have the following options:

  1. Know how to handle it by recalling ‘just in case’ language.
  2. Happen to be with someone who can help you.
  3. Ask someone who happens to be there for help.
  4. Use a tool such as Google translate
  5. Muddle through as best you can.
  6. Disengage from the situation entirely.

We realised that if you could send an SMS or Whatsapp to a number and ask them any language-related question, and get an almost instant answer, then you could potentially open up options 2 and 3 to anybody, at any time. And if you could make the answers rich in information, or funny, or engaging on some other level, you might even usurp tools like Google translate. So that’s what we set out to build the next day: a way of putting an on-call, engaging, English-speaking friend into everybody’s back pocket.

Saturday morning and afternoon

image2We were determined that we should have some kind of version of Ask Me English live by early afternoon Saturday, and the plan was this:

  1. Get a SIM card and an unlocked phone.
  2. Create a Whatsapp account.
  3. Build a one-page website explaining what Ask Me English did and giving out the phone number.
  4. Post about the website in some relevant Facebook groups.
  5. See if anyone Whatsapped us.
  6. Answer the messages ourselves manually.

At this point, success looked like getting all of the above done and getting one incoming Whatsapp from one genuine user.

image1The wording on the site was simple. Here it is in English before we translated it into Spanish:

GOT A QUESTION ABOUT ENGLISH?
MY NAME IS AMÉ; TEXT ME AND I’LL HELP.
Whatever you need to do in English, or whatever questions you have, send me a message and I’ll get right back to you with the information you need.
+44 (0)7516 XXXXXX

Alongside the copy, we showed screenshots of example conversations with Amé. You can see them here translated into English.

The haircut example was designed to show how Amé could be useful in tricky, language-related situations; the screwdriver example was to show how it could surpass Google Translate in usefulness.

Incidentally, the name Amé was our Head of Product Jo’s idea; if you’ve not worked it out already, it stands for Ask Me English. It was a small thing, giving Ask Me English ‘a character’, but it turned out to be one of the best calls we made all weekend.

Saturday evening

Slack for iOS Upload-4At 4pm, the site went live, and Berta shared information about Amé with three Facebook groups of Spanish and Catalan people living in the UK. Then we waited. And waited. And waited.

Almost three hours later, Amé’s phone pinged, a sound we’ve grown to know well. My heart jumped into my throat. I opened Whatsapp and saw a message from someone called Javier. It was a photo of an empty coffee cup. We exchanged the following messages in Spanish:

Javier: What do you see in the photo?

Amé: Hello.

Amé: I see ‘a cup’.

Amé: I hope you enjoyed the coffee!

Amé: Can I help with anything else?

Javier: No, thanks.

A few minutes later, Javier shared a screenshot of the conversation in a Facebook page with 38k members. And then the phone started pinging. And pinging. And pinging.

Between 6:42 on Saturday and 3:59 on Sunday, Amé would exchange over 1000 Whatsapp messages with the 84 people who contacted her to ask questions. If they wrote in Spanish, Amé answered in Spanish; if they wrote in English, she answered in English. She translated a mountain of obscure words and common idioms. She learned that in Spanish, curiosity kills the man, not the cat. She checked the openings and closings of cover letters. She helped people attach files to their emails. She rescheduled job interviews. She recorded words so that people could hear how they were pronounced. She explained why we eat dinner in England so early. She defined the word ‘else’. She gave restaurant recommendations. She dived into conditionals. She taught more than a few swear words. She told people how to get their hair layered and their fringe trimmed. And she helped a woman explain that she loved someone, only more like a brother.

The people who got in touch were funny, curious, cheeky, flirty, suspicious and deadpan in almost equal measure. They were unfailingly polite and grateful for the information they received. We didn’t experience a single instance of trolling. The whole experience was an absolute delight, for us and, I think, for them. This is hugely significant, as one of the principle objectives of the LX design process is to create a usable, useful and delightful product. Amé was clearly delighting people, a fact confirmed later by a very promising Net Promoter Score of 62 (Net Promoter is one way of measuring delight in a product or service).

Interestingly, one of the most common questions Amé fielded was this: ‘Are you a person or a robot?’. The first time someone asked this, we were stumped. We hadn’t prepared for this question at all. Off the top of my head, I typed in Spanish ‘We’re all something’. This was to become the standard retort to the barrage of questions about Amé’s humanity, questions that often amounted to a Turing test (the most common technique being to send me a photo and ask Amé to describe it, which she could of course do unfailingly).

Against this backdrop,  Amé soon developed her own personality: knowledgeable but playful (she loves emojis) but just distant enough to maintain a certain mystique about whether she was human or bot. Key to creating an authentic voice for her was the fact that, in my head, she had started to feel like a real person.Screen Shot 2016-03-22 at 10.11.27

Sunday

Saturday night had been spent almost entirely on the phone. I’d had to walk the twenty minutes to Stratford tube whilst typing in order to keep up with the barrage of messages. At it’s peak, it was easily one per minute. On Sunday we decided that the best thing would be to remove the number from the website at 4pm (exactly 24 hours after we went live) but to allow anyone who had asked a question up to that point to remain in contact with Amé. Anyone contacting us after that point would be invited to join a waiting list. We also decided to send out a survey at 4pm to anyone who had asked a question, to which over a third of our users responded.

The feedback was overwhelmingly positive. When asked to rate on a scale of 1–10 how satisfied they were with the answer that Amé gave them, the average was 8.36. When asked on the same scale how likely they were to recommend Amé to a friend, the average was 8.82. One person, on answering that question, had given Amé a 3 out 10. His rationale: ironically, he’d been expecting a human being at the other end of the line, and had been disappointed to get a bot.

We also began the data analysis, which is all about figuring out the metrics that matter. Anyone can create a fun, viral product with novelty value. What creates sustaining businesses, however, is customer habit – that is, do people want to use your product or service again and again. The metric we chose to measure customer habit was number of sessions. That meant the number of times someone came back to reinitiate contact with Amé. On the first day, the highest number of sessions was four, which is highly encouraging. The average, however, was 1.2, which is not. That meant most users only engaged with Amé once in 24 hours. However, the following 24 hours proved extremely encouraging. 22% of our users re-engaged with Amé the following day, and the average sessions went up by 33% to 1.8. These are all early signs of an engaged audience, which is exactly what we want to see.

What’s next for Amé?

On Sunday morning, it suddenly struck us that we’d made one mistake with our market segmentation. In targeting Spanish-speakers in English-speaking contexts, we’d zeroed in on a group that is most likely to have language problems that were high need, high urgency. The real test of Amé’s mass appeal is whether Spanish speakers living in Spanish-speaking environments also find her useful. To test that, we’re going to run a similar experiment in the next couple of weeks but aimed at Spanish-speakers in Spain.

After that, who knows? We need to figure out the scaling question, of course. Interestingly, 50% of respondents to our survey said that they’d expect Amé to respond to their message in one minute or less. That’s an impossible ask without some kind of AI solution.

Until then, we’ll continue to capture and monitor the data each day – what people are asking and how often – and try to figure out where to go from there. Amé will keep answering the questions, whether it’s about sushi restaurants in Ealing or the difference between update and upgrade. In the meanwhile, if you know a Spanish speaker with a burning question about English, you know where to send them …

 
ELTjam can run hackathons for your organisation. Get in touch for more information.

17 Comments

  1. Lovely post – a deceptively simple but ingenious idea.
    Have you kept the data from the interactions? Have you thought about how you could use it to create a corpus or maybe even a question-and-answer model?
    Who knows, you might not need AI to automate answering the most common questions, and you should at least be able to build a tool to help scale up the workload Ame could handle…

    1. Many thanks, Paul! We have ‘data’ in the sense that we have all of the messages on Whatsapp (17,350 and counting as of this morning!) but there sadly doesn’t seem to be a way of exporting that data into any kind of useful format. So any data analysis would be quite painstaking – a case of trawling through every conversation and picking out the question topics.

      It’s hard to tell without doing any actual analysis, but my sense four months into the experiment is that the range of questions is surprisingly diverse given that all of our users share the same L1. Some questions certainly come up again and again (‘like’ vs. ‘as’, for example) but repeated questions are not that common, and far less common than I’d expected.

  2. Really interesting concept, and it’s clearly got legs. Scaling would definitely be a challenge though, bots and AI have clearly made great improvements recently (think Amazon’s “Echo”), but in order to work even reasonably effectively companies have had to leverage enormous amounts of data. There’s still no bot that’s really effective, even in English – trying this in Spanish would add even more complexity!

    There’s a really interesting article (https://medium.com/chat-bots/bots-hype-or-glory-656f4d614efb#.dj5z5x3fy) on someone’s similar experience in “acting” as an undercover bot. His two month experience showed him that a decent UX (perhaps “LX” in your case?!) is better than any bot in existence currently, simply from the challenges of AI semantic understanding, not to mention returning relevant data!

    Ultimately, I feel this is where Google is looking to go – understanding a user’s query in any language, and returning not just organic and paid search links, but actual answers.

    All in all, great experiment and I think you’ve definitely proved the interest. Thanks for sharing – look forward to seeing where you go with this next!

  3. Great post, Nick. I’m not sure how you can scale up, though, without losing Ame’s charm, mystique and unique character – all the things that secure a high net promoter score. I suspect you may pivot at some point down the line, moving away from Ame the potential mass product to Ame the focused, but resource-intensive data collection/research vehicle.

    By the way, just to reassure me that Ame hasn’t been kidnapped – can you ask her what Andrew Ridgeley had for breakfast today?

More comments

Leave a Reply

Your email address will not be published. Required fields are marked *

TwitterLinkedInFacebook

Other related posts

See all

New decade, new Jam!

My English learning experience – 6 lessons from a millennial learner

Deconstructing the Duolingo English Test (DET)