Will Thalheimer: Does eLearning Work? What the Scientific Research Says!

by Mirjam Neelen & Janet Benson – Learning Experience Designers

What does the scientific research tell us about elearning effectiveness, particularly as it compares with classroom instruction? These are the questions that Will Thalheimer examines in his fresh-from-the-press report. But wait. Why would we even ask these questions? Don’t we already know that eLearning works? After all, it’s used abundantly; tons of people use it every day. Thalheimer rightly says that eLearning, on the other hand, has a reputation for being boring and ineffective and wildly hyped by vendors and eLearning evangelists. So, in summary: No, it’s not clear if it works and it’s not clear when it works, what causes it to work. Do we make sense?

By the way, it’s just awesome that the world of learning professionals can avail of someone like Thalheimer who does all this solid research and then makes it available for free so that we can all benefit from it! Perhaps you can consider hiring him or buying his book to pay him back (and to get more awesomeness!).

We, as learning experience designers at the Learnovate Centre are big advocates of backing everything that we do up with science. Actually, we feel it’s our responsibility. Mind you, Thalheimer’s research includes thousands of scientific studies as his research includes several meta-analyses to find the answer to the two questions.

Comparing eLearning to Traditional Classroom Instruction

Interestingly, based on the five meta-analyses that Thalheimer conducted, he concludes that “eLearning tends to outperform classroom instruction” (p 4). Blended learning creates the largest benefits (so, a mix of classroom and eLearning instruction).

Are you getting excited by this outcome? Don’t be. When diving deeper, it becomes clear that it’s NOT the eLearning that improves learning effectiveness, it’s the LEARNING METHODS that are typically used in eLearning that impact eLearning’s benefits.

Paul A. Kirschner tweeted, in response to Thalheimer’s article, that Clark posed a similar conclusion all the way back in 1983 (also referenced by Thalheimer, by the way), saying that

The best current evidence is that media are mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition (Clark, 1983, p 445).

Which Learning Methods are Most Effective?

So, why would eLearning outperform classroom instruction and why would blended learning outperform both? One possibility is that eLearning design invites the usage of more effective learning methods. But why then would blended learning be even better? Thalheimer suggests that one reason could be that the eLearning used in blended learning experiences uses more effective learning methods. Perhaps it’s the combination of those effective methods and the comfort of a face-to-face setting? Another option is that, because the research analysed on blended learning didn’t compare the specific learning methods used, so perhaps, coincidentally, the blended learning methods investigated in the research used even more effective learning methods than the learning methods used in the eLearning studies. We’re not sure and Thalheimer doesn’t speculate any further.

So, what are these effective learning methods? Not surprisingly (to us anyway) they include elements such as “providing learnings with realistic practice, spaced repetitions, contextually-meaningful scenarios, and feedback” (p 4). Other articles discuss the richness of eLearning, such as the flipped classroom and mobile learning.

Overall, “realistic decision making and authentic tasks, providing feedback on these activities, and spreading repetitions of these activities over time” (p 10) are hugely beneficial for learning.

Now, please open your ears widely and listen cause what we’re about to say is important.

Learning effectiveness depends on the learning methods, NOT on the ‘delivery’ methods (classroom or eLearning). Classroom instruction can ALSO use these beneficial learning methods and eLearning can FAIL to use these most effective learning methods (and that, as we all know, happens too often).

Comparing Learning Methods in eLearning

From the studies that Thalheimer reviewed for this topic, it becomes clear that there is an enormous variety in eLearning approaches and contexts. For example, solving linear algebra problems or making sense of cerebral haemorrhages. This is critical because, especially in the corporate learning space, the feeling is that eLearnings are just boring, ineffective page turners. Thalheimer’s research shows that eLearning can be used for much more than information dumps or learning trivial tasks (e.g., How to Clean Your Desk on a Friday kind of stuff). At the same time, it becomes clear from Thalheimer’s research that many eLearning applications don’t utilise research-recommended learning methods so therefore the potential is … underutilised.

Other Research Relevant to eLearning

Thalheimer also explores a random sampling of research that focuses on things like simulations, simulation games, feedback, animations, digital games, and so forth. Unfortunately, he doesn’t explain how he defines eLearning. One could say that simulations and digital games can be part of eLearning as well.  The research on these ‘other’ types of learning tech show that they investigate “learning methods that tend to go beyond those we typically use in classroom instruction” (p 24). Well, we could use them in classroom instruction but we generally don’t. There’s a lot of variability in the research results but we can conclude that learning technologies seem to be, overall, beneficial in supporting learning, though, and this is important, NOT necessarily better than non-tech interventions.

This is What You Need to Remember at All Times!

  • eLearning and classroom instruction produce similar learning results when learning methods are constant.
  • eLearning tends to outperform traditional classroom instruction when no specific effort is made to hold learning methods constant (suggesting that eLearning solutions more easily invite to use more effective learning methods somehow while classroom instruction still relies heavily on relatively ineffective lectures).
  • What matters for learning effectiveness are the learning methods; NOT the modality (e.g., eLearning or classroom).
  • Blended learning tends to outperform classroom learning and eLearning (probably because the eLearning part of the ‘blend’ often uses more effective learning methods).

We are especially excited about this type of research because it helps us to help you to identify and solve your learning challenges. Don’t hesitate to contact us with any questions!

We’ll be waiting for your call.

References

Clark, R.E., (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445-459. Retrieved from https://www.researchgate.net/publication/247357883_Reconsidering_Research_on_Learning_from_Media

Thalheimer, W., (2017). Does eLearning Work? What the Scientific Research Says! Retrieved from http://willthalheimer.typepad.com/files/does-elearning-work-full-research-report-final.pdf

We’re Hiring – K-12 Pedagogy Specialist

Post: K-12 Pedagogy Specialist
Contract Duration: Specific purpose for length of Learnovate Project
Department: Learnovate Centre, School of Computer Science and Statistics
Salary: Researcher Scale at a point commensurate with qualifications and experience
Closing Date: August 25th 2017

Summary

Candidates are invited to apply for the position of Specialist and Researcher, K-12 Pedagogy in the Learning Design Team at the Learnovate Centre. The function of the role is to provide support to Learnovate Centre projects focused on learners in the K-12/schools space. The work will be conducted in collaboration with a dynamic, multi-disciplinary research team with the aim of developing innovative next-generation learning technology which demonstrates real benefits for school age learners.

Principal Duties:

  • Taking ownership of, defining and developing the pedagogical strategy for K-12 research and innovation projects
  • Ongoing engagement with the Learnovate Schools Network, creating and maintaining relationships with the schools
  • Ensuring that research projects demonstrate real learning innovation through:
    • Defining and implementing a pedagogical strategy which is  appropriate  to the context of learners
    • Creating relevant use cases primarily focusing on K-12 school learning contexts
    • Providing pedagogical direction
  • Conducting evaluations/trials to support research projects
  • Communicating and transforming research outputs for industry partners
  • Presenting/reviewing research projects with industry partners
  • Preparing and writing -publications to support the work of the Learnovate Centre
  • Carrying out other pedagogical tasks which may be assigned

Key Requirements:

  • Qualification to PhD level in the field (or relevant industry experience)
  • Experience of classroom teaching at either Primary or Post-primary level
  • Experience with learning technology deployment and use in classroom setting
  • Experience of curriculum/ course design
  • Experience in assessment design (of and for learning)
  • Excellent research and writing skills
  • Proven ability to prioritise own workload and work to exacting deadlines
  • Excellent verbal communication and interpersonal skills
  • Strong team player

Desirable:

  • Experience working in a research-driven working environment
  • Experience of conducting end user trials
  • Experience in designing technology-enhanced learning environments
  • Experience with Design Thinking, LEAN Start Up and/or Agile development

Background to the Post

The Learnovate Centre is an Enterprise Ireland/IDA funded centre of excellence in learning innovation. The centre brings together state of the art research with the real world business problems of our industry partners. The centre aims to develop innovative industry-led learning technology solutions to benefit the Irish EdTech industry.

Department Summary

The School of Computer Science and Statistics at Trinity College Dublin offers a high quality educational experience to undergraduate and postgraduate students along with exceptional research opportunities for professionals in the fields of computer science and statistics. The school is proud of its reputation as an innovative and energetic centre for study and research.

The research interests of the school are wide and varied, ranging from the theoretical to the practical. The schools researchers are at the cutting-edge of their disciplines, working on prestigious research projects with other professionals in their fields, and with access to significant public and private funding and industry support.

Trinity College Dublin

Founded in 1592, Trinity College Dublin is the oldest university in Ireland and one of the older universities of Western Europe. On today’s campus, state-of-the-art libraries, laboratories and IT facilities, stand alongside historic buildings on a city-centre 47-acre campus. Trinity College Dublin is currently ranked 43rd in the top world universities by the Times Higher Education Supplement Global University Rankings 2009 and 13th in Europe. Trinity College Dublin offers a unique educational experience across a range of disciplines in the arts, humanities, engineering, science and human, social and health sciences. As Ireland’s premier university, the pursuit of excellence through research and scholarship is at the heart of a Trinity education. TCD has an outstanding record of publications in high-impact journals, and a track record in winning research funding which is among the best in the country. TCD has developed significant international strength in its research in eight major themes which include globalisation; cancer; genetics; neuroscience; immunology and infection; communications and intelligent systems; nano and materials science as well as Irish culture and the creative arts. TCD aims to become the world reference point in at least one of these areas of research in the next 10 years

Application Procedure

Candidates are asked to submit a covering letter and a full CV to include the names and contact details of 3 referees (email addresses if possible) to:

Owen White, Centre Director

Learnovate Centre, School of Computer Science and Statistics, Trinity College, Dublin

Tel: +353-1-896- 4910

Email: owen.white@learnovatecentre.org

Please note: No expenses will be paid in travelling to interview.

Learning Tech Ireland 2017 – Reflection

By Peter Gillis

“Working in eLearning/education in Ireland for nearly 20 years, I have been seeking events like this, and really felt this was the first I’ve attended that really provided useful, research-led and practical advice”

Brendan Strong, Director of Education at Society of Medical Professionals

crowd

At Learnovate, as we enter phase 2, we have been working hard over the last few months to design and develop a value proposition that will see the centre answer the real needs of our customers, create ‘pull’ for our services and deliver real impact. We took the decision to focus this year’s conference around the concepts that underline this value proposition, to give members and the wider community an understanding of how Learnovate can help them realise real impact in the area of Learning Tech. We term our approach Impact Led Innovation designed to combine the best of Lean Start-Up and Design Thinking to identify from the outset, learning problems people really care about, validate these assumptions, develop solutions with end users and identify the sustainable impact that will maximise the potential for success.

This involved us taking a very different approach to our conference, rather than showcasing technologies and discussing themes in the area we chose, for this year, to provide a three tiered structured approach to the day. We knew it was risky, but we also knew it was an indirect and slightly messy way of ‘validating our assumptions’ in relation to our value proposition.

The purpose of this blog is not to recount the excellent presentations, but to deliver prompts for those who were there and a flavor to those who could not make it, we would be happy to follow up in person on any aspect of interest.

speakers

Following introductions and an outline of Learnovate’s Value Proposition to deliver Impact Led Innovation for our members, the first stage was to deliver insights from world class experts in the area of Lean Start-Up, problem validation, identifying what customers need and avoiding the pitfalls that can arise.

Tendayi Viki spoke of the zero correlation between R&D and growth, a point that raised a few eyebrows in the audience, however the point here was that pure R&D has no correlation, it is only when R&D is linked to innovation and customers’ needs that it is effective, a point illustrated by Tom Melia from Enterprise Ireland earlier in the day. Tendayi spoke of avoiding what he refered to as ‘Innovation Theatre’ carrying out the exercises to have the appearance of innovation, while taking your eye off the end prize. Developing a term from Steve Blank, the audience was introduced to his ‘Eight Steps to the Epiphany’ (four more than Steve!), they are:

  • Capture Ideas
  • Identify Assumptions
  • Prioritize Assumptions
  • Brainstorm Tests
  • Falsifiable Hypotheses
  • Get Out Of The Building
  • Capture Learnings
  • Make Decisions

Personally I thought his phrase “It’s not iterating if you do it only once” was one of the more humourous and valuable snippets form an excellent session.

Rob Fitzpatrick focused his Keynote on talking to customers to first ‘Learn’ if a problem you believe exists, exists at all and if so does anybody care! Secondly to ‘Confirm’ that customers will use it, and importantly, if they will pay for it! Rob also addressed the three mistakes that are made when talking to customers, Asking for Opinions, Meetings that go nowhere and Wasting time. Your ego may do well indulging in the activities above, but realistically theu get you no closer to understanding if your idea will be successful or not.

3Panel

The second stage was designed to bring the audience a little closer to Impact Led Innovation through three practitioners who have applied the concept in their projects and lessons learned.
Vivienne Ming talked about how at a high level we need to address the human condition, to have high and ambitious targets if we really want to drive change and pull for our ideas. Vivienne referred to the failings of summative assessment in education to attain such outcomes. With reference to Vivienne’s own work with her EdTech company Socos Vivienne referred to the need to engage with users to identify real problems, and to ensure that there is product market fit, that there is a business demand, otherwise the concept while commendable may not survive to achieve the desired impact.
Paidi O’Reilly talked about the lessons he has learned through the application of Design Thinking and Lean principles in the area of EdTech through his time working with Texuna Technologies in collaboration with UCC. From his experience he identified 12 hard lessons learned, including the importance of how we frame a problem, the importance of identifying needs over wants, and personally my biggest take away of the twelve, behavior is the real issue, in Paidi’s words “We need to accept that behavior is probably bigger than technology”.
Neil Peirce from Learnovate gave a hands-on talk about how our Horizon 2020 project has engaged design thinking. Neil gave a detailed talk to show how different aspects have been used to define customers and their pain points. The stages demonstrated identifying users through persona development, validation of same and subsequent revisions based on feedback from interviews. The follow-up workshop process involved identifing both the “As-is” situation, the pain points for users and subsequently the “To-be” situation to discover what could make it work. This informed the development and definition of User Stories to inform the project. Neil pointed out 5 key learnings for carrying out successful workshops:

  • Diversity is important
  • Plan for social barriers
  • Plan for social norms
  • Experts as facilitators
  • Allow for prep-time & documenting results

When it comes to User Testing Neil also pointed out that his team’s three takeaways were:

  • Structured & repeatable sessions, clear objectives
  • Carefully selected questions
  • Test early, test often

groups

Finally we wanted to allow attendees get up close and personal with the concepts through two parallel hands-on workshops. Tendayi focused his workshop on Strategyzer’s Business Model Canvas. Teams of approximately ten identified hypothesis for a learning technology that they believed would be of value. In rapid fire sessions Tendayi challenged the teams to identify and prioritise the assumptions that lay behind their hypothesis, then to prioritise their most risky assumption and consider how they might test this to validate the idea.
Rob’s workshop looked at ‘customer development’. Based on Rob’s book ‘The Mom Test’, Rob hosted an engaging and interactive session looking at how to get around the comfort of compliments and opinions when talking to potential customers, because basically they will lie to you! How to approach customers asking for concrete feedback on specific issues in their past and identify real problems they have, that they value being solved, and are not being addressed at the moment.

In conclusion Owen White finished the day by clarifying one question. Is Learnovate now an innovation consultancy rather than a Learning Technology research centre? Absolutely not, the Impact Led Innovation capability at Learnovate is to support our ongoing development of solutions in the Learning Tech market, combining our innovation expertise with our existing expertise in Learning Design, UI/UX, Technology and Commercial development.

As I mentioned at the start, ‘Mom Test’ beware but our initial feedback from members and attendees was that we were hitting the right note, the topic resonated and people saw clear value in the approach. Among many positive unprompted messages from from attendees on the day was the following from Brendan Strong, Director of Education at Society of Medical Professionals “Working in eLearning/education in Ireland for nearly 20 years, I have been seeking events like this, and really felt this was the first I’ve attended that really provided useful, research-led and practical advice”

We look forward to assisting our member companies with their Impact Led Innovation projects and to growing the impact our industry has on the world stage.

Learning Technologies Summer Forum

By Janet Benson

JB2a

The Learning Tech Summer Forum (LTSF) is an event which is designed to develop and expand upon the popular themes from the February Learning Tech conference and exhibition, which Mirjam blogged about earlier this year.

The conference part of the forum focuses on learning in practice, and featured topics such as emerging technologies, collaborative learning, practical social learning and user experience. The associated exhibition contained over 30 seminars and an exhibition area with nearly 40 exhibitors from research companies such as Towards Maturity to learning tech companies and learning industry suppliers such as eLearning Studios and the Fosway Group.
With a day as packed as this, choosing which events to attend can be difficult, and I found myself wishing for a time-turner from Harry Potter to enable me to optimise my day.

Learning and the Brain

jb2c

After the welcome note from organiser Donald Taylor, the opening address was delivered by Dr. Itiel Dror, a neuroscientist at UCL, and dealt with learning and the brain (http://www.cci-hq.com/home.html). A very funny and engaging speaker, Dr. Dror highlighted that ‘REAL learning is difficult’ while discussing 3 critical (and intertwined) perspectives of real learning at work; namely:
• Acquire – learners need to understand to be able to learn
• Memory – learners need to be able to retrieve information in the long run
• Apply – learners need to be able to apply what they have learned when they return to the workplace

Itiel discussed how the brain has limited resources and that we need to help the brain to learn by guiding it to the right places where possible. He also stated on numerous occasions that “the human mind is not a camera” and that it is active and often requires rewiring in order to change how we do things (we are creatures of habit).

speakersLT17

The now and the next of learning and technology

The first seminar I chose to attend was given by David Kelly of the eLearning Guild and was entitled ‘Emerging Technologies – the now and the next of learning and technology’.
David talked about disruption, how disruption itself is neither positive nor negative, but it’s how we react to it that counts. He spoke about how we need to always look at the risks of doing things, as well as at the potential benefits, something which I believe we need to do more with regard to learning and technology and to ensure we don’t jump on the bandwagon of new techs and fads. He suggested approaching technology with a sense of play, and recommends looking at how technology is changing how people live before we look at how technology might impact on how people learn.
On the future of learning and technology, David Kelly discussed interactive video, virtual reality, augmented reality and wearable technology, and referred to a nice recruitment video from Deloitte: http://www.raptmedia.com/customers/deloitte/.
Similar to ‘Choose Your Own Adventure’ books that I enjoyed as a child, interactive videos look very slick and allow the user to make decisions, but may be expensive to develop and require a certain level of video editing skills as well as access to the relevant software to be effective.
David referred to Karl Kapp in his discussion of game-based learning and Kapp’s view that we cannot implement this type of learning using the linear instructional design approach. Having read Kapp’s book, ‘The Gamification of Learning and Instruction’, I would agree with Kapp’s view on this; however, this is the case with a number of learning approaches and we as learning experience designers need to constantly evolve and upskill.

Micro Learning?

I was interested to learn what Clive Shepherd and Barry Sampson had to say on the topic of micro learning so attended their afternoon talk on the topic.
I am not sold on the notion of ‘micro learning’ and feel that it is simply a buzzword for something that is, essentially, just learning. It is commonly known that learning should be broken into smaller chunks where possible to avoid the risk of ‘cognitive overload’ in learners and the use of videos, infographics, etc. is simply a way of doing this, rather than making it ‘micro learning’.
Clive and Barry defined micro learning as ‘a way of organising self-directed learning into lots of small chunks’ and referred to ‘how-to’ videos as the most common type of micro learning. I believe the word ‘small’ in this definition can be subjective and again, the use of short videos is merely a way of learning, not a type of learning in itself.
The speakers did admit that micro learning cannot be applied to any learning problem and is not inherently motivating, while also stating that spaced practice and repetition are advantages of micro learning. Surely spaced practice and repetition can be applied to most learning scenarios and are not only advantages of so-called micro learning?
I won’t overstress my position on micro learning, but needless to say, I’m still not sold.

UX/LX

My final seminar of the day was delivered by Myles Runham, an independent learning consultant, and concerned the area of User Experience: ‘Why it’s fundamental and how to make it work’.
Myles encouraged the attendees to share their favourite websites and digital products and highlighted that user experience is the primary reason why people switch systems. He also shared the top ten learning tools from 2016, with YouTube at number one and Google Search and Twitter in the second and third spots, respectively.
I agree with Myles’ opinion that user design and learning experience design are not really any different, and I believe as an LX designer that we can use Myles’ 3 rules for a good user/learner experience:
• Simple – know what it’s for/what’s expected
• Consistent – easy to move through/work with
• Standard – familiar/obvious
Myles reiterated that point that we need to focus on our users (learners) and what they need as well as getting to know them and bringing them into the planning and development process. Stakeholders may be focused on a different problem, and this is a key takeaway for me as we move into Learnovate Phase II and our new value proposition.

My Takeaways?

Regarding my personal focus on learning experience design, the key takeaways for me were how most training is really re-training and that we need to analyse how similar or different the previous information is that we want to ‘rewire’.
Also, external motivation such as targets and performance reviews can often get in the way of learning by way of corrupting motivation. We need to get our learners on board before we can even begin to hope for change to occur and where possible we need to give them an experience.
As Dr. Dror says, “it’s not what you teach but what they learn that counts”.
As I had to run for my flight, I missed Donald’s closing address, but with a number of takeaways to bring back to Learnovate, I was satisfied that my attendance at this conference was worthwhile for both me and for the centre and am already looking forward to next year.

Impact Led Innovation – 3 stories from the field

Impact led innovation is all about practical implementation. At Learning Tech Ireland you will get insights from three speakers sharing their experiences applying the concept in real world, learning technology, scenarios.

storiesfromthefield

Vivienne Ming (Named one of Inc. Magazine’s “Top 10 Women to Watch in Tech”)
Vivienne will share two experiences developing products, one in the classroom and one in the home, to solve real problems for real people. AI-driven systems for education, the importance of design thinking for improvement science.

Paidi O’Reilly UCC
Lessons learned from leading an Edtech Research Centre and introducing Design Thinking as a key organisational capability. With empathy-building activities at its core, the Centre specialises in leveraging open source software to develop innovative classroom engagement, assessment, and adaptive learning solutions.

Neil PeirceLearnovate
The aim of the H2020 DEVELOP project is to deliver an adaptive learning environment that dynamically tailors the exploration, comprehension, and planning of learning opportunities and career paths in medium and large companies. Neil will discuss the Design Thinking approach the team took to get a deep understanding of the user needs and how building on these insights DEVELOP seeks to provide effective solutions.

The day also includes; world expert keynotes, 2 workshops and free guides on implementing Impact Led Innovation. Places are filling up, do not miss this excellent opportunity to boost your team’s innovation capabilities. To book now click here

webPage-01