Gartner uses terms like “confusing” and “crowded” when discussing learning technologies in research published last year. The corporate learning market is complex one, packed with more than 500 software providers, researchers found.
One thing that should not be confusing is key performance indicators, or KPIs. What are you getting out of your learning systems? What metrics should you consider, and how does that tie into your overall business strategy? These are not easy questions but the answers can make or break an investment in the most significant investment organizations make in learning technology, the learning management system, or LMS.
Identify the Types of Learning Programs
Measuring the value of learning programs — ultimately the KPI you are looking for — is challenging, according to Katy Tynan, principal analyst at Forrester who covers performance and talent management. It depends on having a clear understanding of the goals of those programs, she said. Know your type of learning programs, and that will help you define your KPIs.
According to Tynan, you can typically break learning programs down into three types:
Typically presented as “read and acknowledge,” the goal of these compliance programs is to make sure employees are aware of a policy and there is a record that the individual has read the policy. “For these,” Tynan said, “the value from the LMS is the ability to easily report on each of these programs so that when an auditor arrives and asks to see the records, they can be quickly and easily provided. Often the ‘training’ value of these compliance programs is minimal or nonexistent.”
Ranging from click-through PowerPoints to complex simulations and VR-based learning, the majority of programs delivered through learning management systems falls in this category, according to Tynan. “The goal here is skill building,” she said. “As such, the measure of success is whether the learner is able to acquire the skill and then demonstrate their competence. Typically this means that the programs include knowledge checks or other demonstrations of understanding or application.”
When creating KPIs for these programs, consider completion percentage as well as performance on knowledge checks. Track learner evaluation of the programs in terms of whether they felt that the content was valuable or whether it helped them increase their skills on the job, Tynan said. Connect these skill-based programs to organizational capability models to ensure that you are developing the skills you need in the organization now and for the future.
Soft Skills/Leadership Learning
Soft skill training is often the hardest to measure, according to Tynan. Well-designed leadership and soft skill programs are typically delivered as a hybrid of synchronous live learning (either classroom-based or live virtual sessions), online e-learning programs, supplemental materials and observational learning.
“As such, only some of the elements of the program would be delivered through an LMS and the true measure of success would ultimately be measured via engagement surveys or other types of leader evaluations,” she said. “It is possible to gather KPIs on completion and have knowledge checks similar to the skill-based learning model, but the ultimate behavior change would be visible only to managers and colleagues on the job.”
Standard KPIs to Consider for the LMS
According to Tynan, the standard LMS KPIs are:
- Utilization: How much of the content is being consumed and by whom?
- Completion percentage: Do learners complete the courses or drop out part way through?
- Evaluation: Do learners rate the courses as engaging, interesting or useful?
- Time spent: How much time do users, departments or the organization as a whole spend on learning through the LMS?
Keep in mind that these are standard KPIs for a reason, Tynan said. “The reason these are the standard metrics is that they are easy things for the system to measure,” Tynan said. “These are stock reports that can be run, but unfortunately they provide very little insight into the ROI of learning programs themselves. But because they are easy metrics to produce, organizations often fall back to using them as KPIs vs. finding ways to measure what is meaningful for the business.”
ROI Comes from Total Cost and Value
The true KPIs of learning programs, which allow the organization to measure ROI, include total cost and total value, Tynan said. Broken down more granularly, they include:
- Total cost to deliver a learning program, including development costs and the cost of the LMS itself.
- Time cost to learners and total value from the learning program.
Consider these factors when evaluating the latter:
- In the case of compliance programs, measure a reduction in noncompliance events, successful audits or other measures.
- In the case of skill-building programs, measure against cost to purchase a program from an external vendor or measure in terms of talent strategy metrics such as recruitment and retention costs.
- In the case of leadership and soft skills, measure through engagement surveys, manager feedback tools or other pulse surveys.
Measure Application of Knowledge on the Job
Don't forget to track the progress of the employees' absorption and application of knowledge. End-of-lesson tests are a vital KPI to track, according to Yaniv Masjedi, CMO at Nextiva. The efficiency and success of the current learning program established by the company should be evaluated, he said, and assessments are excellent methods to pinpoint problem areas.
“When employees fail to meet a certain standard, management should re-evaluate their approach towards their employees’ education," Masjedi said. "Getting input from their employees currently under the program also helps in determining areas that need improvement.”
Related Article: Top Features for Learning Management Systems
Measure When Employees Learn
How many employees log in to begin a learning course in an LMS before the day it is due is a good KPI to track, according to William Entriken, vice president, general manager at Pacific Medical Training. Many employees — and HR departments — treat training like a required drudgery that can be tucked in between or even during meetings, he added. Reviewing how many employees begin before their Outlook alarm tells them it is due is a way to see if workplace employees are engaged.
Consider Your Entire Learning Ecosystem
Lastly, one of the challenges for organizations is that an LMS is used for only a subset of learning programs. Because of that it is difficult to measure or have a holistic view of learning programs across the organization, according to Tynan.
“When implementing any LMS, this is a key consideration,” she said, “since having multiple systems can make it challenging to track and compare utilization, create reports for compliance purposes, and effectively monitor overall utilization of the programs.”