Essential Data Points

Kineo Demystify Learning Analytics

London (UK), August 2022 - Kineo has elucidated the key data points to collect for learning analytics, focused on mining insights from learner activity to make effective decisions.

Today over fifty percent of the top ten companies are data based platforms such as Google, Facebook, and Tencent. This implies that the market today values and prioritises data. It's even been said that data is the "new oil".

In fact we've become such a data driven generation that in 2020, we created 2.5 quintillion data bytes every single day! This massive volume of data, however, will not prove useful unless it's analysed.

Think of a business - you have many functions such as marketing, finance, manufacturing, etc. that are all churning out data. The learning function within an organisation is no different. "Learning analytics" is an approach that helps you collect and store essential learning data in order to generate meaningful insights - insights that can be used for decision making and taking the next steps for individual and team learning and development.

According to SOLAR (the Society for Learning Analytics and Research), learning analytics can be defined as "the measurement, collection, analysis, and reporting of data about learners and their contexts for purposes of understanding and optimising learning and the environments in which it occurs".

This has been hard to do under traditional classroom teaching methods, but application of the process has gained momentum with the rise of online learning and the eLearning age. eLearning makes data collection and analysis much easier and makes learning analytics more effective in assessing learning outcomes.

  • The use of big data and learning analytics can help capture powerful feedback on learning experiences that can translate into designing learning solutions that are more engaging and effective.
  • Learning analytics has the potential to drive a cycle of continuous improvement that transforms the design, development, and delivery of learning solutions.
  • Learning analytics opens up a world of opportunities for the L&D professional. It helps determine what transpired during an eLearning session, why something happened, and patterns of occurrences.

Thus the real focus becomes the kinds of data needed in order to draw these inferences and make decisions that improve learning, raising the question of the specific information learning analytics tools should be picking up.

Three key categories of data should be monitored to make effective decisions:

1. User Data

Data relating to users or learners who are enrolled in your organisation's eLearning is considered user data. This data describes the learner in the context of the organisation. For example, it can encompass the user's role in the organisation, location (in case of a multi-regional office), division (sales, operations, finance, human resources, etc.), and level (or seniority such as junior, mid-level, executive level).

When user data is collected, it helps segment data as part of the analysis, rather than having all the information under a single umbrella. Ungrouped data implies that valuable insights for each group become indistinct due to overlapping information. Grouped data always helps bring to light trends or patterns that are relevant to each user group.

How is user data useful? The learning experiences of users with common roles within a department, division, or geography can be studied to determine the elements lacking in the learning programme.

2. Engagement Data

All the elements that reflect how a learner interacts with the content will give you relevant engagement data. This means examining various aspects such as

  • the length of time that the learner has the module open
  • the number of unique users who started a course
  • the number of unique users who completed a course
  • the number of users who visited a course more than once

Some practical implications of engagement data are

A. Understanding whether the allocated course time was appropriate

  • Do most learners take longer than the allocated time to complete the course? (This may be indicative of a course that's not very engaging).
  • Do most learners take less time to complete the course? (This may be indicative of a course that the majority of learners are already familiar with or one that is not challenging enough).

B. Understanding how (and why) the learners interact with the media files

  • Did learners open media files where they were supposed to? Or did they skip them altogether?
  • Did learners stop certain media files before completion? This may indicate that the information is too long or that the information is not useful.
  • Do certain design elements engage learners better than others?
  • What mix of media is ideal for the type of content being taught? For example, text + video + multiple choice quiz; or video training + simulated scenario-based testing; or video training + pop quiz via mobile learning + social media learning bites; or other media mixes.

3. Performance Data

Performance data is data that characterises how well content was recalled or applied by learners. In other words, it indicates how the learning impacts performance.

Performance data can be used to analyse whether the instructional design was solid and whether the course authors understood the needs of the audience accurately. It can also be used to analyse whether the content was written in an easy-to-understand and memorable style.

The L&D professional can take this data to both assess the effectiveness of the solution and as a baseline to compare to actual performance in the field.

Here are some examples of the kinds of performance data that can help the learning analytics tool provide significant information:

  • data that indicates whether a question was answered correctly on the first try
  • results on an "apply" question-like scenarios, which can demonstrate whether new skills are likely to transfer to the job
  • results on a "recall" question, which can provide insight into whether content is presented clearly and in a memorable way
  • comparisons of learner confidence at the start and end of a course, which can demonstrate the course’s effectiveness
  • data relating to whether the content was relevant to the individual's role

Accurate and relevant data collection is an important part of learning analytics. An effective learning analytics tool will make collecting data on the development, participation, and outcomes of learning possible. This will, in turn, enable better learning outcomes for your team.

Other benefits to prioritising your learning analytics include

  • measuring effectiveness of individual and team learning
  • measuring the impact on the business
  • providing actionable information to improve learning
  • providing data to meet stakeholders' expectations
  • providing data to support the L&D department to meet its goals and objectives