Measuring and Evaluating the Effectiveness of Security Awareness Improvement Methods

Measuring and Evaluating the Effectiveness of Security Awareness Improvement Methods
Author: Eszter Diána Oroszi, CISA, CRISC, CISM, ISO 27001 LA
Date Published: 6 September 2023
Related: Implementing Robotic Process Automation (RPA) | Digital | English

Measuring and improving the security awareness level of users is always an interesting topic among information security experts. There are many different methods for improving the security awareness level in a workplace environment, but they raise many questions. What are the most effective ways to improve security awareness? Which actions can improve the security knowledge of participants, and to what extent? Which elements of security awareness programs do employees most prefer? Are preferred methods more effective than other options? Research conducted between August 2021 and March 2023 aimed to find the answers to these questions by comparing the effectiveness of security awareness improvement actions. Only Hungarian organizations were included in the scope of this research.

The Research

The goal of the research was to evaluate the effectiveness of security awareness improvement methods. The research was carried out at 10 organizations, of which half were public institutions and half were enterprises operating in the private sector. Each organization provided 30 employees to participate in one of six targeted training events. Each user could take part in only one type of improvement action to ensure that the source of the knowledge could be identified. There were no further restrictions―such as arranging the composition of the teams by gender, age or position, for example. Employees could be designated to participate through mandatory assignments or they could apply for the programs. General users with a lower level of IT knowledge and employees who were not trained in information security within the last year were preferred, but this preference was not enforceable. However, the associated risk was negligible, and it was not practical to enlist a total of 300 people who had never taken part in any kind in security awareness training or did not have related IT knowledge. Figure 1 shows the workforce size and figure 2 shows the industry breakdown of the organizations involved.

Figure 1

Figure 2

Figure 3 shows the age distribution of survey respondents, and figure 4 shows when they last trained in security awareness.

Figure 3

Figure 4

To evaluate security awareness improvement actions, an original assessment method was used to compare six selected training opportunities provided to users at the organizations involved: one traditional with in-person participation, one traditional online, one recently popular online (after COVID-19), one formerly popular with in-person participation (before COVID-19), and two gamified educational methods. Based on these requirements the selected methods were:

  • Classroom training (presentation)
  • Online training (live presentation)
  • Elearning materials
  • Campaign elements
  • Escape room
  • Board game

The research methodology employed three similar questionnaires, each containing a unique ID number, some statistical and multiple choice data, and a critical survey question, inviting a free text response: “List all security awareness rules and best practices you would tell a new colleague or family member who asked you about information security.” The use of this method instead of a list of possible answers with checkboxes avoided the possibility that some respondents might mark answers as right, even if doing so did not reflect real knowledge. With the free text requirement, user responses were based on acquired knowledge and applied practice.

The research was anonymous, but to connect questionnaires to each other, users had to generate a 13-digit ID, which appeared on every questionnaire.

Information security personnel with the organizations involved organized training events in support of the research, which lasted for six hours per organization. Employees participated in person, although in the case of online training, it was mandatory for them to connect via a videoconference application. If necessary, the elearning material and campaign elements options could be performed remotely.

As figure 5 shows, before each event the research was explained and the first paper-based questionnaire distributed (a Google Forms link in the case of online methods). Participants had 10–15 minutes to answer the questions. Following the collection of questionnaire 1, presentation of the appropriate program took place for a maximum of 30 minutes. At the end, participants received and completed the second questionnaire. The purpose of questionnaire 2 was to identify how many (and which) new information security knowledge elements would appear in the answers. The last step involving user participation took place one month later. Responses to questionnaire 3, which could be completed online using Google Forms, indicated which elements of the program were permanent (i.e., whether they also appeared as new knowledge in questionnaire 2, or appeared only in questionnaire 3).

Figure 5

Free text answers were classified into 10 categories of security awareness knowledge:

  1. Clean desk policy
  2. Clean screen policy, locking computer
  3. Keys and tokens
  4. Hardware devices
  5. Passwords
  6. Shredding documents
  7. Phishing
  8. Malware
  9. Social media
  10. Phone and smart devices

These categories were selected based on the researcher’s social engineering audit experiences and general security awareness improvement actions.1 Of course, these topics were part of all six of the improvement methods involved in this research. Any additional user-supplied knowledge elements that were not in these categories were added as comments (e.g., private life, unknown visitors, backups, secure printing, data protection).

Introducing the Security Awareness Improvement Methods

All training methods were suitable for conveying general information security knowledge and none of the events were tailored to the participating organizations.

The research excluded highly specialized trainings (e.g., targeted workshops), online methods that were difficult to access or that required software development (e.g., mobile applications [apps], online games), time-consuming programs (e.g., gathering points), and measurement actions with educational goals (e.g., phishing simulations).

The investigated methods included personal training, online training, elearning, campaigns, escape rooms and board games.

Personal Training (Presentation)
Personal training was a traditional classroom presentation. It consisted of an animated presentation for five users participating in person. The event was not interactive, but participants could ask questions after the event. No memos were provided, and the employees did not take notes.

Online Training (Live Presentation)
Online training consisted of a live presentation over Microsoft Teams or another video conference application preferred by the organization. The content of the presentation was the same as the content used in the in-person training. Participants joined the call remotely, even from their home offices. The training was not recorded, and memos were not shared.

Elearning Material
Because this research was conducted at 10 different organizations, a general solution was needed for the delivery of elearning content. Some of the organizations involved did not have an elearning system, while others used a variety of different training systems. For the research, the training presentation was modified to create a detailed, automated program that could be learned alone. The file was shared via Microsoft OneDrive. This method reduced the risk that some users might not be able to use an unfamiliar system to access the material. It is not uncommon for elearning solutions to import training material from presentations.

Campaign Elements
Security awareness campaigns are usually organized during Cybersecurity Month, which is October in Hungary and in the United States, too. During that time, employees typically hear interesting presentations, participate in programs, see posters and news with important messages, and sometimes receive small useful gifts to remind them of the importance of security awareness. Information security campaigns are complex solutions, sometimes covering other training methods. For this research, the focus was limited to classic campaign elements. The campaign package contained:

  • Posters with important messages
  • A security newsletter/intranet post (printed)
  • A security-related crossword puzzle
  • A memory game (printed)
  • A camera cover
  • A keychain with a password encoder

Escape Rooms
Security awareness escape rooms are not unusual gamification elements of information security trainings. Escape rooms are games that require teamwork, and the aim is to mitigate risk based on human factors by highlighting general user deficiencies and bad habits in information security (e.g., simple or written passwords, keys in a pencil box). These games demonstrate why it is important to know and adhere to security rules and illustrate how easy it is to fall victim to human-based attacks if users are not security conscious. Escape rooms use gamification and the methodology of experiential learning to improve the level of participants’ security awareness by pointing out common mistakes and unsafe habits, their possible consequences, and the advantages of security awareness.2 In the case of this research, an escape room with the same scenario was used at all 10 organizations to cover the necessary knowledge elements. There was no customization.

Board Games
Board games as training materials are new forms of gamification in several areas of education, but they have become popular. Tabletop games, puzzles and card games have also grown in popularity.3

The board game used in this research symbolizes a workday (timeline) of six employees (characters) in an office (game board). Playing the game helps participants identify and remember important security awareness rules and best practices (30 knowledge cards) and recognize threats based on human factors (150 action cards), reflecting real incidents and experiences derived from social engineering audits. Under normal circumstances, the game can be played for 60–90 minutes. For this research, gameplay was reduced to three rounds (approximately 30 minutes per game) and action cards were selected according to the previously defined 10 knowledge elements.

Effectiveness Results

The research ultimately generated 284 usable responses, which means that 94.67 percent of the respondents completed the questionnaire. The remaining participants did not fill out one or both of the first and second questionnaires. In terms of gender distribution, 52 percent of the respondents were female, and 48 percent were male. Eighty-four percent of the participants were internal employees. Eleven percent were middle-managers, and 3 percent were top managers of the organization. Two percent were external employees.

The first survey explored what security awareness knowledge the users had in general, and how well they had internalized it using the last question of questionnaire 1, which asks participants to write information security awareness knowledge elements (e.g., rules, best practices, advice) they would tell a new colleague or share with a family member or acquaintance.

As the tree map in figure 6 shows, the top-three known security awareness knowledge elements out of 10 predefined required knowledge elements were choosing and protecting passwords (64.44 percent); clean screen policy, using screen lock (39.08 percent) and knowledge regarding malware protection (35.21 percent). As shown, at the lowest level were shredding paper-based documents (perhaps based on home office use and higher level of digitalization), secure use of social media, and phone and smart devices, which are not always general parts of security awareness trainings.

Figure 6

After presentation of the program, responses to questionnaire 2 showed how the participants’ security awareness knowledge changed. Figure 7 shows the detailed results by topic.

Figure 7

As the in word cloud figure 8 illustrates, the most learned topic involved using a shredder and the secure use of social media sites, phones and smart devices.

Figure 8

The results indicate a close connection between previous knowledge and newly improved security awareness topics. The less the knowledge element was known previously, the more attention was paid to it.

Figure 9 shows how effective the program was overall based on which security awareness knowledge was improved.

Figure 9

It is interesting to see which program element improved the security awareness knowledge of the most participants. Figure 10 shows the percentage of participants for each education method who gained at least one new knowledge element (also recorded in questionnaire 2). Based on that information, if the goal is to increase security awareness knowledge of employees generally, it is recommended to choose classroom training, a board game or an escape room. These methods help sensitize users on information security, and participants can derive the most interesting and useful pieces of knowledge for their circumstances.

Figure 10

One month later, the research took into account how interesting a given method was to users. It did not filter out or remove from the assessment those who did not participate or did not pay attention to the individual programs. Lack of interest and attention in these programs were also important pieces of feedback. As figure 11 shows, classroom training was the most popular method (100 percent) and elearning the least popular.

Figure 11

In addition to investigating the percentage of more security aware users, the research evaluated the average number of new knowledge elements (figure 12). Board games were the most effective program element (1.51) and elearning was the least effective (1.07).

Figure 12

Figure 13 indicates how each of the training methods increased the number of security awareness knowledge elements.

Figure 13

What Employees Think About the Methods

In addition to rating the effectiveness of each educational method, evaluating user preferences could prove useful. First, the research examined how participation in the preferred training method affected the improvement of security awareness. As figure 14 shows, there is not a significant difference in effectiveness based on whether users took part in preferred or nonpreferred educational methods. The number of participants with more security awareness knowledge and the number of new knowledge elements gained is almost the same in both cases.

Figure 14

However, it is interesting to note how the preferences of the participants changed following each program presentation. As figure 15 shows, the popularity of online and traditional methods (i.e., online training, elearning, classroom training) decreased, while gamified program elements increased in popularity.

Figure 15

Expanding the data, it is clear that preference for the board game increased the most, and elearning decreased the most. It is unknown whether the preference changed because there was no prior preference, but affected participants liked the program and came to prefer it, or if affected users had a prior preference but still did not like it.

Questionnaire 2 asked for information about how enjoyable and useful participants found the program. According to figure 16, gamified methods were the most enjoyable events, and the board game was the most useful program.

Figure 16

Overall, 98 percent of participants in gamified events recommended the program they participated in, while elearning was the least recommended solution (figure 17).

Figure 17

Data analysis shows that those who found a program enjoyable became more security-aware than those who did not.

Conclusion

Within the framework of the research, the effectiveness of security awareness improvement programs was examined according to two aspects. The main goal was to learn which method could be used to improve the level of security awareness of most users. The research also examined which options improved security awareness and to what extent. The results confirmed the prior assumption that gamification methods can improve the level of security awareness and reach more employees.

Based on these results, it would be beneficial to consider three questions when planning a security awareness improvement action:

  1. What information security knowledge do employees currently have
  2. How security-aware are the users?
  3. Is the main goal of the program to reach out and sensitize more employees or to directly improve deficiencies?
Data analysis shows that those who found a program enjoyable became more security-aware than those who did not.

Based on the answers to these questions, an organization can choose an appropriate mix of security awareness improvement actions and apply different methods for different purposes. For example, to sensitize employees to security awareness and increase users’ security awareness knowledge, classroom trainings with interesting presentations or gamification elements such as board games or escape rooms are indicated.

It was interesting that when measured one month later, the online training was the most successful in both gaining new knowledge elements and increasing the number of more security aware users.

Any organization can benefit from conducting a similar investigation before planning and organizing a security awareness program. This research can be helpful when choosing the right methods to effectively improve the information security knowledge of employees.

Endnotes

1 Oroszi, E.; Időutazás a Social Engineering auditok korában, avagy mi változott az elmúlt 10 év alatt? (Time Travel in the Age of Social Engineering Audits—What Has Changed in the Last 10 Years?), ISACA 2022 Hungarian Chapter Conference, Hungary, 2022
2 Oroszi, E.; “Using Gamification to Improve the Security Awareness of Users—The Security Awareness Escape Room,” ISACA® Journal, vol. 4, 2020, http://wup.ozone-1.com/archives
3 Oroszi, E.; “Board Games as Security Awareness Improvement Tools,” SECURWARE 2021: The 15th International Conference on Emerging Security Information, Systems and Technologies, IARIA, Greece, 2021

ESZTER DIÁNA OROSZI | CISA, CRISC, CISM, ISO 27001 LA

Is lead consultant and head of the Information Security Consulting Department at a Hungarian information security consulting enterprise. She has 14 years of experience in the field of information security, with a special interest in human-based attacks, social engineering audits and security awareness improvement. She is a doctoral student at the National University of Public Services (Budapest, Hungary), and her research area is measuring and improving the security awareness level of users through gamification. As part of her research, she developed a security awareness board game, which was released in 2022.