Master's thesis: Social Engineering Awareness using Serious Games

Last year I have attained my master's degree in Computer Science majoring in IT-Security. It has been a very long journey, but in the end I passed with distinction. I have to admit, I am especially proud of my master's thesis, which was jointly written with my fellow-student Saed Alavi. We have accomplished something huge with SANIAG!

In the thesis, we have created and designed an actual board game named SANIAG - Saeds and Niklas' Awareness Game. With the board game we wanted to create and scientifically assess a totally new form of awareness measure. Our goal was to increase the awareness levels of the players in respect of social engineering attacks while having a high fun factor. Participants of the scientific study shouldn't be aware of participating in a training. This kind of games designed for a primary purpose other than pure entertainment is called serious games.

SANIAG: game situation

Abstract

Aim: The present study was conducted to investigate the impact of serious games in respect of raising peoples awareness for social engineering. Methods: In order to evaluate a potential learning effect caused by a serious game, we conducted a randomized controlled trial. The subjects (n=38) were randomly allocated to the experimental group or the control group. The awareness levels (social engineering awareness) were determined both at the onset and at the end of the study using a standardized online questionnaire. The experimental group has the intervention being assessed after completing the first questionnaire by playing the serious game SANIAG. The measured data were then quantified in an automated way negating the effects of random selection using a formula scoring. Results: Empirical data were assessed using the statistical test analysis of variance. A two-way ANOVA with repeated measures to evaluate the factors group (categories: experimental and control) and time (categories: pre and post intervention) showed significant interaction effects (F(1, 68) = 5.332; p = 0.0239). Then, we looked at the difference within the groups by conducting a one-way ANOVA. The experimental group achieved a very significant improvement in the pre-post comparison (p < 0.001). There was no statistically significant difference in the pre-post comparison of the control group (p = 0.586). The shown interaction effect manifests in an effect size of 0.27. This means a small effect according to Cohen. Interpretation: Our results suggests that serious games are a feasible approach to raise social engineering awareness.

Serious Game SANIAG

SANIAG let players slip into a role of either a hacker (offensive game playing) or a security consultant (defensive game playing) or both. The game mechanics are based on real life scenarios and social engineering attacks. The players can playfully learn about the terminology, psychological foundations, common attack vectors and their potential impact, effective countermeasures and their importance. We designed the game in way that allows the player to follow different strategies and generates competitivenes. It was important to us to have game elements and dynamics implemented for fun and excitement.

Besides the playing pieces (printed using a 3D-printer) and the board, we have created 271 individual game cards. Most of the cards have unique hand-drawn graphics and short informational texts for learning. The learning content is based on literatur from people like Christopher Hadnagy, Kevin Mitnick or Robert Cialdini and covers all six levels of the Bloom's taxonomy. We also put in knowledge and best practices, which we gathered during our job as IT-consultant.

SANIAG: game cards

The Study

The actual study needed tons of preparations before we were able to conduct the experiment sessions, which we underestimated a little bit. Because we were dealing with individuals and their personal data, we had to address the ethics committee in beforehand and also had to be compliance with data protections laws.

When choosing our study design, we took a lot of trouble ;) We choosed a typical randomized controlled trial. That means randomly allocating the subjects to the experimental group and control group (we developed a script for the group allocation) and having an actual control group, which can be compared against. The control was necessary because of our type of measure using standardized questionaires. By completing the questionnaires, the subjects could have gained knowledge, theoretically. The randomization was necessary, because we were acquiring candidates with different backgrounds and different initial awareness levels (e.g., employees and students) and we had to ensure, that there is balance in respect of the two groups from the beginning. The minimum sample size was determined using G*Power. What we have learned: candidate acquisition isn't fun. All the results we have obtained from the standardized questionnaires were processed and scores were calculated using a VBA script. The actual analysis using statistical methods (i.e., ANOVA) were performed using MS Excel.

The result of the study was significant and showed a positive impact on the awareness level of the experimental group. However, we described a few possible biases like the presence of the experimenters (Saed and me) and the barely sufficient sample size.

One feedback of one of our professors was, that we really 'considered everything' (from a scientific perspective) and even met the standards for a good scientific paper with our master's thesis, which is not considered normal. And this really reflects our goal: a high scientific standard and high validity of the study. And as we are thinking of making an actual product of SANIAG, we appreciate the lots of very positive feedback from the candidates participating in our study.