David George Henry Gill

Submitted in partial fulfilment of
the requirements of Leeds Beckett University
for the Degree of
MSc Computer Security

School of Built Environment, Engineering and Computing

Leeds Beckett University

August, 2020

---

Abstract

Phishing campaigns have become a lucrative way for cyber criminals to gain access and data from companies and individuals, providing a method of penetration that often bypasses technical gateways to users.  The scale of the problem is increasing which would suggest existing solutions are not working and that further research is required.

 

The problem is twofold and involves both a technical and a human level to the attack.  The vast majority of research is aimed at the technical side of the problem with solutions created and implemented that often do not reflect or take into account the human element.  This is exacerbated further by attacks targeting not just the larger companies but the SMEs supplying them; as their security is often lacking.  This paper therefore seeks to identify if providing phishing awareness training to staff within an SME can have a positive impact on their ability to spot a phishing campaign and therefore reduce the risk to the SME.  

This research was carried out on a case study basis with two simulated phishing campaigns interspersed with the delivery of training materials to the participants.  The staff were then surveyed to measure their perception of the training material and how confident they felt with regards to handling phishing campaigns and how aware they were of the problem.

This study commenced at the onset of the Coronavirus pandemic and lockdown and the design of the study was therefore amended to take into account the unique circumstances in an attempt to identify what impact it may have had on the study.

This research concludes that the ongoing provision of phishing awareness training in SMEs with regular surveys and simulated phishing campaigns would prove beneficial to those SMEs looking to improve their security posture.

---

---

Contents

1        Introduction        10

1.1        Aim        12

1.2        The objectives of the case study are: –        12

2        Literature Review        13

2.1        Size of the problem        13

2.2        Technical solutions        15

2.3        Human element        16

2.4        SMEs        18

2.5        Rationale        20

2.6        Professional Views        21

3        Methodology        24

3.1        Summary        24

3.2        Tools        26

3.3        Ethics        28

3.4        Participant selection        30

3.5        Researcher        31

3.6        Experimental setup        31

3.7        Sample Size & Response Rate        37

3.8        Evaluation of methods        38

4        Findings        40

4.1        First Simulated Phishing Campaign        40

4.2        Second Simulated Phishing Campaign        43

4.3        Significance of simulated campaigns        46

4.4        Survey results        47

5        Conclusion        54

5.1        Recommendation        57

5.2        Impact of coronavirus on study        58

5.3        FUTURE SUGGESTIONS        59

---

List of Figures

Figure 1: Growth of phishing attacks 2010-2016 (Gupta et al., 2018)        13

Figure 2: Cyber security incidents (2019a)        14

Figure 3: Internal communication during first simulated phishing campaign        42

Figure 4: Results of first simulated phishing campaigns        43

Figure 5: Internal communication during second phishing campaign        45

Figure 6: Results of second simulated phishing campaign        46

Figure 7: Survey Results – Importance of emails        48

Figure 8: Emails received per day        48

Figure 9: Survey Results – Training useful        49

Figure 10: Survey Results – Phishing Awareness        50

Figure 11: Survey Results – Phishing confidence        50

Figure 12: Survey Results – How carefully checked emails        51

---

List of Abbreviations

SPF – Sender Policy Framework (email authentication)

SME – Small to medium sized enterprise

GDPR – General Data Protection Regulations

KPI – Key Performance Indicator

---

Acknowledgements

Thalita Vergilio – Supervisor

My Wife, for proofreading.

---

---

1. Introduction

Phishing is the use of emails to try and trick individuals into releasing their personal information or in order to install software to provide further attack vectors against the victim.

Phishing can take several forms with generic high volume attacks seeking to find vulnerable systems and or individuals across a broad range of users.  More specific phishing emails aimed at a particular individual are also referred to as Spear Phishing or Whaling.  They are increasing in complexity and scope but often use fake links within the email or attachments containing malware in the guise of documents.  The fake links often lead to websites that look identical to the original and are therefore very difficult to identify as fake without closer inspection.

Phishing is becoming an increasingly fertile hunting ground for cyber criminals (2019a) providing them with multiple avenues into small and medium sized companies and their systems. It is an attack that relies on both technology and psychology in order to gain a foothold into an organisation to carry out further exploits (Tschakert & Ngamsuriyaroj, 2019) (Abbasi et al., 2012) (Zeydan et al., 2014) (Gupta et al., 2018) (McElwee et al., 2018).  The problem is of increasing size and complexity and shows no signs of abating (Gupta et al., 2018) (McElwee et al., 2018).

There is an abundance of existing research on the problem of phishing  with the vast majority of this focused entirely on creating technical solutions (Kumaraguru et al., 2010).  Some of this research has led to commercial products like PhishGuru (Kumaraguru et al., 2010).  While some of these technical solutions do focus somewhat on the psychological side of the problem the vast majority see the problem as a purely technical one. However, the increased sophistication of phishing attacks makes technical solutions difficult (Stembert et al., 2015), which is why the aim of this study is to evaluate if awareness training can provide SMEs with a method for reducing their vulnerability.

The vast body of research, even when covering the human element, is mainly focused on big organisations or government offices (Innab et al., 2018) (Chatchalermpun et al., 2020).  There is very little focus on those organisations further down the supply chain that feed into these government offices or corporations.  In fact the supply chain is frequently overlooked in terms of security protection and is thus becoming an active area for cyber criminals to target (Melnyk et al., 2019).  

The business used in this case study is an eCommerce company.  They process thousands of transactions each month and have a heavy reliance on digital communication.   With the increased fines introduced by GDPR there is an added urgency to ensure everything possible is done to reduce the risks from such campaigns and to mitigate the impact on the organisation if possible.  Part of this work is focused on identifying what steps can be taken to help members of staff identify and report potential phishing emails.

The nature of phishing email campaigns is such that it only takes one person in an organisation to fall into the trap of the email for it to be effective.  They therefore have a very low threshold to be considered a success from a cyber criminal’s point of view which could explain the high incidence and increasing volume of phishing attacks.  It is therefore an important area of research in order to identify methods to improve the current situation.

1. Aim

The aim of the project is to run a case study in a Small to Medium Sized Enterprise (SME) to investigate the effectiveness of awareness training towards the identification of phishing emails.  The results of the case study will be used to identify if training can lead to a reduction in the susceptibility of users to the risk of phishing attacks and to subsequently recommend if the training practice should be expanded to similar businesses.

2. The objectives of the case study are: –

•    To critically review the literature on phishing emails and current tools used to help increase awareness

•    To analyse the susceptibility to phishing email campaigns in an SME

•    To create and provide awareness training to staff with regards to phishing emails

•    To review the impact of awareness training on the susceptibility of staff to phishing emails

•    To make a recommendation on whether awareness training should be expanded to similar organisations.

This research looks at how to improve the security of the human element of the attack with training in a SME in order to fill the gaps highlighted above.  This can then be used as a reference point for similar organisations in deciding if awareness training is suitable for their organisation.

---

---

2. Literature Review

3. Size of the problem

Phishing (as previously defined in the introduction) is a recurring and increasingly problematic area of cyber security with an increasing number of incidents reported to the ICO (2020) and within the Proofpoint surveys (2018) showing that it is of growing concern and becoming ever more sophisticated in its delivery (Legg & Blackman, 2019); with phishing emails becoming ever more targeted and personal.

Figure 1: Growth of phishing attacks 2010-2016 (Gupta et al., 2018)

As the above graph (figure 1) highlights the number of phishing attacks is increasing showing no sign of abating with a 5753% growth between 2004 to 2016 (McElwee et al., 2018).  This is highlighted further  by recent ICO data (2020) showing the number of incidents reported to them involving phishing in the fourth quarter of 2019-20; 280 incidents representing 10% of all reported incidents and 42.9% (figure 2) of all cyber related incidents, making it a major source of cyber security attacks.  Some research has shown that thousands of phishing websites are created on a daily (Kaila & Nyman, 2018) basis making the problem ever more acute.

Figure 2: Cyber security incidents (2019a)

This expanding problem inevitably has an impact on organisations with research showing that “…some phishing attacks have convinced up to 5% of their recipients to provide sensitive information to spoofed websites.” (Dhamija et al., 2006). With further data showing that between 70% to 80% of people responded to an email from a friend’s spoofed email address (Dhamija et al., 2006) (Finn & Jakobsson, 2007).  In 2003 alone approximately 2 million people gave their information to spoofed websites (Dhamija et al., 2006).  This level of reported vulnerability amongst recipients of phishing campaigns confirms it is an important area for continuing research.

The scope of the phishing problem was further highlighted during the study by multiple stories in the press regarding the increase in the number of phishing attacks related to the coronavirus pandemic (Jolly, 2020) (Muncaster, 2020) (The Economic Times, 2020) although it was too soon to have any academic papers to verify if this was actually the case.

The increasing scope of the problem is further highlighted by research showing the expanding size of the internet user base and the corresponding increases in websites and phishing related incidents. (Gupta et al., 2018).  Some of the existing research suggests that the impact of continued phishing campaigns on online sales could restrict customers from making purchases (Kirlappos & Sasse, 2012); due to a reduction in trust with online platforms like online banking and eCommerce platforms (Aburrous et al., 2010).  This could have a major impact on companies like the SME used in this study as it completes the majority of its sales online.

2. Technical solutions

A review of research on protection against phishing campaigns returns an abundance of research focused on the technology (Mishra et al., 2018) (Megha et al., 2019) (Ayodele et al., 2012) (Yousif et al., 2019) (Sankhwar & Pandey, 2017) (Salem et al., 2010) (Zeydan et al., 2014) used to mitigate the risks. These tools tend to be very complex; often integrating directly into email servers and therefore requiring technical expertise to implement. Others supply extensions to browsers in order to help users identify phishing websites (Zeydan et al., 2014).  They are increasingly expensive and have very little data to support their effectiveness (Carella et al., 2017); with suppliers providing data biased towards their proprietary software. In fact recent research (Legg & Blackman, 2019) shows how cyber criminals are finding ways round these technical solutions with automatic detection often failing (Stembert et al., 2015) to identify the threats.  Phishing is after all a technical attack that looks to exploit human weakness (McElwee et al., 2018).  This human aspect is an important area of research as some users will often ignore warnings provided by software (Kumaraguru et al., 2010) and find the technical solutions hard to understand and use (Kumaraguru et al., 2010).  

The problem of defending against these types of attacks is that there is no silver bullet (Melnyk et al., 2019) to protect organisations due to the fact it is largely targeting the human factor (Aburrous et al., 2010).

3. Human element

Some of the existing research does discuss the human element but then spends a considerable amount of time covering the technical aspects of providing phishing protection through email servers; providing further evidence of the continued focus on the technical side of the solution (Stembert et al., 2015) (Gupta et al., 2018).   However, the vast majority of research largely ignores the human element of the vulnerability (Salem et al., 2010) which goes against the main vector of the attack; to compromise an individual’s email account via psychological manipulation. But this is not unexpected as most organisations are ignorant of the threat (Alazri, 2015). As “Human behaviour remains the real challenge in dealing with security threats” (Stembert et al., 2015) there is a need for further research to counter the human element of phishing attacks in order to keep both users and companies safe (2019b).  

One method of addressing the psychological side of the vulnerability is to provide awareness training to staff and there are several papers on the effectiveness of this approach regarding phishing protection (Carella et al., 2017) (Innab et al., 2018) (Kumaraguru et al., 2010). However, these studies focus mainly on Government organisations and larger corporations (Innab et al., 2018) (Tschakert & Ngamsuriyaroj, 2019); probably due to the ease of access and the potential damage that can be caused in these larger organisations.

Even where the human factor was a major consideration in the existing research it was carried out in lab conditions that were not indicative of real world situations where the user would be under far more cognitive load in terms of their day to day functions (Kumaraguru et al., 2010). However, it is acknowledged this is possibly for ethical reasons as quantitative experimental methods would be more transparent to the user and therefore potentially negatively impact the results (Finn & Jakobsson, 2007).  There are ethical considerations to be taken into account whenever there is research into phishing campaigns involving participants as this inevitably leads to an element of deception.

Some of the experimental studies focused on directing participants to review web pages that users are directed to from phishing campaigns without considering the content of the email that contained the link that got them there in the first place (Aburrous et al., 2010).  Without identifying how a user got to the website it is unclear how training on identifying a phishing website would benefit the user.  The website itself can inject malicious code on to the browser or trick the user into downloading malicious software disguised as a browser extension etc.  The problems with this approach is that it is taking the user out of their normal context and workload.  It also makes the participants more alert to the fact they are part of a study and they would not therefore act in a natural way.

Other research like that from Proofpoint, used surveys (2018); but some users will not want to admit they have fallen foul to a phishing campaign for psychological reasons. It is therefore possible that many such attacks go unreported and therefore calls into question the method of collecting data purely via surveys (Aburrous et al., 2010) (Kwak et al., 2020) (Finn & Jakobsson, 2007).

The existing research with regards to user education and phishing have not had the desired impact on the problem with data showing a continued increase in phishing incidents (Gupta et al., 2018).  What these studies do show is that the standard demographic variables do not apply to the susceptibility of individuals to phishing campaigns (Dhamija et al., 2006) with sex, age and computer experience and usage all proving to have little impact on the susceptibility of the user to a phishing campaign.  For example some research highlighted a particular case in the last US presidential campaign that highlighted how easy it is for anyone to fall a victim to this approach; although it is not clear what level of training the victim had received prior to the attack so it is difficult to draw any firm conclusions from the incident (Gupta et al., 2018).  It does however once again show that research is required to identify solutions to the problem.

4. SMEs

Where user education was a key part of the research the existing literature is predominantly dominated by research in large organisations or government offices (Innab et al., 2018) and largely ignores SMEs. However, a lot of SMEs provide niche services or products to bigger corporations and they therefore have now become an important element of security protection for those corporations (Melnyk et al., 2019). There have been recent examples of attacks at Target and Home Depot in the US that have both been caused by SMEs within their supply chain (Melnyk et al., 2019). This shows that attackers are seeking out the weakest link in the supply chain in order to compromise the larger organisation.  With the anticipated growth in the digital economy this is an important area to ensure trust is maintained (Melnyk et al., 2019) and to provide larger corporations with the security they need to continue trading with SMEs.

There is limited research on cybersecurity within SMEs and the studies that do exist look at cybersecurity in a holistic way, providing guidelines rather than covering specific areas like phishing (Kabanda et al., 2018).  What those holistic studies do show is that SMEs are constrained by various factors: Budget, Staffing, Management support and Attitudes.  Staff inside SMEs often take shortcuts in order to get work done which can circumvent security measures (Nycz et al., 2015).  This is often done when resources, knowledge and required skills are lacking or when they find cyber security tools difficult to understand and implement effectively (Kabanda et al., 2018). Which could explain why many SMEs use outsourcing services in order to help them cover their cyber security responsibilities (Nycz et al., 2015).  However, as previously noted (Gupta et al., 2018) phishing continues to be a growing problem for all organisations, including SMEs.

Attitude in particular is a problem as many SMEs do not take cybersecurity seriously at all (Kabanda et al., 2018) with some thinking they are too small to be the target for an attack but as pointed out in the Proofpoint Human Factor Report for 2019 “smaller companies may be more vulnerable due to relative lack of controls and awareness, both of which create lucrative potential outcomes for threat actors” (2019b).  Often SMEs only become concerned with cybersecurity when they have a problem (Nycz et al., 2015) with management often seeing cybersecurity spending as unnecessary (Melnyk et al., 2019). Other studies show that SMEs were the least likely to meet any new compliance requirements (Melnyk et al., 2019).   This leads to the conclusion that SMEs can be the weakest link in security for larger companies that use them (Melnyk et al., 2019) and therefore an important potential attack vector for cyber criminals.  SMEs also struggle to find talent with the required skills and knowledge to help them with cyber security issues because of their size and the fact people are not willing to work in that area of the economy (Globe & Mail (Toronto, Canada), 2019). They also struggle to afford specialist cyber security staff which forces them to use existing staff in a part time manner and without the resources required. They then have to scramble to protect themselves (Globe & Mail (Toronto, Canada), 2019) when things go wrong.

However, SMEs could use cyber security as a marketing advantage when dealing with larger organisations.  This could be done by them taking the matter seriously and having a professional approach therefore making them more attractive to larger partners (Lloyd, 2020).

1. GDPR

Cyber security issues within SMEs are further compounded by the introduction of GDPR which has made it a legal requirement for small and medium size enterprises to protect themselves against potential data breaches and to report them within 72 hours.  This can cause SMEs major problems, especially when the potential fines are taken into account.  GDPR regulations (Queen’s Printer of Acts of Parliament, n.d.) make data controllers responsible for their own compliance as well as that of any third Party processors, which will make more organisations insist that their suppliers are compliant.  But the lack of a professional approach to cyber security in SMEs was highlighted by how few were prepared for the introduction of the GDPR regulations (International Financial Law Review, 2018) just 100 days before the regulations were due to come into force.

5. Rationale

In contrast to much of the existing research the aim of this research is to assess the impact of phishing awareness training on SMEs. These organisations tend to be too small to justify the expense of a technological solution without considerable justification of its effectiveness, which according to research is lacking (Carella et al., 2017). They may also lack the technical expertise required to implement many of the technical solutions available.  Which highlights the need for additional research in protection against phishing campaigns and how important it is to find a solution considering the increasing size of the problem and the potential vulnerabilities in SMEs and their position in the supply chain.  So, it is difficult for an SME to decide their best option in phishing protection with the existing research largely covering technical solutions, larger companies and uncommon contexts, i.e. laboratory-based experiments.  There is therefore a need to carry out this research to help them make an informed decision on providing their staff with phishing awareness training.

The case study for this research is to be run in an eCommerce organisation which relies heavily on email for communication with suppliers and occasionally customers.  This heavy reliance on email communication makes them particularly vulnerable to phishing campaigns but more so in the coronavirus pandemic (Jolly, 2020) (Muncaster, 2020) (The Economic Times, 2020); where online communication is heavily relied on to continue operations.  

6. Professional Views

There is an argument in professional security circles that awareness training can be counterproductive.  The Phishprotection website blog asks the question if awareness training can do more damage than good (#, 2018). Which is also backed up by the National Cyber Security Centre Blog post – “The trouble with phishing” (R, 2018). However, neither of these blog posts back up their opinion with any evidence for their arguments so it is difficult to identify if this is a legitimate concern.  A story from defensesystems.com highlights some of the potential dangers of simulated phishing campaigns; recounting how a simulated attack based on US Army pension contributions caused major concerns (Cheng, 2014) for the participants.  This provides some guidance on the type of simulation to avoid and also covers some of the concerns raised by the previously mentioned blog posts.  It also raises the ethical question on the risk of psychological damage to participants during a simulated campaign.

Some security professionals advocate the use of continuous context based awareness training (Oster & Tucker, 2019) with a counter argument that if this is used incorrectly it can simply become a vanity metric showing how successful the training has been without actually affecting internal company policy and procedures (Oster & Tucker, 2019).  Although previous research shows that users can be trained to protect themselves from phishing campaigns (Kumaraguru et al., 2010) expert opinion identifies that this passes the burden of protection to the user and not to the systems they are using.

There is also the concern in professional circles that the effectiveness of training could impact on the user’s day to day work if they were to suspect too many emails and therefore raise too many false positives (Canfield & Fischhoff, 2018).  The suggested solution being regular training to give more experience but less potential to raise false positives. This argument is also backed up on the tech.newstatesman website which argues that simulated phishing campaigns can stop staff from answering legitimate emails and therefore impacts productivity (NS Tech, 2017).  The post also specifically argues against the use of a naming and shaming strategy as part of the simulation. Which would certainly be expected to have a negative psychological impact on participants and potentially cause them to withdraw from any research.

This case study is not trying to identify if awareness training on its own can solve the problem but attempting to validate if it is worthy of inclusion in a program of defence against such threats.  The opinion of professionals is again split with some suggesting a balance of technology and training is the correct way to go (Mann, 2008).  The aim of the training should be to provide participants with awareness of the threat of phishing campaigns not only to their work environment but also to their personal life.  It is felt that making the problem more relatable to the personal side of their life will make it more effective.  With the suggestion that it should avoid technical terms and involve all levels of the business, providing information on the systems, policies and procedures in use to help support the users (Mann, 2008).  This is backed up by research that shows demographics have no bearing on susceptibility; with even computer usage habits making little difference (Aburrous et al., 2010).  It is felt by some professionals that the most effective way of providing the training is to provide face to face briefing with “content [that] is interesting and relevant and also that that delivery is entertaining, and critically, ‘memorable’” (Mann, 2008).  Placing this training in context provides better opportunities for participants to relate and validates the use of phishing simulations as a method of training (Mann, 2008).

The limited research on awareness training has shown that the combination of simulated phishing campaigns with training can have an impact on the susceptibility of users to phishing attacks (McElwee et al., 2018).  With alternative methods of training like regular email bulletins, incentives and management messaging shown to be ineffective (McElwee et al., 2018).

The purpose of this case study was to analyse whether awareness training would be an effective part of a strategy based on the premise that technical tools can only provide a partial solution, at a price and with external help required.  The conclusion therefore is that this case study, limited to the organisation in question with simulated phishing campaigns and a subject survey can produce the required data in the correct context to enable similar organisations to make a decision on their approach going forward.

---

---

3. Methodology

4. Summary

This research attempts to analyse if the provision of phishing awareness training to a group of individuals within a limited case study will reduce their susceptibility to phishing campaigns and raise their awareness of phishing emails.  The approach taken was to use observation of participants during simulated phishing campaigns with a final survey to obtain the participants’ perception of their awareness and confidence dealing with phishing emails.  

In order to measure the effectiveness of the training a baseline observation was carried out using a simulated phishing campaign.  This experiment was carried out in order to obtain quantitative data to get a more objective view of the results (Bell Judith, Waters Stephen, 2018) and to provide a comparable baseline for a later simulated phishing campaign.  This was the first part of a short (two experiments) longitudinal study that surrounded the provision of phishing awareness training.  The experiments were constructed to record the number of participants who clicked on links contained in the simulated phishing emails from the experiments.  The results were then compared to see if the percentage of those that did click on the links in the experiments changed after the training.  

The training was provided as shared online materials sent to the participants for them to review in their own time.  Although the initial case study design was to provide face to face training, the coronavirus pandemic and subsequent lockdown made that impossible.

The training and simulations were then followed up with a short qualitative survey to measure the confidence and awareness participants felt in dealing with a real phishing campaign after they had received training. The survey was required in order to validate the effectiveness of the training on the participants’ perception of phishing campaigns and their awareness of the dangers and their subsequent behaviours. It was also difficult to assess the effectiveness of training on a purely quantitative basis  (Bell Judith, Waters Stephen, 2018) using just the simulated phishing experiments.  The use of two different experiments pre and post training combined with a survey also provides a method of triangulating the results against the participants’ perception of their own susceptibility to phishing campaigns compared to the results of the simulated campaigns.

Most of the previous research in measuring the effectiveness of phishing awareness training take a laboratory approach (Kumaraguru et al., 2010) in order to measure its effectiveness.  However, experimental studies do not take into account the ability of participants to respond while under the normal cognitive load of their daily jobs.  They also know that they are the subject of a study which could significantly alter their behaviour and therefore the results of the study (Finn & Jakobsson, 2007).

This research attempted to answer the practical question of whether it is worth investing in the provision of phishing awareness training for SMEs.  The use of a mixed method approach covers the need to observe participants and obtain data on the number of participants who were caught by phishing campaigns, but also to analyse how well participants thought they could deal with phishing campaigns.  Some research highlights that how users state they respond to phishing campaigns in surveys does not reflect the reality (Kwak et al., 2020) (Aburrous et al., 2010) (Finn & Jakobsson, 2007).  The research therefore used a mixed methods approach embedded within a case study.  The case study approach was selected on the basis that this research is exploratory and case studies are well suited to this approach (Yin, Robert K, 2018).  It was also felt to be the appropriate method to use in the context of a very common scenario of working within an SME; as previously noted in the literature review this an under researched area in terms of cybersecurity and phishing in particular with a lack of case study research based on SMEs (Melnyk et al., 2019)

The particular SME was selected as their parent company had reported a potential data breach caused by a phishing campaign.  The SME is an expanding ecommerce company that makes extensive use of digital communication in order to operate; with multiple locations across the country, but with the head office based local to the researcher.  The SME is split across different departments each with different responsibilities in a fairly standard structure.  Access to the SME was also convenient as at the start of the research the researcher was employed as technical lead at the company with responsibility for cyber security.  

2. Tools

There are several tools available, both commercial and free, that allow their users to run phishing campaigns.  Before the commencement of the pilot study some of the free tools were evaluated to identify if they would provide the required tools and data to carry out the simulated phishing campaigns. A lot of the tools shared similar functionality of allowing you to send phishing emails but also provided the registering of domains for the email addresses and websites.  The evaluation was done on the basis of measuring if any of the tools would improve how quickly the phishing websites could be implemented.

2. Speedphishing framework

This tool is available within the Kali Linux distribution (n.d.) and also available on github as a standalone Python framework (Compton, 2020).  At the time of the evaluation this looked like a promising tool as it could be run from a live booting disc of Kali Linux.  It also looked fairly easy to use but had very little documentation to identify how best to implement it for the purpose of the phishing campaigns and was unclear what data would be recorded and available.

3. Kingphiser

This tool is available via github (cyberconsultant3199, 2020). It also provided links to a wiki to show how it could be used.  The tool looked very technical to implement and the documentation assumed an existing level of technical knowledge of phishing.  The functionality it offered did seem to be comprehensive.

4. GoPhish

This tool is available from the GoPhish.com website (n.d.). The installation looked very technical although a guide was available elsewhere (Lamb, 2019).  Although the functionality looked ideal for the test, the level of technical detail provided in the documentation was very light and it was unclear what data would be stored during the campaigns.  

5. Phishing Frenzy

This looked a very promising tool but the installation and code was using Ruby which the researcher was unfamiliar with and proved problematic.  The tool is available on github (pentestgeek, 2020) and does have links to documentation (n.d.).  The added functionality provided through its analytics and tracking tools would have been very useful but would have stored data on the participants that would have contravened the participant agreement.

6. Apache web server

The researcher has experience of web development and provision of web servers using Apache so this was also considered as a possible tool for implementing the phishing websites part of the simulations.

A lot of the tools evaluated seemed very good and would have provided more functionality and data to the researcher but lacked the documentation required to fully implement for the research.  This would have taken time to learn, configure and implement.  There was also a concern with some of the tools with regards to what data they stored against participants and how this would fit in with the anonymity of the study.  The researcher therefore decided to implement the phishing website using Apache web server due to previous knowledge and experience and the limited amount of time to implement the research.

7. Domain name and email addresses

The domain name used in the simulated phishing campaigns was registered with LCN.com who provide domains hosting and webmail against those registered domain names (LCN.com, n.d.).  This allowed the researcher to create the phishing domain, attach two email addresses to that domain and send all of the phishing campaigns using their webmail facility.

3. Ethics

This research used phishing campaigns as a simulation in order to observe how participants would respond to them in a naturalistic manner.  Any phishing campaign, whether simulated or not, involves deception.  This runs counter to academic guidelines (n.d.); but in order to ensure participants were not able to distinguish the simulated campaigns from real ones the deception used was essential to the design of the case study. If a participant was informed of the study it would potentially alter the results (Finn & Jakobsson, 2007).  The use of deception can also be justified on the basis that the research is covering an under researched area with regards to phishing within SMEs.  There is no substitute for falling victim to a phishing attack and the educational value of falling for a simulated phishing campaign rather than a real one can have value (Baillon et al., 2019).

The university research guidelines (n.d.) quite clearly state that no research should be carried out without the subjects’ prior knowledge and consent; without giving them the opportunity to withdraw.  However, as previously stated the case study needed to observe the natural reaction to simulated phishing email campaigns.  Consent was therefore requested from all participants following the first simulated phishing campaign.  Participants were advised of the first simulation and given the opportunity to continue and advised of the forthcoming second campaign and subsequent survey.  

Due to the inherent deception within the case study it was important to minimise the potential risk to the participants.  One of the potential risks identified prior to the study was the compromise any of the participants’ standing within the organisation (Finn & Jakobsson, 2007) if they were perceived to be susceptible to phishing campaigns.  The case study design was therefore set up to not record any information relating to an individual’s performance within the experimental campaigns but instead recorded a simple count of IP addresses that had actively visited web pages that formed part of the simulation.  It was therefore not possible to identify who had responded to the campaigns and all data collected related only to the number of users who responded and not to any individual email addresses of those respondents.  This removed the possibility of any individual subject being identified by the researcher within the study or by any other participants.  

8. Personal Data Collected

In order to carry out the research each participant had their email address recorded to identify if their email address had bounced following the first phishing campaign. This was later extended to record if they had received the consent forms and how they had responded. This data was stored in an excel spreadsheet and deleted once the second simulated phishing campaign had concluded.

The only other potentially personally identifiable information recorded was the IP address of the user when they clicked on the links in the simulated phishing campaigns.  This was recorded in server logs and appears in appendices 4 and 5 with the IP addresses removed to provide anonymity.  At the time the case study was designed the SME was operating from 5 different locations. With the onset of the coronavirus lockdown most of these locations were closed and the participants moved to working from home.  The original assumption that most users would access their email from their respective work premises would have made individual identification incredibly difficult.  However, with the move of most participants working from home it was identified that IP addresses would have to be anonymised.  Therefore, the server logs used to collect the observed data were permanently destroyed when the cloud-based servers they run on were decommissioned a month after the final phishing campaign.  The data used to calculate the click through rate was anonymised before extraction from the server and appears in appendices 4 and 5 respectively.

This research was approved through the standard University ethical approval process and the approval email can be found in appendix 7. As well as the University approval the case study also required permission from the senior management team in the SME to carry out the research.  This was sought and obtained directly from the Managing Director before the project commenced and recorded in appendix 8.  All participants were sent a consent form which also advised them that all data collected would be anonymous and no other participants would be able to identify their performance within the simulated phishing campaigns.

4. Participant selection

Previous research has shown that demographics have very little impact on phishing susceptibility (Aburrous et al., 2010) so as broad a range of users was required to ensure that all groups and areas of the SME were covered and to obtain as many participants as possible.  Permission was therefore requested and obtained from the organisation to obtain all email addresses currently in use (Appendix 8 Email permission).  

A total of 64 individual email addresses were selected for the initial phishing simulation which represented all staff within the SME with an email address at the start of the study.  However, once the first simulated campaign was started 8 of these email addresses bounced with the message that they no longer existed, so the participant number dropped to 56.  Further information identified that 2 of the email accounts were set up as group accounts and it would therefore be impossible to obtain consent to use further.  From the original provision of the email addresses and the first simulated campaigns some participants left the organisation which left 48 potential participants for the case study.  The same group of participants were then asked to further participate in the study following the first simulated phishing campaign.  This involved the obtaining of consent forms which reduced the number of participants to 19.  However, this was still from a range of business areas within the SME.

5. Researcher

The researcher was involved in the setup of the web server, email accounts and domains names required to carry out the simulations.  They also sent emails to participants for the phishing experiments and provided the training material.  However, the original plan of doing this face to face had to be amended following the Coronavirus pandemic.  This was therefore done online again with the researcher providing the material.  

6. Experimental setup

In order to carry out the collection of data via observation of participants during the phishing simulation a new domain name had to be registered.  This was registered with a domain name very similar to that of the SMEs parent organisation; it was the same domain name with the ‘.com’ replaced with ‘.co.uk’.  Two email accounts were set up on this domain to send the emails to the participants for the two campaigns.  One was the name of the previous managing director who had recently moved to the parent company. This was done in order to make it more of a spear phishing attempt and more believable (Dhamija et al., 2006) to the participants.  The second was set up as a generic user account with a generic first name that was common in the SME and the parent company.  

A web server was set up using a cloud provider (Digital Ocean) with the server version of Ubuntu.  Apache web server was installed on the server and configured to serve static HTML pages.  DNS settings were configured on Digital Ocean which pointed the previously registered domain name to the new server.  

9. Pilot Study

Due to the technical nature of the simulations a pilot study involving the sending of an email from one of the faked email addresses to the company email domain (an email address of a former employee was provided by the SME) was carried out. This highlighted issues with the emails going directly into the junk folder.  An SPF record (Görling, 2007) was therefore added to the domain record for the sending email accounts and a second test email sent, this again hit the junk email folder.  Further research identified the possibility of whitelisting the sending email domain on the company’s Mail server.  After obtaining the consent of the SME a request was sent to the IT provider of the SME in order to implement this and once this was completed a further email was sent to test the receipt.  This attempt reached the inbox of the target email domain and therefore provided enough data to confirm that the simulated phishing campaigns could continue.

The second part of the pilot study was to check the web server and DNS settings and to identify how easy it would be to analyse the web server logs to obtain the data required for the phishing simulations.  This identified that the logs not only contained data related to the pilot study links but also contained requests from automated systems and bots looking for web sites to crawl at the index page of a domain.  The web page set up for the simulated campaign was therefore amended to move it away from the index page and give it a unique name so that only those clicking the link in the emails would go to that particular page and thus removing the spurious bot data.

Once the pilot study phishing email was sent the apache log on the server was traced using a tail command combined with a grep (‘tail -f | grep phishing.html’) for the particular page to identify just the page the phishing campaign linked to.  This provided the data required for the experiment and highlighted those that had clicked on the link in the email.  It was identified at this stage that it was possible that the same IP address could show in the logs if participants visited the page more than once or if there were multiple participants in the same location. This led to the data being compared by IP address and also the user agent (web browser) being used.  A manual process was established to remove the second record to remove any possibility of double counting.  However, it should be noted that with the majority of participants working from home the IP addresses were all unique in the logs.

10. First phishing simulation

The first simulated phishing email sent was a message asking the participants to click on a link to review the contents of a website being built for the SME by the parent company (appendix 2).  This contained a link to a website on the same domain as the sending email address.  The subject of the email was selected on the basis of research identifying the most effective phishing campaigns like lost items, shared documents, confidential documents, evacuation plans and requests to link on social media sites (2018).  The link in the email sent the users to the web server controlled by the researcher and detailed above.  The campaign was configured to serve a static content page written in HTML and CSS. The page was designed to be a partial debrief of the research and gave details that they had been sent there due to a phishing experiment; providing links to phishing education sites and some hints and tips on avoiding phishing campaigns.   This was in order to provide additional training if participants clicked through to the web pages; which has been shown to be an effective way of training participants in phishing awareness (Kumaraguru et al., 2010).

11. Second phishing campaign

This was setup in the same technical manner as the first campaign with the same static web page on the same web server being used as the landing page for the campaign.  On this occasion the email was sharing a document link to new procedures in place for the sharing of personal data (Appendix 4).  It was sent using a generic user name in order to avoid the use of the same email as the first campaign.

The campaign was carried out in a similar manner as the first with the same methods used to identify the IP address of participants clicking on the link in the campaign email.  The length of the delay was designed on the basis of previous research (Kumaraguru et al., 2010) with similar experimental setup having a delay of 1 week.  However, this research wanted to measure the effectiveness over a longer more realistic/common frame of time.  The original design was intended to run the second campaign within 4 weeks of the first.  The plan was subsequently changed due to issues caused by the coronavirus pandemic and lockdown which made it difficult to obtain the consent forms required to carry out the rest of the research.  There was therefore a 2-week delay between the first simulated campaign and the training and then a further delay of 4 weeks until the second simulated campaign.

12. Training

Due to the coronavirus pandemic plans were changed with regards to the implementation of training.  This was originally scheduled to be carried out in face to face sessions within the SME offices itself and to also provide a debrief to participants with regards to the first simulated campaign.  At this stage the participants were to be told of the nature of the research (Baillon et al., 2019).  This was then changed to create resources that could be sent through to participants.  This included the presentation that was going to form the basis of the face to face training sessions,   an interactive page from google on training to avoid phishing scams (n.d.) and the NCSC stay safe online training material (n.d.).  Although there is a concern that this is not as effective as face to face sessions and online materials are more difficult to validate if they have been accessed.

The online training material was identified on the basis of areas that previous research identified as at-risk behaviour; which included the following guidelines for avoiding phishing emails.

1. Never click on links in emails and type the address into the URL bar of the browser
2. Reading the message in a preview pane without opening it
3. Check the displayed name of the sender
4. Never trust phone numbers in emails
5. Checking for spelling errors – phishing emails are more likely to contain spelling and grammatical errors
6. Analysing how the email is addressed and what salutation is used
7. Analysing attachments – If possible with a virus/malware scanner
8. Never respond to emailed requests for personal information
9. Use of the terms urgent or danger in the subject matter
10.  Be suspicious for websites that ask for too much information

(Kumaraguru et al., 2010) (Scheau et al., 2016)

13. Participant observation

In the original case study design it was planned to do observation of the participants with regards to their interaction with the phishing campaign email and the website.  No other form of participant observation was planned during the study, but the staff within the SME communicate considerably via SLACK.  This form of informal internal communication increased considerably during the coronavirus pandemic and subsequent lockdown.  This led to one of the participants who had received the phishing email putting a message on the general slack channel with reference to it being a phishing email.  This was also repeated in the second simulated phishing campaign.  The researcher was unable to identify any previous phishing research that used data obtained from informal messaging in this way and was therefore unable to validate how effective this had previously been.  The messages were forwarded directly to the researcher to provide evidence of the response to the simulated phishing campaign and have been anonymised and included in the findings.

14. Survey

The final set of data was obtained via a survey.  This was setup using the website SmartSurvey and was limited to fifteen questions.  The purpose of the survey was to identify if the user felt they were more able to cope with phishing campaigns, to measure if they felt the training was effective and to record how aware they were of the dangers of phishing campaigns.  This would be then used to triangulate against the difference between the first simulated phishing campaign and the second.

The original survey questions were designed to identify how much the participants were using email for their day to day work and also how prepared they felt with regards to any future phishing campaigns.    At the onset of the study the SME closed their offices due to the coronavirus pandemic and moved to a remote working model for the majority of staff.  Their warehouse facilities remained open which had some office space but the head office was closed.  There were also press reports of an increase in the number of coronavirus related phishing emails being distributed (Jolly, 2020) (Muncaster, 2020) (The Economic Times, 2020).  The survey design was therefore amended in an attempt to factor in the possible impact of the office closures on the outcome of the study by asking specific questions.  These were aimed at identifying if the participants felt their workload had increased and if they felt there had been an increase in the number of phishing emails sent to them.

Before the survey was sent to the participants it was tested on three factors.  The first was to check that the URL link approach to sending the surveys out worked. The second was to see how long it took to complete and the third to identify any logical or grammatical issues in the questions.  It was then corrected in preparation for the final send to the participants.  The final survey was sent to all participants who returned their consent form.  This was 19 participants from the initial list of 48 email addresses.

The details of the survey questions appear in Appendix 1.

7. Sample Size & Response Rate

8. Methods of analysis

For the simulated phishing campaigns data was stored and calculated using basic Excel spreadsheet functionality.  For both campaigns the number of emails sent was compared to the number of unique clicks on the link to the phishing website based on the IP address and the user agent (browser) being used.

For both phishing experiments the number of emails sent was recorded manually after sending the email.  This number was then reduced on the first phishing campaign following the receipt of bounce emails showing that the email account was no longer reachable and further confirmation from the SME of those addresses that were no longer in use or those email addresses that were being sent through to a group of individuals.  

This led to a participant group of 48 for the first simulated phishing campaign.  All of the participants were then sent consent forms following the first simulation, 19 of which consented to their continued participation on the research.  13 of the 19 consenting participants completed the survey sent through to them following the second simulated campaign.

 

The survey results were exported from the SmartSurvey website as an Excel spreadsheet and analysed in Excel using basic spreadsheet functionality.  The summary data from the survey was placed as a table in the spreadsheet and used to create graphs for inclusion in the final report.  The individual responses were exported from the survey website and placed in the same spreadsheet and used to identify any patterns in the responses to the questions.

The raw data from the survey can be found in appendix 6.

8. Evaluation of methods

The methods used were able to obtain the required data to evaluate and analyse the effectiveness of the training. However, there was originally no consideration given to the observation of internal communication via slack, but this became a valuable way of monitoring the response to the phishing campaign.

It would have been possible to set up an experiment asking participants to identify phishing emails in a training environment.  However, as stated previously this would have taken them away from their normal work environment and given them notice on what was happening (Finn & Jakobsson, 2007).  Which would have impacted on the validity of the results.  The purpose of the study was to identify the response to phishing campaigns in as normal a working environment as possible.

Doing purely quantitative experiments in a training or lab environment would also have been impractical due to the coronavirus pandemic and lockdown.

It has been shown that for psychological reasons participants in phishing surveys are not always honest; due to the possible psychological harm that could be caused by admitting vulnerability to phishing campaigns (Finn & Jakobsson, 2007).  There is then the fact that not everyone will know they have been the victim of a phishing attack and therefore will not be able to report it in a survey (Baillon et al., 2019).  It was therefore thought it best to avoid a simple survey of users.  Plus using just a survey would only record the participants perception of their vulnerability to phishing campaigns and not their actual vulnerability.  For similar reasons interview and focus groups were not considered as this may have caused more discomfort for the participants and would have been difficult considering the restrictions in place due to coronavirus.

An alternative approach would have been to run simulated phishing campaigns in a longitudinal study and observe any changes in the response rate to the simulations.  This would have provided relevant raw data but phishing attacks are a psychological and technical vulnerability and it was therefore felt necessary to identify how participants felt with regards to phishing campaigns as well as how they responded.

By using a mixed method case study approach with naturalistic observation of participants this research shows the impact of phishing awareness training in a common working environment which is an under researched area of computer security.

---

---

4. Findings

The purpose of this research was to identify if providing phishing awareness training inside an SME would produce a positive result on the impact of phishing emails received.  Data was collected via two simulated phishing campaigns, a survey and some unplanned observations of behaviour during those simulated campaigns.

9. First Simulated Phishing Campaign

The first simulated phishing campaign was sent to the provided list of 64 email addresses within the SME.  The list was provided by the SME and no validation or verification was done against those email addresses prior to the first campaign.

The emails were sent in 4 batches on the morning of the 6th April from 7:48am to 8:04am.  A Monday was selected because previous experience of the researcher working in the organisation showed that colleagues were more likely to open and respond to messages on a Monday morning.  The email was not personally addressed to the recipients but was addressed to all and was very specific to the organisation.  A copy of this email can be found as appendix 2.  Although it cannot be considered to be a Spear phishing campaign as it was not targeting an individual user it was targeted at the SME in particular.

From 7:48am until 17:00pm of that day the web server logs were monitored for any traffic coming through to the phishing campaign page.  This was done by running a tail command on the corresponding apache log and looking for relevant entries.  The email account that was used to send the emails was also monitored to identify any replies and any email bounces sent back.  This was done to verify if the email accounts  being used for the simulation were valid and still active.

By 8:11am there had been 4 unique visits to the site (unique visits were determined by the IP address of the client and the user agent used to visit the site) and 8 emails bounced back.  By the end of that day the logs (appendix 4) were showing 17 unique site visits to the phishing campaign page.  This gives a response rate of 30.36%. This corresponds with previous experimental research which found rates between 30-40% (Aburrous et al., 2010) (Dhamija et al., 2006) (Kumaraguru et al., 2010).

Although the original design of the case study was not intended to include any observation of internal communication there was evidence within the SME of informal internal communication warning staff of the presence of the phishing campaign.  This was shared with the researcher by the Managing Director of the company and shows some pre-existing knowledge and awareness within the organisation of what to look for in a phishing campaign.

As you can see from the screen print below (figure 3) all users were warned of the email and also why the sender thought it was a phishing campaign.  This corresponds to a drop in the number of log records showing for the page on the server.  This also served as another piece of internal and unplanned training to the participants.

Figure 3: Internal communication during first simulated phishing campaign

There is no conclusive evidence of the impact of this internal message (figure 3) as it was not possible to identify how many participants read the phishing email after this message. However as can be seen from the server logs for that day (appendix 4) there were no subsequent visits to the page after the time of this message.

Figure 4: Results of first simulated phishing campaigns

The unique visits to the phishing campaign page represented 30.36% of the emails sent.  There was no recording of the number of emails that were opened so it is unknown what the percentage was against the number of open emails.  This can be seen more clearly in the above graph (figure 4).

The web server logs were monitored for the next four days but no further logs were recorded for the relevant phishing page.

10. Second Simulated Phishing Campaign

The second email campaign was sent to the 19 consenting participants on the morning of the 21st May from 8:45am to 8:51am.  The emails were once again sent in batches and were not personally addressed to individuals. A copy of this email can be found in appendix 3.  Although it was addressed from a similar account to the initial campaign it was designed to look like a shared file on Google drive with a link to a document detailing a new policy on sharing personal data.

The phishing campaign server and the sending email account were monitored in a similar way to the first campaign from the time of the first email to the end of the day.  By 9:26am there had been 4 unique visits to the phishing campaign website, but no subsequent visits followed.

Similar to the first simulated campaign there was informal internal communication highlighting the possibility of the email being a phishing campaign (figure 5).  

Figure 5: Internal communication during second phishing campaign

Again, there is no clear evidence of a link but there were no subsequent visits to the site after the time of the message in figure 5, as can be seen by the log data in appendix 5.

Figure 6: Results of second simulated phishing campaign

The 4 unique visitors represented 21.05% of the recipients sent the campaign email.  When compared to the first phishing campaign this represents a drop of 9.31%.  This can be seen more clearly in the above graph (figure 6).

Once again the web server logs were monitored for the next four days but no further logs were recorded for the phishing page.

11. Significance of simulated campaigns

The statistical significance of the two campaigns was calculated using functions within Excel.  This gave a Standard Error value of 0.1054 which means there is no statistical significance in the results of the simulated campaigns due to the small number of participants in the second simulated campaign.

12. Survey results

The survey was sent to all 19 of the consenting participants of the study.  13 of the 19 participants completed the survey.  The survey questions can be found in appendix 1 and the full results can be found in appendix 6.

The purpose of the survey was to triangulate the results obtained from the two simulated phishing campaigns and to obtain the participants’ perception on the training they received and their awareness of the dangers of phishing campaigns.  A secondary element to the survey was added to identify the influence of the Coronavirus pandemic on the participants during the research in order to identify any impact on the study.

The first set of questions were designed to identify how important emails were to the day to day work of the participants and what volumes they received.  This was relevant as the study was aimed at participants from all areas of the business and those with a higher volume of emails would have different levels of computer usage and experience.

Figure 7: Survey Results – Importance of emails

As can be seen from the above graph (figure 7) 45% of the participants reported that email was either very or extremely important to their role, with the same percentage identifying that they received over 30 emails per day.  Nobody reported that email was not important at all to their job.

Figure 8: Emails received per day

As can be seen in figure 8 there was a range of responses with regards to the number of emails received per day.  This question was included to identify what the normal load of emails was for each participant to check if the cognitive load with regards to emails received was excessive.  Using a mid range for each of the groups above (and 55 for the 50+ group) the mean average of emails received by respondents was 32.69.

Figure 9: Survey Results – Training useful

The question asking if the participants had received training reported 63% saying they had received training, with 50% stating this was very or extremely useful (figure 9).  One respondent reported that the training was not helpful at all but also responded that they had not received any training.  

 

The training was aimed at providing awareness of phishing and giving participants suggested ways of identifying phishing emails and tips to avoid being caught.  There were therefore questions designed to identify if this message had been received and was being followed; in other words to further validate if the training was effective by asking participants about their behaviour as well as how useful they perceived the training to be.  

Figure 10: Survey Results – Phishing Awareness

Figure 11: Survey Results – Phishing confidence

 91% of respondents stated they were aware of the dangers of phishing campaigns, with 36% of participants reporting they were confident of identifying a phishing email and 54% stating they were somewhat confident, the full responses are shown in the above charts (figures 10 and 11).

In terms of behaviour when receiving emails 54% of participants responded that they always or usually checked the email address from which it was received.  This was relevant to the study as the email address used was on both occasions a variant of the parent company address.  In terms of links in emails 90% of respondents reported that they clicked the links in emails rather than type in the address manually.

Figure 12: Survey Results – How carefully checked emails

The vast majority of respondents said that they checked their emails carefully with 61% saying they checked them extremely or very carefully (figure 12).

The section of questions specifically aimed at the coronavirus pandemic were designed to identify if the participants had been put under more workload due to the coronavirus pandemic, how secure they felt working from home and to identify if they had been exposed to more phishing campaigns as had been reported in various press articles (Jolly, 2020).    63% of respondents reported an increase in workload with 54% reporting a subsequent increase in email messages.

72% of respondents reported that they had received about the same level of phishing emails as usual and 81% reported that they did not feel more vulnerable to attack working from home.

The 6 respondents that thought their email was either extremely important or very important to their job also checked their email either carefully or extremely carefully.  They also reported that they were either extremely aware or very aware of the dangers of phishing emails. 5 of this 6 reported that they were either very or extremely confident that they could spot a phishing email. 5 of the 6 either always or usually check the email address of the sender and all of this cohort reported that they had received phishing awareness training.

The 1 respondent who replied that their emails were not so important to their job also reported that they did not check their emails carefully, they had not received any training and were not confident of spotting phishing emails or were aware of the dangers they contained.

All of the respondents (9) who reported that they received phishing awareness training felt they were either extremely or very aware of the dangers of phishing emails. 8 of them also reported that they checked their emails extremely or very carefully. 6 of the 9 reported that they felt the training was extremely or very helpful. 5 of them said they felt extremely or very confident of being able to spot a phishing email. 6 of them reported that they always or usually checked the email address of the person sending the email. 2 of them said they rarely did. 8 of them reported that they clicked the link in emails directly.

Of the 4 respondents who reported that they had not received any phishing awareness training 3 of them were aware of the dangers of phishing emails and the same 3 were confident that they could spot a phishing email.

There were 6 respondents who perceived emails to be somewhat important or not so important.  Two of these respondents still said they checked their email carefully.  Although this group also included the two respondents who said they checked their email ‘Not very carefully’. All but one of them reported that they were extremely or very aware of the dangers of phishing emails.  With one of them stating they were not at all aware.  This last respondent also said they had received no phishing email training at all and was all the only respondent reporting that email was not so important to their role. This participant also reported that they were not at all confident of being able to spot a phishing email.

Of the respondents who received 50+ emails per day all of them checked emails extremely carefully or very carefully, they all reported that they had received phishing awareness training. With 1 reporting it was extremely helpful and 3 stating it was very helpful and they were all extremely aware or very aware of the dangers of phishing emails. Their confidence in spotting phishing emails was 1 extremely confident, 2 very confident and 2 somewhat confident.  They all reported that they usually or always checked the sender of the email.

---

---

5. Conclusion

The simulated phishing campaigns were carried out either side of the provision of phishing awareness training.  It is important to point out once more that the participants were not aware of the first simulated phishing campaign.  

The first simulated phishing campaign resulted in 30.36% of those receiving the email following the link provided in the email and the second campaign resulted in 21.05% of participants following the link.  On the face of it this would suggest that the training produced a positive result in terms of phishing responses.  However, the initial campaign was with a larger number of participants and therefore any margin of error in the results would be reduced.  The percentages also look poor compared to a recent pilot study in Thailand (Chatchalermpun et al., 2020) which caught between 12% and 15% of participants.  Although that study was focused on financial services exclusively and highlights the possibility that susceptibility rates could be linked to organisational function rather than demographics.  Due to the small number of participants in the second simulated phishing campaign the results were not statistically significant and therefore no real conclusion can be drawn on if the awareness training had any impact on the participants ability to identify phishing campaigns.

On both simulated phishing campaigns there was some observation of internal communication via the company slack channels identifying the email as a phishing campaign.  It is notable that there were no subsequent participants clicking on the link in the email after this point.  This was not anticipated behaviour as the SME had no recognised procedures for handling phishing campaigns.  However, guidance from the Proofpoint’s surveys (2018) suggest this manner of reporting is very effective at increasing awareness within the organisation. Their informal procedure seems to be effective in supporting the business in reducing the impact of phishing campaigns, although it is not clear how this would operate with a more targeted spear phishing campaign. This also suggests that there was a pre-existing awareness of the dangers of phishing campaigns, possibly invalidating the survey question asking how aware the participants were of the dangers of phishing emails. Both of the simulated phishing campaigns had evidence of internal communication in the SME.  These highlighted to other members of staff within the SME that there were potential phishing emails being received.  This again shows pre-existing awareness of phishing campaigns, but also provided further training for participants.  One of the messages detailed why they thought the message was a phishing campaign.  It is arguable that these messages had more of a positive impact on the participants response to the simulations than the training provided.  This is highlighted by the fact that no participants clicked the links in the simulated phishing emails after the time of the message.  

63% of the respondents reported that they had received training on phishing awareness.  This either means the remainder did not see or open the training material sent to them.  This highlights the problem with sending the material to them to work on independently to actually having a face to face training session with all participants.

Previous studies (Kumaraguru et al., 2010) (Dhamija et al., 2006) ran simulated campaigns in laboratory conditions or with full knowledge of the participants and with a week gap between the first and second campaign.  This study used a gap of over a month to validate if training material would be retained for a longer period.  The results of the simulated campaign only show a small improvement, but the survey shows that participants felt they were aware of the dangers of phishing campaigns.

The survey reports show that 92% of respondents were extremely or very aware of the dangers of phishing emails.  This would suggest that the training was successful in raising the awareness of the dangers but as noted above does not account for any prior awareness.  This highlights a potential flaw in the research, suggesting an initial survey identifying awareness of the dangers of phishing may have been beneficial.  This would have provided a clearer indication of the success or otherwise of the training in terms of participants’ awareness.  

In terms of confidence and behaviours in dealing with phishing emails the survey results were not so positive.  This would suggest that the training succeeded in improving awareness but failed to provide participants with the skills required to handle phishing emails in a confident manner.  This is highlighted by both the survey and the simulated phishing campaigns.  Both phishing campaigns were based on similar looking email domains to the SMEs parent company and yet the survey data states that 38% of respondents rarely or sometimes check the email address of the sender.  This highlights that the participants require reinforced training to instil this habit.  These 38% could be the respondents who were responsible for clicking on the link in the second simulated phishing campaign but due to the design of the research it was not possible to verify.  The survey data and the response to the phishing campaigns were both anonymous and therefore it was not possible to collate data from individual participants.

Both campaigns also relied on the recipient clicking on a link in order to record a hit, and only 1 of 13 respondents said they typed in the URL address manually rather than following the link.  This represents a major issue; with phishing campaigns often triggered by following a link and highlights a further area for development of the training material in terms of changing participant behaviours.  However, it is acknowledged that users with a heavy workload would find it impractical to do this for each email they receive. The other alternative would be to implement on the mail server a method to highlight to the users that the email contains a link that is external to the company.  Although as previously stated in the literature review this may prove difficult for SMEs like the one used in the case study.  

The original case study design argued that informing the participants of the initial phishing campaign would have a possible detrimental impact on the viability of the results of the research.  However, the small drop of 9% in the number of participants caught in the second campaign would suggest that this is not necessarily the case.  With a sufficient enough gap between the original participatory consent form and the simulated campaign the data suggests the study would have been sufficient to run with users in their normal working conditions.  It is also possible that the 9% drop was a result of the participants being aware of the fact there was a second phishing campaign imminent.  This reduction in the rate of participants clicking on the links in the simulated campaigns could be due to the awareness training or the fact that the campaigns had taken place and generated internal communications which increased awareness.  This would suggest that a program of ongoing training would be beneficial to the participants in a large longitudinal study.

In conclusion the data would suggest that the awareness training had a positive impact on the participants with regards to their response to phishing emails, but more targeted training would be required to give users confidence in dealing with phishing emails with a particular emphasis on knowing which links they can follow and checking email addresses.  It would also be beneficial to extend any future study to a wider range of SMEs and participants to gather more data and responses.  

13. Recommendation

The aim was to make a recommendation to SMEs on the use of Phishing Awareness training in order to tackle the growing problem of phishing emails.  Although the data from the simulated campaigns is not definitive the data from the survey would suggest that providing the training improves overall awareness of the problem supporting previous research indicating that users can indeed be trained to protect themselves (Kumaraguru et al., 2010).  However, the data from the simulations does not necessarily provide a significant drop in the overall problem.  It is suggested therefore that any training be implemented with the use of a survey, as in the case study, that could be used to identify any shortcomings in the training and highlight areas for improvement.  This could be combined with a process of simulated phishing campaigns to provide the SME with more confidence in dealing with phishing attacks.  As previously shown (Dhamija et al., 2006) this training would be best addressed to all levels of the SME and all departments.  The training would be beneficial as an ongoing endeavour (Canfield & Fischhoff, 2018) and combined with policies and procedures that allow users to flag phishing campaigns to their colleagues and therefore avoid the issue of the simulated campaigns providing a vanity metric as a KPI (Oster & Tucker, 2019). Any policies and procedures introduced should allow the SME to monitor and review their continuing response to simulated and real phishing attacks. Another possible process to put in place is to embrace the informal communication highlighted in this study and provide members of staff with a facility to publicise suspected phishing emails in an open and transparent manner.  For the SME in this case study a specific SLACK channel would have provided sufficient coverage of the attack.  It is acknowledged and needs to be highlighted that the approach suggested here does not provide a ‘silver bullet’ in terms of phishing protection.  As long as phishing attacks succeed cybercriminals will continue to use them in order to obtain access to systems.  It is therefore recommended that this approach becomes part of a wider cyber security strategy that encompasses tools where and when appropriate.  This approach would also appear to be in accordance with much of the professional opinion with regards to protecting against phishing attacks.

14. Impact of coronavirus on study

The design of the case study had to change due to the impact of the coronavirus and this was reflected in the amended survey and training provision.  During the pandemic there had been numerous reports of the vast increase (Jolly, 2020)  (n.d.) (The Economic Times, 2020) (Muncaster, 2020) in the number of coronavirus related phishing campaigns.  In the survey one respondent reported a lot more and one reported some more, and 9 respondents reported they were about the same.  Although this is a very small survey it does not match up with this reported greater number of attacks.  The participants’ response to the coronavirus questions would therefore suggest they did not feel that additional phishing attacks caused them any concern during the pandemic.  

The survey results identified that 68% of the respondents believed they had more work to do because of the coronavirus and 15% believed they were receiving more phishing related emails.  This increased workload may have impacted on the simulated phishing campaigns in a detrimental manner (Kumaraguru et al., 2010) (Legg & Blackman, 2019).

However, coronavirus did mean that the training material was forwarded to participants for them to review in their own time, rather than delivered face to face.  With 9 of the respondents reporting that their work had increased during this period it is not known if they had engaged with the training material due to the increase in their workload.  It would have been beneficial to extend the survey to ask if they had engaged with the training material rather than a simple question on if they had received training.  It is possible with the extra work they reported in the survey that they did not fully engage with the material.  It is also unclear what difference, if any, it made to the outcome of the two simulated phishing campaigns.  

15. FUTURE SUGGESTIONS

Only two phishing campaigns were carried out in one SME separated by training.  In order to obtain a better understanding of the impact of awareness training on phishing campaigns a bigger longitudinal study would provide more data points to analyse the impact of the training.  An SME was selected for the study due to the paucity of existing literature in this area and their often-reported attitude towards security (Melnyk et al., 2019).  The SME in this study was eRetail based and it would therefore be beneficial to do an extended case study across multiple SMEs from multiple sectors of the economy to get a better picture of how training can impact these organisations and identify if security attitude has a bearing on the impact.  This could be identified by providing a survey to the SMEs at the onset of the study.  Another interesting area for future security research in SMEs is whether their attitude to cyber security is improved by being part of a study.  This could form part of a survey that would also identify their attitudes to security across more SMEs.  Such work could also attempt to incentivise SMEs to take cyber security more seriously in order to gain a competitive advantage (Lloyd, 2020) and appear more favourable to larger organisations.  Using a bigger study would also provide an opportunity to obtain a larger pool of data in order to identify if awareness training has a statistically significant impact on the outcome of the simulated phishing campaigns.

This study used the webpage in the phishing campaign as a means of providing further guidance to the user on avoiding phishing campaigns.  It would be useful for further research in this area to identify if this method would help to improve awareness and reduce susceptibility to phishing emails.  Although previous research in this area would suggest it is beneficial (Kumaraguru et al., 2010).  It also used two very similar variations of phishing emails in the simulations and it would be useful to extend this area of the research to include variations to identify if certain types of phishing emails were more productive; or if the context of the SME has a bearing on the success of different types of phishing emails (Baillon et al., 2019).

Although this study ran a phishing simulation without prior knowledge or consent it is acknowledged it would have been useful to survey the participants as the first stage of the study and also at the end to measure participant perception of their awareness and confidence.  The anonymous nature of the study also made it impossible to collate data from the phishing campaigns and the survey at an individual level.  Although this was done for ethical reasons, having the additional data it provided would have provided more insight into the impact of awareness training on individual participants.  An alternative would be to run the study with two groups of participants; one aware of the study and one group unaware to act as a control group.  Although this again raises the ethical question, it would be similar to the use of placebos in medicinal trials.

Previous phishing experiments (Aburrous et al., 2010) have reported that participants felt violated and were unhappy that they were not informed of the study (Finn & Jakobsson, 2007) in advance.  This could possibly be the cause of the drop in participants in this study from the original 48 to 19.  The survey in the study missed the opportunity to ask the existing participants how they felt when they found out about the initial phishing campaign.  In this study, however, the researcher was well known to the participants which may have had a positive impact.  This was evidenced during the simulated phishing campaigns when the researcher received personal messages from participants who had been caught by the phishing campaigns. These responses were positive:-

“Good catch on phishing email, on mobile looks so genuine..”

 “You phishing again?”

Although not possible for all future researchers it would appear having a professional connection with the participants prior to the study is beneficial. However, for security professionals working within or with the SME this could prove beneficial.

In both simulated campaigns there was evidence of internal communication that had not been considered in the design of the research. In both cases this appeared to impact on the results of the study.  Any future study could be extended to include the observation of internal communication in relation to the phishing campaigns, although it is unclear how this would be achieved.  Using a service like Sendgrid or Mailchimp would provide the researcher with data on when the emails were opened and also give data on what percentage of the phishing emails were read; a metric unavailable in this case study.  This would allow the researcher to correlate the opening of emails against the times of any internal communications linked to the phishing campaign.  It would also possibly allow future research to identify what percentage of users opened the email on mobile devices and if they were more or less likely to be vulnerable to phishing attacks.  Although this did not form part of this study the data gathered from the phishing campaigns shows a third of those participants who visited the phishing website were using mobile phones.  But without data to identify how many people read the emails on particular devices it is impossible to draw any conclusions from this other than to identify it as a potential for further research. It could also have formed a question on the participant survey and used to compare against the phishing results.

The coronavirus pandemic meant face to face training was not possible and it is acknowledged that memorable classroom based training (Mann, 2008) may have had a different impact on the study.  The training could also have been run through a video training programme and as noted previously the training could be improved by giving participants the required behaviours to help them handle a phishing email.  More than 50% of the survey respondents felt they had more work due to the coronavirus pandemic, which was also reflected in emails received from participants when seeking consent.  It is acknowledged that the increased workload (Legg & Blackman, 2019) could have impacted on the result of this study and an increased cognitive load would make interesting further research in phishing campaigns.  Using the previous suggestion of a wider case study over multiple SMEs, a survey could be used to identify the volume of emails received and compare this against simulated phishing campaigns.

        

References

# (2018) Can Phishing Awareness Training Cause More Harm Than Good? | Phishing and Ransomware Protection | Business Email Compromise [Online]. PhishProtection.com. Available from: <https://www.phishprotection.com/blog/can-phishing-awareness-training-cause-more-harm-than-good/> [Accessed 13 October 2019].

2019 State of the Phish – Cybersecurity Insights Report | Proofpoint UK (2018) [Online]. Available from: <https://www.proofpoint.com/uk/resources/threat-reports/state-of-phish> [Accessed 1 April 2020].

Abbasi, A., Zahedi, F. & Chen, Y. (2012) Impact of Anti-Phishing Tool Performance on Attack Success Rates. In: 2012 IEEE International Conference on Intelligence and Security Informatics, June 2012. pp. 12–17.

Aburrous, M., Hossain, M. A., Dahal, K. & Thabtah, F. (2010) Experimental Case Studies for Investigating E-Banking Phishing Techniques and Attack Strategies. Cognitive Computation, 2 (3), p. 242.

Academic Regulations (n.d.) [Online]. Available from: <https://www.leedsbeckett.ac.uk/public-information/academic-regulations/> [Accessed 20 December 2019a].

Alazri, A. S. (2015) The Awareness of Social Engineering in Information Revolution: Techniques and Challenges. In: 2015 10th International Conference for Internet Technology and Secured Transactions (ICITST), December 2015. pp. 198–201.

Ayodele, T., Shoniregun, C. a. & Akmayeva, G. (2012) Anti-Phishing Prevention Measure for Email Systems. World Congress on Internet Security (WorldCIS-2012), Internet Security (WorldCIS), 2012 World Congress on, June, pp. 208–211.

Baillon, A., Bruin, J. de, Emirmahmutoglu, A., Veer, E. van de & Dijk, B. van (2019) Informing, Simulating Experience, or Both: A Field Experiment on Phishing Risks. PLoS ONE, 14 (12) December, pp. 1–15.

Bell Judith, Waters Stephen (2018) Doing Your Research Project, A Guide for First-Time Researchers. Open University Press.

Canfield, C. I. & Fischhoff, B. (2018) Setting Priorities in Behavioral Interventions: An Application to Reducing Phishing Risk. Risk Analysis: An International Journal, 38 (4) April, pp. 826–838.

Carella, A., Kotsoev, M. & Truta, T. M. (2017) Impact of Security Awareness Training on Phishing Click-through Rates. In: 2017 IEEE International Conference on Big Data (Big Data), December 2017. pp. 4458–4466.

Chatchalermpun, S., Wuttidittachotti, P. & Daengsi, T. (2020) Cybersecurity Drill Test Using Phishing Attack: A Pilot Study of a Large Financial Services Firm in Thailand. 2020 IEEE 10th Symposium on Computer Applications & Industrial Electronics (ISCAIE), Computer Applications & Industrial Electronics (ISCAIE), 2020 IEEE 10th Symposium on, April, pp. 283–286.

Cheng, J. (2014) Out-of-Control Army Phishing Test Results in New Guidelines – [Online]. Defense Systems. Available from: <https://defensesystems.com/articles/2014/03/18/army-phishing-email-test.aspx> [Accessed 1 November 2019].

Compton, A. (2020) Tatanus/SPF [Online]. Available from: <https://github.com/tatanus/SPF> [Accessed 6 July 2020].

cyberconsultant3199 (2020) Cyberconsultant3199/Phishing-KIngphisher [Online]. Available from: <https://github.com/cyberconsultant3199/Phishing-KIngphisher> [Accessed 6 July 2020].

Data Protection Act 2018 (n.d.) [Online]. Available from: <http://www.legislation.gov.uk/ukpga/2018/12/contents/enacted> [Accessed 26 June 2020].

Data Security Incident Trends (2019a) [Online]. Available from: <https://ico.org.uk/action-weve-taken/data-security-incident-trends/> [Accessed 12 December 2019].

Data Security Incident Trends (2020) [Online]. Available from: <https://ico.org.uk/action-weve-taken/data-security-incident-trends/> [Accessed 13 June 2020].

Dhamija, R., Tygar, J. D. & Hearst, M. (2006) Why Phishing Works [Online]. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2006. New York, NY, USA: Association for Computing Machinery, pp. 581–590. Available from: <https://doi.org/10.1145/1124772.1124861>.

Finn, P. & Jakobsson, M. (2007) Designing Ethical Phishing Experiments. Technology and Society Magazine, IEEE, 26 February, pp. 46–58.

Gophish – Open Source Phishing Framework (n.d.) [Online]. Available from: <https://getgophish.com/> [Accessed 6 July 2020b].

Görling, S. (2007) An Overview of the Sender Policy Framework (SPF) as an Anti‐phishing Mechanism. Internet Research, 17 (2) January, pp. 169–179.

Gupta, B. B., Arachchilage, N. A. G. & Psannis, K. E. (2018) Defending against Phishing Attacks: Taxonomy of Methods, Current Issues and Future Directions. Telecommunication Systems, 67 (2) February, pp. 247–267.

Hackers Are Using Covid-19 Disruption to Infiltrate Corporate Networks (2020) The Economic Times [Online], 27 March. Available from: <http://link.gale.com/apps/doc/A618601142/STND?u=lmu_web&sid=zotero&xid=5fd17803> [Accessed 21 April 2020].

Innab, N., Al-Rashoud, H., Al-Mahawes, R. & Al-Shehri, W. (2018) Evaluation of the Effective Anti-Phishing Awareness and Training in Governmental and Private Organizations in Riyadh. In: 2018 21st Saudi Computer Society National Computer Conference (NCC), April 2018. pp. 1–5.

Jolly, J. (2020) Huge Rise in Hacking Attacks on Home Workers during Lockdown. The Guardian [Online], 24 May. Available from: <https://www.theguardian.com/technology/2020/may/24/hacking-attacks-on-home-workers-see-huge-rise-during-lockdown> [Accessed 30 June 2020].

Kabanda, S., Tanner, M. & Kent, C. (2018) Exploring SME Cybersecurity Practices in Developing Countries. Journal of Organizational Computing & Electronic Commerce, 28 (3) July, p. 269.

Kaila, U. & Nyman, L. (2018) Information Security Best Practices: First Steps for Startups and SMEs. Technology Innovation Management Review, 8 (11) November, p. 32.

Kirlappos, I. & Sasse, M. A. (2012) Security Education against Phishing: A Modest Proposal for a Major Rethink. IEEE Security Privacy, 10 (2) March, pp. 24–32.

Kumaraguru, P., Sheng, S., Acquisti, A., Cranor, L. F. & Hong, J. (2010) Teaching Johnny Not to Fall for Phish. ACM Transactions on Internet Technology (TOIT), 10 (2) June, p. 7:1–7:31.

Kwak, Y., Lee, S., Damiano, A. & Vishwanath, A. (2020) Why Do Users Not Report Spear Phishing Emails? Telematics and Informatics, 48 May.

Lamb, J. (2019) Practical Phishing with Gophish [Online]. Medium. Available from: <https://medium.com/airwalk/practical-phishing-with-gophish-7dd384ad1840> [Accessed 15 February 2020].

LCN.com (n.d.) Domains, Hosting, and Cloud Servers, and SSL [Online]. LCN.com. Available from: <https://www.lcn.com> [Accessed 6 July 2020].

Legg, P. & Blackman, T. (2019) Tools and Techniques for Improving Cyber Situational Awareness of Targeted Phishing Attacks. In: 2019 International Conference on Cyber Situational Awareness, Data Analytics And Assessment (Cyber SA), June 2019. pp. 1–4.

Lloyd, G. (2020) The Business Benefits of Cyber Security for SMEs. Computer Fraud & Security, 2020 (2) February, pp. 14–17.

Mann, I. (2008) Hacking the Human, Social Engineering Techniques and Security Countermeasures. In: Hacking the Human, Social Engineering Techniques and Security Countermeasures. UK, pp. 195–209.

Many Small Firms Are Still Unprepared for GDPR (2018) International Financial Law Review [Online]. Available from: <https://go.openathens.net/redirector/leedsmet.ac.uk?url=http%3a%2f%2fsearch.ebscohost.com%2flogin.aspx%3fdirect%3dtrue%26db%3dedsbig%26AN%3dedsbig.A532882148%26site%3deds-live%26scope%3dsite%26authtype%3Dathens> [Accessed 24 June 2020].

McElwee, S., Murphy, G. & Shelton, P. (2018) Influencing Outcomes and Behaviors in Simulated Phishing Exercises. In: SoutheastCon 2018, April 2018. pp. 1–6.

Megha, N., Remesh Babu, K. R. & Sherly, E. (2019) An Intelligent System for Phishing Attack Detection and Prevention. 2019 International Conference on Communication and Electronics Systems (ICCES), Communication and Electronics Systems (ICCES), 2019 International Conference on, July, pp. 1577–1582.

Melnyk, S. A., Speier-Pero, C. & Connors, E. (2019) Blockchain Is Vastly Overrated; Supply Chain Cyber Security Is Vastly Underrated. Supply Chain Management Review, p. 32.

Mishra, A. K., Tripathy, A. K. & Swain, S. (2018) Analysis and Prevention of Phishing Attacks in Cyber Space. 2018 First International Conference on Secure Cyber Computing and Communication (ICSCCC), Secure Cyber Computing and Communication (ICSCCC), 2018 First International Conference on, December, pp. 430–434.

Muncaster, P. (2020) COVID-19: Tackling a Cyber-Pandemic. Info Security, 17 (2) Q2, pp. 8–9.

Nycz, M., Martin, M. J. & Polkowski, Z. (2015) The Cyber Security in SMEs in Poland and Tanzania. 2015 IEEE International Conference on Electro/Information Technology (EIT), January, p. AE.

Oster, B. & Tucker, E. (2019) Staff Phishing Testing: Raising Awarness vs Failing Concept. Info Security, 16 (2) Q2, pp. 56–57.

Our Most Advanced Penetration Testing Distribution, Ever. (n.d.) [Online]. Available from: <https://www.kali.org/> [Accessed 6 July 2020c].

pentestgeek (2020) Pentestgeek/Phishing-Frenzy [Online]. pentestgeek. Available from: <https://github.com/pentestgeek/phishing-frenzy> [Accessed 6 July 2020].

Phishing Frenzy – Manage Email Phishing Campaigns – Penetration Testing (n.d.) [Online]. Available from: <https://www.phishingfrenzy.com/resources/getting_started> [Accessed 15 February 2020d].

R, K. (2018) The Trouble with Phishing – NCSC [Online]. Available from: <https://www.ncsc.gov.uk/blog-post/trouble-phishing> [Accessed 1 November 2019].

Salem, O., Hossain, A. & Kamala, M. (2010) Awareness Program and AI Based Tool to Reduce Risk of Phishing Attacks. In: 2010 10th IEEE International Conference on Computer and Information Technology, June 2010. pp. 1418–1423.

Sankhwar, S. & Pandey, D. (2017) A Comparative Analysis of Anti-Phishing Mechanisms: Email Phishing. International Journal of Advanced Research in Computer Science; Udaipur [Online], 8 (3) March. Available from: <http://search.proquest.com/docview/1901445040/abstract/10AA38CC6B66430EPQ/1> [Accessed 27 March 2020].

Scheau, M. C., Arsene, A.-L. & Dinca, G. (2016) Phishing and E-Commerce: An Information Security Management Problem. Journal of Defense Resources Management, (1), p. 129.

Should You Really Phish Your Own Employees? (2017) NS Tech, 15 August [Online blog]. Available from: <https://tech.newstatesman.com/business/phishing-employees> [Accessed 13 October 2019].

Small Businesses Struggle to Compete for Cybersecurity Talent (2019) Globe & Mail (Toronto, Canada), p. B5.

Stay Safe Online Top Tips for Staff (n.d.) [Online]. Available from: <https://www.ncsc.gov.uk/training/top-tips-for-staff-web/story_html5.html> [Accessed 12 May 2020e].

Stembert, N., Padmos, A., Bargh, M. S., Choenni, S. & Jansen, F. (2015) A Study of Preventing Email (Spear) Phishing by Enabling Human Intelligence. In: 2015 European Intelligence and Security Informatics Conference, September 2015. pp. 113–120.

Take Jigsaw’s Phishing Quiz (n.d.) [Online]. Available from: <https://g.co/phishingquiz> [Accessed 19 April 2020f].

The Human Factor 2019 Report – Modern Cyber Attacks | Proofpoint (2019b) [Online]. Available from: <https://www.proofpoint.com/uk/resources/threat-reports/human-factor> [Accessed 1 April 2020].

Tschakert, K. F. & Ngamsuriyaroj, S. (2019) Effectiveness of and User Preferences for Security Awareness Training Methodologies. Heliyon, 5 (6) June, p. e02010.

UK and US Security Agencies Issue COVID-19 Cyber Threat Update (n.d.) [Online]. Available from: <https://www.ncsc.gov.uk/news/security-agencies-issue-covid-19-cyber-threat-update> [Accessed 1 July 2020g].

Yin, Robert K (2018) Case Study Research and Applications. USA: SAGE Publications.

Yousif, H., Al-saedi, K. H. & Al-Hassani, M. D. (2019) Mobile Phishing Websites Detection and Prevention Using Data Mining Techniques. International Journal of Interactive Mobile Technologies, 13 (10) October, p. 205.

Zeydan, H. Z., Selamat, A. & Salleh, M. (2014) Survey of Anti-Phishing Tools with Detection Capabilities. In: 2014 International Symposium on Biometrics and Security Technologies (ISBAST), August 2014. pp. 214–219.

---

Appendix 1  – Survey Questions

The original design of the questions was as follows:-

Do you feel that email is important for you to do your job?

This was answered using a scale from Extremely import to not important at all.

In a typical work day how many emails do you receive?

This was answered using a range of answers from 0-10 to 50+

How carefully do you check the emails do you receive?

This was answered in the range Extremely careful to not carefully at all

Do you feel there is technology at work to protect you from phishing?

This was a simple agree/disagree question

Have you received any training to detect phishing emails?

This was a simple yes no answer but was followed up with a question if they responded yes to ask them how useful they thought that training was which was answered on a scale from Extremely helpful to not helpful at all.

Are you aware of the dangers of phishing emails?

This was originally designed as a yes no answer but later changed to extremely aware to not aware at all.  This was changed to allow for the participant to measure a vague level of awareness.

Are you confident you can spot a phishing email?

This was again measure against a scale from extremely confident to not confident at all and was designed to further validate the impact of the training.

Do you check the email of the person sending you the email?

This was set as a range of always to never.  It was once again designed to test the recall of the participants with regards to one of the feature of the training.  To always check the email address being sent from.

Do you click on the link in an email or type the adress manually?

This was a simple yes no answer but was given again to validate one of the key components of the awareness training,

The original survey was amended in light of the Coronavirus pandemic in order to add questions related to what impact the virus had on workload and the number of emails.

---

Appendix 2 – First Phishing Email

Hi All,

XX have been busy building some additional pages to the XXXXXX website to include the eRetail companies and have built a page for XXXXXX.  Check it out and let us know what you think.

---

Appendix 3 – Second phishing email

Hi All,

We have had a couple of occurrences lately of emails with customer details being forward inappropriately. Read the linked document to ensure you are doing the right thing when sharing information https://drive.google.com/file/d/1RVxyk5fn-9bRNoMckssFHgc-V2SrRwxGmr/view?usp=sharing

Thanks

Andy

---

Appendix 4 – Phishing Campaign 1 Results

xx.xx.xx.xx- – [06/Apr/2020:06:47:53 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:06:55:12 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:06:55:21 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:07:57 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:14:55 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Mobile/15E148 Safari/604.1”

xx.xx.xx.xx- – [06/Apr/2020:07:19:50 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Linux; Android 7.0; HUAWEI VNS-L31) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Mobile Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:19:56 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:34:04 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Linux; Android 9; SM-G950F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:34:14 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:41:29 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:48:45 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:48:48 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Linux; Android 9; FIG-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Mobile Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:54:10 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:57:37 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:07:57:46 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36”

xx.xx.xx.xx- – [06/Apr/2020:08:40:19 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Mobile/15E148 Safari/604.1”

xx.xx.xx.xx- – [06/Apr/2020:08:40:34 +0000] “GET /hawk.html HTTP/1.1” 200 1675 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36”

---

Appendix 5 – Phishing campaign 2 results

xx.xx.xx.xx- – [21/May/2020:07:41:44 +0000] “GET /file.html HTTP/1.1” 200 1797 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36”

xx.xx.xx.xx- – [21/May/2020:07:44:42 +0000] “GET /file.html HTTP/1.1” 200 1797 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36”

xx.xx.xx.xx- – [21/May/2020:07:45:30 +0000] “GET /file.html HTTP/1.1” 200 1797 “-” “Mozilla/5.0 (Linux; Android 9; FIG-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Mobile Safari/537.36”

xx.xx.xx.xx- – [21/May/2020:07:51:58 +0000] “GET /file.html HTTP/1.1” 200 1797 “-” “Mozilla/5.0 (iPhone; CPU iPhone OS 13_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Mobile/15E148 Safari/604.1”

xx.xx.xx.xx- – [21/May/2020:08:10:10 +0000] “GET /file.html HTTP/1.1” 200 1797 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36”

---

Appendix 6 – Survey Results

Do you feel email is important to your job?
Extremely Important	6
Very important	1
Somewhat important	5
Not so important	1
Not at all important	0
	13
In a typical work day how many emails do you receive?
0-10	2
11-20	3
21-30	1
31-40	2
41-50	0
50+	5
	13
How carefully do you check the email you receive?
Extremely Carefully	2
Very Carefully	6
Carefully	3
Not Very carefully	2
Not carefully at all	0
	13
Have you received any training on detecting phishing emails?
Yes	9
No	4
	13
if so do you think the training was helpful to allow you to avoid phishing emails?
Extremely helpful	3
Very helpful	4
Somewhat helpful	4
Not so helpful	0
Not helpful at all	1
	12
Are you aware of the dangers of phishing emails?
Extremely aware	6
Very aware	6
Somewhat aware	0
Not so aware	0
Not at all aware	1
	13
Are you confident you can spot a phishing email?
Extremely confident	2
Very confident	4
Somewhat confident	6
Not so confident	0
Not at all confident	1
	13
Do you check the email address of the person sending you an email?
Always	3
Usually	5
Sometimes	3
Rarely	2
Never	0
	13
Do you click on the link in an email or type the address manually?
Click the link	12
Enter the address manually	1
	13
Do you feel that you have received more phishing emails since the start of the coronavirus pandemic?
Lots more	1
Some more	1
About the same	9
Less	0
Not noticed	2
	13
Do you feel working from home makes you more vulnerable to phishing attacks?
Yes	2
No	11
	13
Do you feel your workload has increased due to the coronavirus pandemic?
A lot more	5
Some more	4
About the same	2
Less	2
Not noticed	0
	13
Have you received more emails due to the coronavirus pandemic?
A lot more	3
Some more	5
About the same	5
Less	0
Not noticed	0
	13

---

Appendix 7 – Ethics approval email

Application Ref: 70392
Applicant Name: DAVE GILL
Project Title: Effectiveness of awareness training on the impact of phishing campaigns

Dear DAVE GILL, Muthu Ramachandran, the Local Research Ethics Co-ordinator, can confirm that the above research project has been given ethical approval and may commence. Please see your online application for any comments or recommendations.

This project has received research ethical approval in line with the Research Ethics Policy and Procedures of Leeds Beckett University.

Please note that if you wish to make substantial changes to the project, new ethical approval would be required.

Sent on behalf of the Local Research Ethics Co-ordinator.

---

Appendix 8 – Email approval from SME

Re: Phishing Case Study Request

<xxxxx.co.uk>

Mon 09/03/2020 17:25

To:

•  Gill, Dave George Henry (Student) <xxxxxx@student.leedsbeckett.ac.uk>

Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe.

This is approved.

 

From: “Gill, Dave George Henry (Student)” <d.gill9722@student.leedsbeckett.ac.uk>
Date: Thursday, 5 March 2020 at 17:54
To: XXXXXXXXXXXXXXXXXXXXXXXXXX
Subject: Phishing Case Study Request

 

Hi xxxxx,

 

Following on from our conversation about running a simulated Phishing campaign I would just like to summarise what is proposed and seek your permission to continue.

 

I would like to send simulated phishing emails to staff in order to record how many people follow links in the emails and engage with the website.  I would then propose some form of training to help people identify phishing emails followed by a survey to allow them to record how confident they feel in dealing with Phishing emails at work.

 

All data will be shared (if you wish) with you and no personal data will be recorded.  I would also point out that I will provide a copy of the final report before submission if required.

 

Thanks

 

 

Dave Gill