By Vanessa Bandeira Lins Teixeira

The search for security practices to make life more protected is not unusual. We notice it from the creation of a password to use an account to the simple act of activating a car alarm. However, in a computing environment, it’s necessary to understand that the action of protecting oneself virtually requires more specific protocols that can prevent the user from being attacked or, at least, mitigate the effects associated with it.

In general, an environment with vulnerabilities allows an attacker to apply techniques that result, for example, in access to confidential information, resulting in the need to adopt measures that prevent the occurrence of security incidents, making it less susceptible to vulnerability exploitation.

The fact is that the recurrent security breaches to information and communication systems portray the gradual growth of system “gaps” left by users, making them targets for possible attacks. This can be illustrated by the 23543 vulnerabilities disclosed through CVEs by the National Institute of Standards and Technology – NIST in 2022 (number released up to December 12, 2022) [9], an increase of approximately 60% over the numbers verified in 2017 [12]. Given this considerable rise in the number of vulnerabilities, users are faced with the obstacle of how to prioritize patches in their systems through practices that mitigate the occurrence of attacks on computer systems.

These vulnerable environment scenarios are part of the reality experienced by security teams, more specifically blue teams, who work on the daily assessment of systems in order to protect assets from vulnerabilities that affect critical devices or systems in companies. This type of daily security assessment is part of the context present in the management of vulnerabilities and compliance of an environment.

Besides identifying threats, it’s fundamental that practices to mitigate vulnerabilities are also used in the operating systems, in order to keep corporate environments protected, considering that at any given moment some vulnerability may be being exploited by hackers.

One thing is certain: new software is developed every day and each one of them can bring its own loopholes. Moreover, the configurations of these applications can also increase them. As a result, security-oriented configurations are missing from a significant part of today’s operating systems. These assets, which are often not configured correctly with security requirements in mind, become easy targets for a number of security attacks. In this way, it becomes clear that applying compliance policies to operating systems helps preserve the environment through security settings, resulting in environments that are less exposed to attacks from malicious users.

1. Understanding Security Compliance Policies for Operating Systems

A large number of institutions focused on cyber security focus their efforts on developing security practices. The Center for Internet Security (CIS) is a non-profit organization whose mission is cyber defense, and one of its main deliverables is compliance policies for operating systems.

A compliance policy is a document that aims to ensure that these systems follow a procedure with the best security recommendations in order to support the organization’s security objectives. In other words, these are recommendations structured into categories, composed of security configuration items for the environments, influencing the reduction and mitigation of vulnerabilities and threats. These recommendations are thoroughly reviewed and updated by information security professionals.

2. Operating System Security Methodology Considering Compliance Policies

The importance of assessing the security of operating systems becomes even more crucial when this analysis is done on server operating systems. Threats related to vulnerabilities found in those systems tend to generate more significant risks since these systems are generally used for critical business activities.

In order to serve as a proof of concept for a security evaluation of operating systems considering compliance policies, as well as a basis for other future security evaluation projects, this research has proposed the definition of a methodology that consists of a workflow divided into 5 processes, as presented below and illustrated by Image 1. It’s also worth mentioning that the definition of this methodology considered a set of requirements, conditions and situations (such as the analysis of OS assemblies available in the market, the most commonly used vulnerability tools for OS analysis, research on Compliance Management available in the Academy, leading security centers on Vulnerability and Compliance Management, among others) that were studied by the author, and was also based on her experience and involvement in the vulnerability and compliance management projects she has been working on for the past few years at Tempest.

Image 1: Modeling of the methodology for evaluating the security of operating systems considering compliance policies | Source: Created by the author

For the application of this proposed methodology for security assessment of operating systems considering compliance policies, it’s important to understand the flow of the process used, in order to detail the activities required for its realization. This process is laid out as follows:

  1. Specify what the focus of the analysis is;
  2. Establish the metrics that will be used;
  3. Select and configure the environment that will be the target of the security assessment;
  4. Describe the artifacts used, so that the security analysis can be performed;
  5. Evaluate and present the results obtained, in order to generate useful information and recommendations regarding the current security level of the system.

A better description of each of these steps of the proposed methodology will be presented below.

2.1. Define the objectives of the assessment

In the initial activity the objectives to be followed within the safety assessment are defined. These can include, for example: assessing the security of mobile operating systems considering compliance policies; analyzing the impacts of compliance policy enforcement against system vulnerabilities; or ranking the security level of (desktop) operating systems looking at different types of user configurations. By defining the objectives, the assessment loses its subjective character and gains an objective/practical character. This is important because it’s not recommended to state whether a configuration is correct or not without objectives and metrics in place.

2.2. Determining the metrics

This step will define the metrics for analyzing the results, systematizing the quantification. In other words, to conduct a system security assessment using compliance policies, it’s important to define how the analysis of these results will be carried out. Examples of metrics include: number of vulnerabilities (quantification of vulnerabilities found based on their criticality) and level of compliance (percentage number that indicates how many of the security controls are in line with the adopted compliance policy).

2.3. Selecting and setting up the environment

This activity consists in the definition and configuration of the assets that will be part of the environment. For example, this activity will define the hardware used (including information such as processing, storage, etc.) and software installed (operating systems, user applications, among others).

An important point to be emphasized is linked to the need to know the profile to which the assets are destined. For example, operational systems that are intended for use, such as servers, can be structured with a domain or, in the case of Windows servers, must have domain controller settings. It’s important to note that in this activity information such as operating system version is also detailed, because depending on this version, the amount of vulnerabilities may differ.

Finally, it’s worth mentioning that in this step all the environment configuration, including network aspects, such as IP address definition, must be done.

2.4. Describing the design and execution of the assessment

This next phase is the definition and execution of the assessment, following the objectives and characteristics defined in the previous sections. The description of the proposed assessment project is composed, among other artifacts, of operating system audit files and the definition of an environment verification policy.

The process of creating audit files, as can be seen in Image 2, starts from the initiative to verify that the operating system settings comply with security recommendations. Through the audit files, it’s possible to enforce compliance policies in a way that is interpreted by the operating systems. From this, compliance evaluation can be performed. An example of the basic structure of an item in an audit file can be seen in the image below. Each item is checked using the custom_item flag. These markers are structured with keywords that will be interpreted by the vulnerability scanning tool in order to identify whether or not the operating system is compliant with security recommendations.

Image 2 . Basic structure of an item from the audit file | Source: Created by the author

The use of the vulnerability scanning tool is essential for the collection of information related to the state of the environment and the identification of possible non-compliance. This data is obtained through the use of audit files that define the expected value for each security setting applied to the operating systems, based on the compliance policies of these systems. In order to do this, it’s necessary that this tool receives the required configurations for its installation on the asset that will host it.

The project description is instituted with the creation of an environment verification policy. For the procedure of checking the configurations of a system, the vulnerability scanning tool is assigned the identification information of the analyzed asset on the network, such as: IP address, authentication credentials, system audit file, domain to which the machine belongs, if any, and a selection of audit plugins that are made available by the chosen vulnerability scanning tool. This information is necessary for the vulnerability scanning tool to identify the asset to be evaluated.

2.5. Analyze and Present the Results

The final step of the methodology consists of analyzing the data that was returned after the assessment was performed. The information returned by the scan tool can be structured in files such as html, pdf, or csv, containing the information related to the configurations that were scanned. This data may consist of configuration items that failed when analyzed from the perspective of the compliance policies of the chosen operating systems.

It’s important to consider the metrics previously determined as described in subsection 2.2, relating them to the objectives defined for the application of the methodology. These metrics will help in filtering and organizing the data, such as the creation of graphs that facilitate the visualization and presentation of information, in order to achieve the intended goals defined in step 2.1.

3. Case Study for the Validation of the Methodology and Analysis of the Results Obtained

As a way of validating the application of the proposed methodology, the research made use of a case study that is presented in this section. This study aimed to analyze the security of server operating systems commonly used by companies since these types of operating systems mostly host shared services and the occurrence of vulnerabilities in these systems can impact higher risks. In addition, this research seeks to encourage good practices capable of facilitating users’ understanding of the vulnerabilities and threats present, showing the main alternatives for correcting the settings.

The metric adopted in this case study to measure the compliance level is the compliance index, which represents the return identified by each compliance policy considering the configurations of the operating systems [7]. This metric assists in the analysis and categorization of the results. The possible values for this metric are: Accepted, for the configuration that conforms to the security recommendations, and Failed, for the configuration that does not conform to the security recommendations.

For the selection of the operating systems to be adopted in this evaluation, a listing published by CVE Details with the top 50 operating system versions with the most security vulnerabilities related to them was used[11].

Of the 50 operating systems, Debian 8 was the operating system that stood out in the analyzed list. From this selection, an environment was established for the emulation of this system in the default settings.

The creation of the audit files was performed based on the Debian 8 compliance policies provided by CIS benchmarks, which is a leading organization in security benchmarks and one of the most adopted for this purpose.

When performing configuration checks on the operating systems, relevant data for the security assessment was gathered [7]. This data was extracted from the vulnerability scanning software in .csv format and treated using the dplyr library, which allows data manipulation with mathematical and statistical functions in the R language.

For Debian 8, out of the grand total of items scanned and classified into 6 categories (Initial Configuration; Services; Network Configuration; Logging and Auditing; Access, Authentication and Authorization; System Maintenance), 263 returned the status of failed and 152 items were accepted. The result can be seen in Image 3, which shows the number of items per category.

Image 3 . Compliance of the configuration items in the Debian 8 Operating System | Source: Created by the author

Looking at the categories of Initial Configuration, Network Configuration, Logging and Auditing, and Access, Authentication and Authorization, it is noticeable the large number of failed items that were identified in these 4 categories. For the Initial Configuration category, 44 items returned a failed status. These items, in general, correspond to separate partition settings, user file permissions, and software update management. The Network Configuration category showed 48 non-compliant items that are related to network configuration recommendations, such as protocols and firewall rules. And finally, the items that showed the most configuration failures are evaluated. For the Logging and Auditing category, 121 nonconformities were detected and in the Access, Authentication and Authorization category, 41 items failed. These numbers may represent threats to the logging and auditing systems since the files belonging to this category are responsible for monitoring suspicious behavior in the system. In this sense, the Access, Authentication and Authorization category is generally configured by items that deal with user password management, user and group identification, and the access privileges to system resources specified for each user or group. Although the categories described by Services and System Maintenance returned 7 and 2 failed items, respectively, it can be seen that their impact was lower when related to the amounts of failures in the other categories.

The numbers extracted through the operating system security assessment made it possible to classify the information by system categories that had more failed configurations. This classification can serve as a guideline in the application of good security practices to the items that will be prioritized:

  • Policies related to system passwords: present criteria that must be followed by all users, ensuring that security during access to corporate systems is followed.
  • Access, authentication and authorization policies: these settings should be applied in order to prevent malicious users from accessing a company’s internal network and being allowed to access an organization’s important data.
  • Policies for monitoring the settings applied by users: The compliance policies related to this classification basically address the software accesses and installations performed by users. This management helps identify unknown accesses.
  • Network settings policies: Through these policies, network access controls are carried out.

 3. Conclusion

Even considering that the security configurations of operating systems have a subjective character, it’s possible, by means of metrics, to define how these configurations will be evaluated. By analyzing the path followed by the methodology proposed in this research, it can be concluded that, increasingly, seeking to mitigate and resolve security vulnerabilities are considered relevant and complex tasks. However, by analyzing the flow of logic proposed by the methodology, it’s feasible to systematize the vulnerability mitigation process in IT infrastructure assets, by means of security tools and documentation made available in the current scenario. In fact, what should not be left aside is that for every security configuration, the impact of the setting on the asset must be balanced against the threat arising from its non-application.

Despite the relevant results obtained in the application of the evaluation methodology proposed in this research, the discussion about compliance policies in operating systems still needs to go beyond the methods presented here. With this immersion in compliance management, it’s expected that other case studies will be conducted with the application of the methodology that was described. The purpose of this article is to encourage and clarify issues in an area that needs to go even more in conjunction with vulnerability management since a large part of the vulnerabilities arises from configurations that do not comply with security recommendations.


[1] Angraini, R. A. Alias, and Okfalisa, Information security policy compliance: Systematic literature review, the Fifth Information Systems International Conference, 23-24 July 2019 – Procedia Computer Science, vol.161, pp. 1216 – 1224, 2019, Surabaya, Indonesia.

[2] A. Gorbenko, A. Romanovsky, O. Tarasyuk, and O. Biloborodov, Experience report: Study of vulnerabilities of enterprise operating systems, in 2017 IEEE  28th  International  Symposium  on  Software  Reliability Engineering (ISSRE), Oct 2017, pp. 205–215. 

[3] A. Mounji and B. Le Charlier,Continuous assessment of a unix configuration: integrating intrusion detection and configuration analysisin Proceedings  of  SNDSS  ’97:  Internet  Society  1997  Symposium  on Network and Distributed System Security, 1997, pp. 27–35.

[4] A. L. Mesquida and A. Mas, Implementing information security best practices on software lifecycle processes: The iso/iec 15504 security extensionComputers and Security, vol. 48, pp. 19 – 34, 2015.

[5] C. of Internet Security, “CIS Benchmarks” Oct 2020, Acessed: 2020-10-24. [Online]. Available at: cis-benchmarks/

[6] C. of Internet Security, “CVE Details”, 2020. Acessaded: 2020-12-19. [Online]. Available at: php

[7]I. Chalvatzis, D. A. Karras, and R. C. Papademetriou, “Evaluation of security vulnerability scanners for small and medium enterprises business networks resilience towards risk assessment” in 2019 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), 2019, pp. 52–58.

[8]I. O. of Standardization, “ISO/IEC 27000:2018 information technology – security  techniques – information  security  management  systems – Overview  and  vocabulary. Fev 2018.  Acessed 2 nov 2022 . Available at:

[9]NIST, “CVEs Received and Processed”. Acessed 1 Dec 2021. Available at:

[10] N. Veerasamy, High-level methodology for carrying out combined red and blue teams”,  in 2009 Second International Conference on Computer and Electrical Engineering, vol. 1, 2009, pp. 416–420.

[11] O. H. Alhazmi and Y. K. Malaiya, “Quantitative vulnerability assessment of systems software” in 2005, Annual Reliability and Maintainability Symposium. Proceedings, pp. 615–620.

[12] S. N. Scott Caveza and R. Quinlan, “Tenable’s 2020 threat landscape retrospective”, Tenable Network Security, Jan 2021. [Online]. Available at: 2020-threat-landscape-retrospective