The importance of independent third-party testing of anti-malware solutions

Being a part of the Threat Labs of one of the world’s most prominent security companies, you can imagine that securing our customers is at the center of our lives. We have fantastic teams working in different fields, from phishing and scams to analyzing complex malware campaigns. We extensively use AI in all layers to gain efficiency and efficacy, and we work with law enforcement tracking threat actors to shut down their businesses.  

Our goal is to create a safe world and to keep our customers safe. We think we do a great job, but that’s not an objective measure. How can we make sure we are offering the best possible solutions to our customers? We can do it by having independent third parties testing our security solutions from different points of view, helping us to find out which are our strong points, where we have room for improvement and how we perform compared to our main competitors.  

On top of that, we can also obtain valuable information from proof-of-concept technologies and new products and solutions we are developing. It is an extended Quality Assurance team that will help us improve the quality of our solutions. For many years, we’ve worked with many renowned testers, participating both in their public test series as well as commissioning private tests. 

We value transparency, and that is why we participate in tests that have an open approach and give the necessary information to users so they can make informed decisions based on facts.  

Security solutions are quite complex, and its testing is too. Most of the testers we work with are AMTSO members that follow the AMTSO standard and its spirit. AMTSO is an international non-profit association that focuses on addressing the global need for improvement in the objectivity, quality and relevance of anti-malware testing methodologies. Since its beginning back in 2008, AMTSO has gathered the best experts in the field to create different documents that offer guidance for anyone that is interested in performing good tests; you can find all of them available for free on AMTSO’s website. 

Being transparent does not mean that a test is good; however, it will give the user the information they need to decide whether that test and its results are relevant for them. For example, a user may be concerned about the great number of scams and phishing they receive on a daily basis and needs to find a solution that excels in that field; a transparent test will disclose the methodology it is using and the relevant metrics. The user can even compare different tests to see which one is more pertinent to their needs. 

From a Threat Labs point of view, the main advantages of independent antivirus testing are obvious: They provide an objective assessment, real-world testing, and in-depth analysis. They also help us with continuous improvement and support us in building trust and credibility with our customers. 

Of course, it’s important to consider the limitations and potential biases of individual tests. Anti-malware/antivirus tests can vary significantly in their methodologies, criteria, and metrics. This can lead to inconsistent results and make it challenging to compare different products, that’s why we consider transparency to be key. 

Back in 2012, this was the performance of the different security solutions, based on public results published by AV-Comparatives: 

AV-Comparatives Real-World Protection Test, March-June 2012 

As you can see, there was definitely room for improvement. A lot of improvement. Using a limited number of test cases, the protection rate was under 95% for most products, whatever anti-malware solution was being used. Now let’s fast-forward 10 years and see how current solutions perform: 

AV-Comparatives Real-World Protection Test, February-May 2022   

There is a huge change, most security solutions have improved their users’ protection dramatically. And one of the main drivers for this change are the very same tests that measure our efficacy, that force us to improve and find as many ways as possible to protect users without compromising other aspects, such as the computer performance.  

Overall, the anti-malware testing industry provides valuable insights into the effectiveness of antivirus and anti-malware solutions. Organizations and individuals should use a combination of test results, expert opinions, and their own research to make informed decisions about which anti-malware solution to use.