After working with AV-Comparatives for many years, I have personally developed a great respect for the way they curate files for their tests. They work diligently to select files that are relevant, are not in that “unwanted” category (which vendors would lobby to dispute out of their test), and they are able to source hundreds of thousands of recent files for the test.\u00a0 That said, one thing we found is that it is incredibly difficult, if you’re using a traditional scoring model, to attempt to source a perfect number of files that represent ecosystem prevalence.<\/p>\n
For one, many malware families rely on non-PE components to spread. Jenxcus is a good example \u2013 its VBS (Visual Basic script) component is one of the most frequently blocked files on our customers’ computers. However, its PE component is seen comparatively rarely, so it’s quite difficult to source enough Jenxcus PE files for a test to equate to that family’s ecosystem prevalence.\u00a0 Samples from some families might be easier to source than others (more willing to be found or submitted to public sources).\u00a0 These constraints make it practically impossible to select a test set that perfectly equates to the ecosystem.<\/p>\n
Looking at the prevalence model<\/h3>\n
Enter the prevalence model.\u00a0 AMTSO<\/span><\/a> through the Realtime Threat List (RTTL<\/span><\/a>) has been making strides lately to encourage vendors to share malware prevalence information with testers to help testers build better test sets.\u00a0 While we have been moving toward critical mass and getting closer to the needed features to make that project work, Microsoft offered to sponsor AV-Comparatives and provide telemetry details from over 200 million computers in over 100 different countries to them to create a prevalence-weighted model.<\/p>\n The following chart shows how the test set stacks up to the ecosystem (# of files in comparison to ecosystem prevalence).<\/p>\n Figure 1: \u00a0In general, files selected for the most prevalent malware families (those in the high category) were underrepresented using the traditional method of scoring and those in the low category were overrepresented.<\/em><\/p>\n When you drill down a layer to the highly prevalent malware families, you can start to see why the numbers don’t line up.\u00a0 Some of the file infector families, like Sality<\/span><\/a> and Virut<\/span><\/a>, had a very large sample set (a great representation of the family in fact). However, other prevalent families, like Gamarue<\/span><\/a>, Dorv<\/span><\/a>, Jenxcus<\/span><\/a>, and Sventore<\/span><\/a> were underrepresented.\u00a0 Sventore was new \u2013 there was only one file to represent that family.\u00a0 Gamarue, Dorv, and especially Jenxcus didn’t have nearly enough recent PE files available for the test to allow them to equate to their ecosystem prevalence.<\/p>\n \u00a0Figure 2:\u00a0\u00a0Another\u00a0example of the\u00a0test scores\u00a0not lining up.<\/em><\/p>\n The prevalence-weighted model takes into account the prevalence of the tested file, the malware family associated with the file, and the malware family’s partition (high, moderate, low, very low) to calculate each file’s impact to the test which balances the score with the actual customer impact in the ecosystem.<\/p>\n For more details about the exact calculation method, you can see the AV-Comparatives report released today<\/span><\/a>.<\/p>\n The charts above show how the prevalence model balanced test scores to make them more accurately represent a vendor’s detection capabilities.\u00a0 In essence, missed files were scored to represent the malware people were more likely to encounter, which is good information for consumers.\u00a0 However, prevalence-weighting the score can mean that vendors (at least those who monitor malware prevalence) might have very similar test scores.\u00a0 Therefore, additional context is probably needed to help consumers make decisions.<\/p>\n![\"](\"http:\/\/www.microsoft.com\/security\/portal\/blog-images\/a\/MMPCAVC.png\")
<\/a><\/p>\n![\"A](\"http:\/\/www.microsoft.com\/security\/portal\/blog-images\/a\/MMPCAVC2.png\")
<\/a><\/p>\n