WO2008014507A2 - Systems and methods for scoring scanning vendor performance - Google Patents

Abstract

Systems, computing devices, and methods for qualifying scanning vendors in the payment-by-card industry are provided. The qualification of vendors can ensure compliance with security standards in the payment-by-card industry. The qualification of a vendor involves testing vendor ability to discover and identify an established set of vulnerabilities in a simulated data situation. Vendor scoring procedures involve consideration of the exploitability, severity, and weight of vulnerabilities discovered.

Description

SYSTEMS AND METHODS FOR SCORING SCANNING VENDOR

PERFORMANCE

Cross- Reference to Related Applications This application claims the benefit of U.S. Provisional Application Serial

No. 60/833,969, filed July 28, 2006, and International Patent Application No. PCT/US07/70709, filed June 8, 2007, both of which applications are incorporated by reference herein in their entireties.

Background of the Invention The growing popularity of electronic payment transactions signals an increasing need for electronic data security. Networks and servers have become the increasing target of unscrupulous individuals. The lure of large sums of money and goods drives these individuals to mount new and more sophisticated attacks on financial institutions, vendors, and individual purchasers. Although many transactions still take place in a face-to-face situation, an increasing amount of purchases are made online, over the phone, or through the mail — where there is no card present at the merchant site. This environment requires reevaluation of data security procedures and fraud prevention procedures in the payment-by-card industry. Some financial institutions have developed programs that are designed to help members, merchants and service providers - Third Party Processors (TPPs) and Data Storage Entities (DSEs) - to proactively protect them and the overall payment system against the threat of compromise. These programs seek to accomplish this by identifying vulnerabilities in security processes and procedures. The compliance of participating members, merchants and service providers with the programs may be verified by various procedures, for example, onsite reviews, security self-assessments, and third-party security scans. To assist in determining compliance, vendors ("scanning vendors") may conduct the security scans to determine whether any vulnerabilities exist in the security processes and procedures at the participating members, merchants and service providers.

Consideration is now being given to improving procedures for security compliance by entities in the electronic payment-by-card industry. In particular attention is directed to procedures for qualifying scanning vendors who provide security scans and other activities for compliance with security programs.

Summary of the Invention

Systems and methods for qualifying a scanning vendor are provided. The systems and methods involve test scans of simulated data situations in a test bed by the candidate vendor. The test bed includes defined vulnerabilities. A Scoring System finds matches between a candidate vendor's findings and a validator's base of vulnerabilities, and assigns a score representative of the candidate vendor's demonstrated scanning capabilities. For purposes of computing the score, vulnerabilities are characterized and categorized by their features (e.g. severity, exploitability, time varying status), and accordingly assigned weights for scoring. A baseline set of vulnerabilities presented in the test bed may include a snapshot of vulnerabilities, together with their features (weight, confidence level, etc.). The vendor's findings are compared or matched against a baseline snapshot of vulnerabilities present in the test bed at the time of the vendor's scan. A vendor score is computed as a weighted function of the matched vulnerabilities.

Brief Description of the Drawings

Further features and advantages of the disclosed subject matter will become apparent from the following detailed description taken in conjunction with the accompanying figures showing illustrative embodiments of the disclosed subject matter, in which:

FIG. 1 is a schematic illustration of a test site, which emulates a merchant site, for testing the capabilities of scanning vendors, in accordance with the present invention.

FIG. 2 illustrates an exemplary process for vendor compliance verification, in accordance with the present invention.

FIG. 3 is a schematic illustration of exemplary organizational groups and processes that may be involved in implementing process 200, in accordance with the present invention.

FIG. 4 illustrates further details of the organizational processes involved in implementing the process of FIG. 2, in accordance with the present invention. FIG. 5 is a schematic illustration of the dynamic characteristics of the accreditation system and process that can accommodate changes in technology and evolving vulnerabilities in or threats to payment-by-card infrastructure and data.

Description of the Invention Systems and methods for qualifying scanning vendors are provided. Vendor qualification procedures, which take into consideration the exploitability, severity, and weight of test vulnerabilities, are provided. In particular, scoring procedures for qualifying scanning vendors are provided are described. In an embodiment of the invention, the vendors may be scored according to an exemplary scoring system and procedures outlined in Appendix B.

The qualification of vendors can ensure compliance with security standards in the payment-by-card industry by enabling selection or certification of vendors who have shown acceptable levels of ability in identifying vulnerabilities. The vendor qualification systems and methods may be like the systems and methods described in commonly assigned International Patent Application No. PCT/US07/70709, incorporated by reference herein. An implementation of the invention presents vendors under test ("candidate vendors") with simulated data situations (e.g., an emulated merchant site) for scanning. The simulated data situations may include one or more vulnerabilities ("baseline vulnerabilities"), which the candidate vendor is expected to discover and identify. The baseline or test vulnerabilities may be assigned weights according to the criticality of the vulnerabilities. For example, a vulnerability that is easy to exploit or has a severe consequence if exploited may be assigned a high weight. The test vulnerabilities may be updated periodically to reflect contemporary threat situations. The emulation site may include a series of highly defined electronic environments that may include Known, Quantified and Ranked series of flaws, misconfigurations and errors.

Candidate vendors may be given access to an test environment to be evaluated and judged based on the quality of their ability to identify vulnerabilities in test environment. A candidate vendor may be given access to the electronic environments by the opening of a remote access e-gate for a test period of time in which the vendor is to discover as many Known, Quantified and Ranked flaws as the vendor can using the vendor's own testing systems and procedures. The remote e-gate may limit the number of candidate vendors that are allowed to access the simulated data situation at a given time. The remote e-gate may include network port, password access, and any other facilities necessary to permit a remote vendor to access the target systems. In another implementation, the test period may be unlimited with the remote e-gate staying open until the vendor has discovered a threshold number of vulnerabilities. The vendor may scan the simulated data situation to discover and identify vulnerabilities therein. The vendor's success (or lack of success) in identifying vulnerabilities may be evaluated and scored. (See Appendix B). The vendor may be qualified, at least in part, according to the scores it is given. In this way, merchants, service providers or other entities, may determine whether a particular scanning vendor is sufficiently skilled and competent to audit their systems. The industry is also assured that the evaluations by a qualified scanning vendor meet appropriate industry standards..

The vendor's capabilities may be evaluated (e.g., by a Validator) with respect to an established set of requirements. The criteria may be fixed or may vary according to the requirements of the tests. For example, fixed criteria may permit controlled comparisons of candidate vendors. Varying or changing criteria may permit candidate vendors to continually keep their skills up to date, so as to be able to identify the latest vulnerabilities.

Appendix A of commonly assigned International Patent Application No. PCT/US07/70709 lists an exemplary set of criteria. For convenience, Appendix A is reproduced herein. For further convenience, FIGS. 1-5 of International Patent Application No. PCT/US07/70709 and portions of the related description are reproduced herein.

A Scoring System (Appendix B) finds matches between a candidate vendor's findings and a validator's base of vulnerabilities. A vulnerability may include any flaw, misconfiguration issue or problem that is present in the simulated data situation or test bed at a given time, intentionally set or not.

Vulnerabilities are characterized and categorized by their features (e.g. severity, exploitability, time varying status) and accordingly assigned weights for scoring. Each vulnerability is accurately documented (e.g., via industry links). A vendor's test score may be based on consideration by weight of the vulnerabilities identified by the vendor in the simulated data situation. Certain vulnerabilities may be flagged as "showstoppers." A showstopper may include a vulnerability that may be required to be found by a scanning vendor, regardless of its importance/weight. In some cases a vulnerability may exist on one host only. In other cases the vulnerability might be present on several hosts. Such a vulnerabilities may be associated with a list of hosts on which it is present. Further, each host may be associated with a confidence level (e.g., present or potential) corresponding to a degree of certainty that the vulnerability does in fact exist on a given host.

For purposes of scoring, vulnerabilities may be grouped into categories and/or subcategories, for example, by commonality of features or according to the operating system or hardware device they are associated with. Each category/subcategory may be assigned a minimum percentage threshold, which the candidate vendor must reach for achieving a "category score." Further, each category/subcategory may be assigned relative weights according to its importance in regard to all other categories that lie in the same group. A baseline set of vulnerabilities (i.e., the Known, Quantified and Ranked flaws) presented in the test bed may include a snapshot of vulnerabilities, together with their features (weight, confidence level, etc.). The Scoring System reviews a scanning vendor's findings report against a list of vulnerabilities. The process may include creating a baseline at the time the scan occurred, matching vulnerabilities from the vendor's report against a baseline snapshot, and computing the vendor's test score. Matches between a vendor's report and the vulnerabilities baseline may be determined, for example, using industry links, and/or the name or description of the vulnerability. Whenever a match is found, the Scoring System can select the host(s) on which the vulnerability is found and the level of confidence reported by the vendor. Once a report has been reviewed, the Scoring System computes a vendor score. The score may be computed by vulnerability in each category ('vulnerability scoring"), and then grouping the vulnerability scores to produce category scores. Exemplary procedures for computing the vendor scores are outlined in Appendix B.

Vendors may be required to demonstrate their skill in data security validation trials. In the validation trials, vendor processes and services may be evaluated and benchmarked so that competent vendors may be certified or designated as approved for conducting the desired security scans and testing of entities. The validation trials may present vendors with simulated data situations that may include defined or exact vulnerabilities and established criteria to select them. The simulated data situations may be presented to vendors electronically via a simulation test site (e.g., FIG. 1, Emulated Merchant Test Site 110). The data simulations may be designed with weights assigned to specific vulnerabilities and with a defined methodology for assigning a certification score to a vendor seeking certification or approval. The defined scoring methodology may be rigorous and assigns scores based on assessments of the criticality of the vulnerabilities tested. Vendors may be certified or put on a list of approved vendors according to their certification scores.

Exemplary validation trials of the scanning vendors may be based on processes that rate performance of software security scanners, methods that rate the execution of passive penetration, and testing and methods that rate the execution of both on-site (i.e. customer site) and remote wireless scans. The validation trials may be advantageously designed to allow accurate determination of compliance to the scanning vendor standard and permit objective comparison of vendor capabilities. The validation trials may include or extend assessments of vendor capabilities to web applications, in addition to conventional payment network components. The validation trials of the scanning vendors may include a method to assess the performance of scanning vendors when conducting passive penetration testing at web sites. Passive penetration testing may be desirable, as industry entities cannot afford disruptive penetration testing.

Further, the validation trials may be designed flexibly so as to accommodate evolving payment-by-card industry technology. For example, the validation trials may include assessments of the performance of scanning vendors when conducting wireless security testing at web sites. As wireless security testing may require specific skills and may be closely dependent on the topology of the scanned site, a number of constrains may be applicable on security scanning. The validation trials structure and methodology may be designed accordingly. The simulated data situations used in the validation trials may include highly defined electronic environments, which may be designed to include a Known, Quantified and Ranked series of flaws, misconfigurations and errors. Additionally, the scope of testing may include both testing the security of both non-web network components as well as that of web applications and wireless configurations. The test systems may be configured using off-the-shelf hardware and software or may be designed using customized components, depending on the level of realism desired.

Vendors seeking certification (i.e. test subjects or candidate vendors) may be allowed to access the electronic environments through , for example, a remote access e-gate. The test subjects may be allotted a test period in which to discover the flaws using their own testing systems and procedures (e.g. their own Vulnerability Testing Systems). Their results are scored. The scoring system may be aligned with industry standards.

The technical part of the vendor certification or accreditation process may include an assessment of security skills of the scanning vendors enrolled in the program. Under the program, scanning vendors may be required to demonstrate that, by means of a remote test scan in an information technology (IT) infrastructure test bed, they are able to identify and report vulnerabilities commonly encountered in network environments. The IT infrastructure test bed may simulate data situations at members, merchants, TTPS, DSEs, and/or other payment-by-card industry entities or organizations. The Third Party Processor (TPP) may be any member service provider (MSP) who performs transaction and cardholder processing Program Services for one or more members. The Data Storage Entity (DSE) may include an organization, other than a member, merchant, or TPP that stores and has access to the account data of the financial institutions. Examples of DSEs may include, but are not limited to, Web hosting companies, payment gateways, terminal drivers and processors. Upon successful or satisfactory demonstration of their vulnerability identification skill, the vendors may receive a certification. To insure up-to-date certification, the vendors may be subject to a periodic (e.g. a yearly) revalidation process. A vendor Compliance Testing Program may be an important element of a security framework for payment card products used in the e-commerce space. The Vendor Compliance Testing Program requires members/entities to determine whether their merchants and services are compliant based on reports issued by approved vendors. Under an accreditation program, for example, a payment card association may provide testing services provided to scanning vendors. FIGS. 1-5 show structural, process and functional aspects of the accreditation program. Under the program, vendors seeking accreditation may demonstrate their scanning skills in an emulated test site 110 run by the payment card association. (See FIG. 1). An exemplary test site may emulate a merchant site. Vulnerabilities and mis-configurations may be deliberately introduced and may constitute a baseline of vulnerabilities. These baseline vulnerabilities may be stored in a security flaw database. This database may be updated regularly (See e.g., FIGS. 4 and 5). FIGS. 2 and 4 show an exemplary process for checking compliance of scanning vendor capabilities and services. FIG. 3 shows exemplary entities and organizations that may be involved in implementing the process of FIGS. 2 and 4. FIG. 3 also shows an exemplary relationship between entities and organizations and the interactions between these entities and organizations. FIG. 5 shows exemplary dynamic characteristics of the accreditation system and process that can accommodate changes in technology and evolving vulnerabilities in or threats to payment-by-card infrastructure and data.

In an implementation of the accreditation program, scanning vendors may be tested against an established set of criteria (e.g., Set A, which is listed in Appendix A). The test results are evaluated to determine whether the scanning vendors satisfy the established set of requirements to be accredited. With reference to FIG. 1, scanning vendors seeking accreditation or placement on the approved vendors list, may demonstrate their scanning skills in a test site 110. Test site 110, which may emulate a merchant site, may include emulations of cardholder data, wireless access points (WAPs), web applications, firewalls, and other features, components, and platforms of the merchant site. A baseline set of merchant site vulnerabilities and misconfigurations may be deliberately introduced in test site 110. In one example embodiment, the baseline may include state of the art e-commerce security flaws. The set may be updated regularly to keep up with new or changing threats in e- commerce and electronic communications. Test site 110 may be accessed locally or remotely (such as via the Internet) by vendors for application scanning and network scanning.

The candidate vendors are expected to discover and identify the baseline vulnerability set by scanning the test site. The establishment of a documented set of baseline requirements (e.g., Set A) and the provision of different accessibility options may allow vendors to be tested competitively on an equal basis, under the same or similar test circumstances. Candidate scanning vendors may be graded on the tests results, and accordingly successful vendors may be selected for approval. FIG. 2 shows an exemplary compliance verification process 200, which may be implemented for placing or certifying candidate vendors on an approved Scanning Vendors list. At step 210, vendor 10 may apply to testing entity 20 for compliance testing. At step 215, testing entity 20 may assess vendor 10's initial eligibility based, for example, on general business criteria. The business criteria may, for example, include the vendor's credit rating, history of work in the security arena, potential clients, etc. If eligible, vendor 10 may be registered as a candidate vendor for testing. At step 220, a vendor customer interface to the emulated merchant site may be enabled. At step 225, a scanning test may be set up. At step 230, the candidate vendor 10 may conduct test scans of the emulated sites, and may provide a scan report (e.g., a listing of the merchant site vulnerabilities and misconfigurations that it was able to identify) to testing entity 20. At step 235, testing entity 20 may evaluate the scan report and may assess the test results. At step 245, the testing entity 20 may provide notification of the test results, for example, via a letter of compliance, and/or web postings of the results.

FIG. 3 shows exemplary organizational groups and processes that may be involved in implementing process 200 to generate a Compliant Security Vendor list 390. The organizational groups in the testing entity may include, for example, a test team 310, a vulnerability management team 320 and an approval team 330. Other organizations may include a vulnerability update group that updates a database of vulnerabilities, a reporting group, etc. These teams may interact with other organizational groups (e.g., Legal, Steering committees, Fraud management groups and SDP program Groups, etc.). In practice, the organization of the groups may vary according to the requirements of the test program. Test team 310 may be the organizational group responsible for administering tests to vendors (using testing process 200). The vulnerability management team 320 may be responsible for the design and updating of testing process 200, for example, by updating a baseline vulnerability set in the emulated test site to reflect contemporary threat circumstances. The vulnerability management team 320 may have or develop vulnerability management process 270 for the design and updating of testing process 200.

Approval team 330 may be responsible for implementing approval process 250, which may include establishing suitable selection criteria and evaluation of scan reports generated by candidate vendor 10. According to the results of approval process 250, candidate vendors may be placed on a compliant security vendor list. Such a list may signify which vendors have satisfactorily passed the tests.

FIG. 4 is a schematic, which shows exemplary, interactive relationships between testing process 200, approval process 250 and vulnerability management process 270 for placing candidate vendors on an approved Scanning Vendors list. Testing Process 200 may include Test administration, IT management and Report review functions with reference to the tests given to vendors and the resulting reports. Testing Process 200 may be based on a pre-established Scanning Vendor standard. Such a standard may establish an industry-wide standard that all scanning vendors may be required to achieve before they are listed as approved. Approval Process 250 may be based on a pre-established Approval policy. Of course, in practice, the policies and standards may be updated on the fly. Vendors may be periodically retested to receive the most up to date certification. Approval process 250 may begin with candidate vendor registration. Registration may include, for example, vetting, vendor administration, approval determination, financial management, development strategy, scoring setting, and other functions. Vulnerability management process 270, which may be based on a pre-established vulnerability management policy, may identify and update the baseline set of vulnerabilities in the test for scanning vendors. Vulnerability management process 270 may include, for example, vulnerability maintenance, IT selection, and scoring system maintenance functions. Testing Process 200 and approval process 250 may exchange test requests and internal reports and share commonly accessible vendor data stored in a database.

Similarly, Testing Process 200 and Vulnerability management process 250 may exchange and share commonly accessible vulnerability data stored in a database.

FIG. 6 shows another exemplary role of vulnerability management process 250 in vendor compliance testing and approval. Vulnerability management process 250 may involve monitoring public domain information to identify nascent, developing and receding threats and security flaws in e-commerce communications. The information may be analyzed on a dynamic cycle or a response to major event basis to determine when the baseline vulnerabilities set used for vendor evaluations should be modified or updated to reflect contemporary circumstances. New vulnerabilities may be identified from candidate vulnerabilities for addition to the set of active vulnerabilities. Similarly, obsolete vulnerabilities may be removed from the active set of vulnerabilities.

In accordance with the present invention, software (i.e., instructions) for implementing the aforementioned systems and methods can be provided on computer- readable media. It will be appreciated that each of the steps (described above in accordance with this invention), and any combination of these steps, can be implemented by computer program instructions. These computer program instructions can be loaded onto a computer or other programmable apparatus to produce a machine, such that the instructions, which execute on the computer or other programmable apparatus create means for implementing the functions of the aforementioned systems and methods. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the functions of the aforementioned systems and methods. The computer program instructions can also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions of the aforementioned systems and methods. It will also be understood that the computer- readable media on which instructions for implementing the aforementioned systems and methods are be provided, include without limitation, firmware, microcontrollers, microprocessors, integrated circuits, ASICS, and other available media.

It will be understood that the foregoing is only illustrative of the principles of the invention and that various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims

What is claimed is:

1. A method for scoring a scanning vendor's performance in a qualification trial in which the scanning vendor conducts security scan of a test bed emulation of a payment-by-card industry entity, the method comprising:

establishing a list of vulnerabilities, wherein a vulnerability is any flaw, misconfiguration issue or problem that is present in the test bed at a given time,

assigning an importance weight to each vulnerability for purposes of scoring the scanning vendor's performance;

creating a baseline snapshot of vulnerabilities present in the test bed at the time the scanning vendor conducts the security scan of a test bed;

matching vulnerabilities identified by the vendor scan with vulnerabilities in the baseline snapshot,

computing a vendor performance score as a function of the weights of the matched vulnerabilities and the weights of the all the vulnerabilities in the baseline snapshot.

2. The method of claim 1, wherein assigning an importance weight to each vulnerability comprises assigning a weight in consideration of features of the vulnerability, wherein the considered features include one or more of a severity level of the vulnerability, an exploitability level of the vulnerability, a frequency of occurrence of the vulnerability, and an applicability of the vulnerability in a particular industry.

3. The method of claim 1 , wherein assigning an importance weight to each vulnerability comprises assigning a showstopper attribute to the vulnerability indicating that it is required to be found by the scanning vendor regardless of its importance weight, and wherein computing the vendor performance score comprises determining if the matched vulnerabilities included the required vulnerability.

4. The method of claim 1, wherein establishing a list of vulnerabilities further comprises associating each listed vulnerability with a list of hosts on which it is present.

5. The method of claim 4, further comprising associating each listed host with a confidence level reflecting a degree of certainty that the vulnerability does in fact exist on a given host,

wherein matching vulnerabilities identified by the vendor scan with vulnerabilities in the baseline snapshot further comprises consideration of the confidence levels of the vulnerabilities, and wherein computing a vendor performance score further comprises consideration of the confidence levels of the vulnerabilities.

6. The method of claim 1, wherein one or more of the listed vulnerabilities is associated with an industry link, and wherein matching vulnerabilities identified by the vendor scan with vulnerabilities in the baseline snapshot further comprises utilizing the industry link for vulnerability matching.

7. The method of claim 1, further comprising associating each listed vulnerability with a status indicating its stage in a vulnerability life cycle, and wherein computing the vendor performance score comprises consideration of the statuses of the vulnerabilities.

8. The method of claim 1 , further comprising:

grouping the listed vulnerabilities into one or more categories, each category comprising one or more sub-categories;

assigning an category weight to each category/sub-category representing its importance in regard to all other categories/sub-categories; and

assigning each category/sub-category a threshold percent or value representing the minimum number of vulnerabilities therein that are required to be found by the scanning vendor to pass the qualification trial,

wherein computing the vendor performance score further comprises consideration of the statuses of the category weights and thresholds.

9. A computer readable medium comprising a set of program instructions that are executable to perform the steps of:

establishing a list of vulnerabilities, wherein a vulnerability is any flaw, misconfiguration issue or problem that is present on the test bed at a given time,

assigning an importance weight to each vulnerability for purposes of scoring the scanning vendor's performance;

creating a baseline snapshot of vulnerabilities present in the test bed at the time the scanning vendor conducts the security scan of a test bed;

matching vulnerabilities identified by the vendor scan with vulnerabilities in the baseline snapshot,

computing a vendor performance score as a function of the weights of the matched vulnerabilities and the weights of the all the vulnerabilities in the baseline snapshot.

10. The computer readable medium of claim 9 further comprising a set of program instructions that are executable to perform the step of assigning an importance weight to each vulnerability by assigning a weight in consideration of features of the vulnerability, wherein the considered features include one or more of a severity level of the vulnerability, an exploitability level of the vulnerability, a frequency of occurrence of the vulnerability, and an applicability of the vulnerability in a particular industry.

11. The computer readable medium of claim 9, wherein assigning an importance weight to each vulnerability comprises assigning a showstopper attribute to the vulnerability indicating that it is required to be found by the scanning vendor regardless of its importance weight, and wherein computing the vendor performance score comprises determining if the matched vulnerabilities included the required vulnerability.

12. The computer readable medium of claim 9, wherein establishing a list of vulnerabilities further comprises associating each listed vulnerability with a list of hosts on which it is present.

13. The computer readable medium of claim 12 further comprising a set of program instructions that are executable to perform the step of associating each listed host with a confidence level reflecting a degree of certainty that the vulnerability does in fact exist on a given host,

wherein matching vulnerabilities identified by the vendor scan with vulnerabilities in the baseline snapshot further comprises consideration of the confidence levels of the vulnerabilities, and wherein computing a vendor performance score further comprises consideration of the confidence levels of the vulnerabilities.

14. The computer readable medium of claim 9, wherein one or more of the listed vulnerabilities is associated with an industry link, and wherein matching vulnerabilities identified by the vendor scan with vulnerabilities in the baseline snapshot further comprises utilizing the industry link for vulnerability matching.

15. The computer readable medium of claim 9 further comprising a set of program instructions that are executable to perform the step of associating each listed vulnerability with a status indicating its stage in a vulnerability life cycle, and wherein computing the vendor performance score comprises consideration of the statuses of the vulnerabilities.

16. The computer readable medium of claim 9 further comprising a set of program instructions that are executable to perform the steps of:

grouping the listed vulnerabilities into one or more categories, each category comprising one or more sub-categories;

assigning an category weight to each category/sub-category representing its importance in regard to all other categories/sub-categories; and

assigning each category/sub-category a threshold percent or value representing the minimum number of vulnerabilities therein that are required to be found by the scanning vendor to pass the qualification trial,

wherein computing the vendor performance score further comprises consideration of the statuses of the category weights and thresholds.