Chapter 2: Introduction to New and Expanded Material

This document contains considerable new material and material expanded from previous versions of the voting standards. This section provides an introduction to and overview of major features of the VVSG, those being

• Organization of the VVSG, requirements structure, and classes;
• Usability performance metrics;
• Expanded human factors coverage;
• Software Independence, Independent Voter-Verifiable Records voting systems, and the Innovation Class;
• Open-ended vulnerability testing and expanded security coverage;
• Treatment of COTS in voting system testing;
• End-end testing for accuracy and reliability;
• New metric for voting system reliability; and
• Expanded core requirements coverage.

2.1 The New Structure of the VVSG

The VVSG structure is markedly different from the structure of previous versions. First, the VVSG should be considered as a foundation for requirements for voting systems; it is a foundation that provides precision, reduces ambiguity, eliminates repeated requirements, and provides an avenue for orderly change, i.e., the addition of new types of voting devices or voting variations.

It was necessary to focus on providing this robust foundation for several reasons.  First, previous versions suffered from ambiguity, which resulted in a less-robust testing effort.  In essence, it has been more difficult to test voting systems when the requirements themselves are subject to multiple interpretations.  This new version should go a long way towards reducing that ambiguity.

Secondly, there are simply more different types of voting devices than anticipated by previous versions, and new devices will continue to be marketed as time goes by.  The VVSG provides a strong organizational foundation so that existing devices can be unambiguously described and development of new devices can proceed in an orderly, structured fashion.

2.1.1 VVSG Standards Architecture

The VVSG has been reorganized to bring it in line with applicable standards practices of ISO, W3C and other standards-creating organizations.  It contains three volumes or “Parts” for different types of requirements:

Part 1, Equipment Requirements, provides guidelines for manufacturers to produce voting systems that are secure, accurate, reliable, usable, accessible, and fit for their intended use. Requirements in VVSG 2005 that were ambiguous have been clarified.  In those cases where no precise replacement could be determined and no testing value could be ascribed, requirements have been deleted.

Part 2, Documentation Requirements, is a new section containing documentation requirements separate from functional and performance requirements applying to the voting equipment itself.  It contains requirements applying to the Technical Data Package, the Voting Equipment User Documentation, the Test Plan, the Test Report, the Public Information Package, and the data for voting software repositories.

Part 3, Testing, contains requirements that apply to the national certification testing to be conducted by non-governmental certified testing laboratories. It has been reorganized to focus on test methods and to avoid repetition of requirements from the product standard. Although different testing specialties are likely to be subcontracted to different laboratories, the prime contractor must report to the certifying authority on the conformity of the system as a whole.

The requirements in these Parts rely on delimitation and strict usage of certain terms, included in Appendix A, Definition of Words with Special Meanings.  This covers terminology for standardization purposes that must be sufficiently precise and formal to avoid ambiguity in the interpretation and testing of the standard. Terms are defined to mean exactly what is intended in the requirements of the standard.  Note: Readers may already be familiar with definitions for many of the words in this section, but the definitions here often may differ in small or big ways from locality usage because they are used in special ways in the VVSG. 

The VVSG also contains a table of requirement summaries, to be used as a quick reference for locating specific requirements within sections/subsections.  Appendix B contains references and end notes.

2.1.2 Voting System and Device Classes

Voting system and device classes are new to the VVSG. Classes in essence form profiles of voting systems and devices. They are used as fields in requirements to connote the scope of the requirements. For example, Figure 2-1 shows the high-level device class called vote-capture device. There are various requirements that apply to vote-capture device; this means that all vote-capture devices must satisfy these requirements (e.g., for security, usability, etc.). There are also requirements that apply more specifically to, say, IVVR vote-capture devices and those explicit devices underneath it, such as VVPAT. These devices inherit the requirements that apply to vote-capture device, that is, they must satisfy all the general vote-capture device requirements as well as the more specific requirements that apply. In this way, new types of specific vote-capture devices can be added in the future; they must satisfy the general requirements that all Vote-capture devices are expected to satisfy, but at the same time they can satisfy specific requirements that only apply to the new device. This structure assists in unambiguously making it clear to manufacturers and test labs which requirements apply to ALL vote-capture devices, for example, as opposed to which requirements apply specifically to just VVPAT. This structure also allows for the addition or modification of new or existing device requirements without affecting the rest of the standard.

Figure 2-1 Voting device class hierarchy

2.1.3 Requirements Structure

Requirements are now very specific to either a type of voting variation or a type of voting device (as stated in the previous section, the voting device can be a general profile of certain types of voting devices or be a profile of a more specific voting device). The requirements contain expanded description text and more precise language to make requirements explicit and to indicate the general test method to be used by the test lab for determining whether the requirement is satisfied in the voting system under test. As appropriate, the requirement also contains a reference to versions of the requirement in previous standards (e.g., VVSG 2005 or the 2002 VSS) so as to show its genesis and to better convey its purpose.

2.1.4 Strict Terminology

The terminology used in the VVSG has been considered carefully and is used strictly and consistently.  In this way, requirements language can be made even more clear and unambiguous.  Hypertext links are used throughout the VVSG for definitions of terminology to reinforce the importance of understanding and using the terminology in the same way.

However, it is important to understand that the terminology used in the VVSG is specific to the VVSG.  An effort has been made to make sure that the terms used in the VVSG mean essentially the same thing as used in other contexts, however at times the definitions in the VVSG may vary in big or small ways.

Figure 2-2 illustrates the relationships and interaction between requirements, device classes, and types of testing from Part 3, all in the framework of strictly used terminology.

Figure 2-2 Interaction between requirements, definitions, and parts of the VVSG

2.2 Usability Performance Requirements

Usability is conventionally defined as "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" [ISO98a]. In VVSG 2005, the usability guidelines relied on three assessment methods:

1. Checking for the presence of certain design features which are believed to support usability, and for the absence of harmful design features;
2. Checking for the presence of certain functional capabilities which are believed to support usability; and
3. Requiring manufacturers to perform summative usability testing with certain classes of subjects and to report the results. However, the VVSG 2005 reporting requirements do not specify the details of how the test is designed and conducted.

While all these help to promote usability, methods 1 and 2 are all somewhat indirect methods. The actual "effectiveness, efficiency and satisfaction" of voting systems are never evaluated directly in the 3^rd method.

This version of the VVSG uses a new method based on summative usability testing that directly addresses usability itself, i.e., measured mainly in how accurately voters cast their ballot choices. The features of this new method include:

• The definition of a standard testing protocol, including a test ballot, set of tasks to be performed, and demographic characteristics of the test participants.  The protocol supports the test procedure as a repeatable controlled experiment.
• The use of a substantial number of human subjects attempting to perform those typical voting tasks on the systems being tested, in order to achieve statistically significant results.
• The gathering of detailed data on the subjects' task performance, including data on accuracy, speed, and confidence.
• The precise definition of the usability metrics to be derived from the experimental data.
  
• The definition of effectiveness benchmarks against which systems will be evaluated.

Obviously, the implementation of such complex tests is more difficult than simply checking design features. However, performance-based testing using human subjects yields the most meaningful measurement of usability because it is based on their interaction with the system's voter interface, whereas design guidelines, while useful, cannot be relied upon to discover all the potential problems that may arise. The inclusion of requirements for performance testing in these Guidelines advances the goal of providing the voter with a voting system that is accurate, efficient, and easy to use.

2.3 Expanded Usability and Accessibility Coverage

In addition to usability performance metrics, the treatment of human factors, i.e., usability, accessibility, and privacy, has been expanded considerably. Table 2-1 summarizes the new and expanded material.

2.4 Software Independence

Software independence [Rivest06] means that an undetected error or fault in the voting system’s software is not capable of causing an undetectable change in election results. All voting systems must be software independent in order to conform to the VVSG.

There are essentially two issues behind the concept of software independence, one being that it must be possible to audit voting systems to verify that ballots are being recorded correctly, and the second being that testing software is so difficult that audits of voting system correctness cannot rely on the software itself being correct. Therefore, voting systems must be ‘software independent’ so that the audits do not have to trust that the voting system’s software is correct; the voting system must provide proof that the ballots have been recorded correctly, e.g., voting records must be produced in ways in which their accuracy does not rely on the correctness of the voting system’s software.

This is a major change from previous versions of the VVSG, because previous versions permitted voting systems that are software dependent, that is, voting systems whose audits must rely on the correctness of the software. One example of a software dependent voting system is the DRE, which is now non-conformant to this version of the VVSG.

2.4.1 Independent voter-verifiable records

The VVSG requires that, to be software independent, all voting systems include an IVVR vote-capture device, that is, a vote-capture device that uses independent voter-verifiable records (IVVR). IVVR can be audited independently of the voting system software but do not necessarily have to be paper-based. IVVR relies on voter-verification, that is, the voter must verify that the electronic record is being captured correctly by examining a copy that is maintained independently of the voting system’s software, i.e., the IVVR.

Voter-verifiable paper records (VVPR) is a form of IVVR that is paper-based. Currently, the voting systems that can satisfy the definition of software independence use VVPR, such as with

• optical scanners used in conjunction with
  • manually-marked paper ballots or
  • an EBP or EBM; and
• VVPAT.

Figure 2-3 illustrates this in a tree-like structure. At the top of the tree is software independence; as stated previously all voting systems that are conformant to the VVSG must be software independent. One route to achieving software independence is to use IVVR. The VVSG contains requirements for IVVR, of which VVPR is one (currently the only) type. If different types of IVVR are developed that do not use paper, systems that use them can also be conformant to the VVSG “as is.” In other words, new types of IVVR that do not use paper are already “covered” by the IVVR requirements in the VVSG; new requirements will not necessarily need to be added.

Figure 2-3 Voting systems that can conform to current requirements in the VVSG

2.4.2 The Innovation Class

Use of IVVR is currently the only method specified by requirements in the VVSG for achieving software independence. Manufacturers that produce systems that do not use IVVR must use the Innovation Class as a way of proving and testing conformance to the VVSG. The innovation class is for the purpose of ensuring a path to conformance for new and innovative voting systems that meet the requirement of software independence but for which there may not be requirements in the VVSG. Technologies in the innovation class must be different enough to other technologies permitted by the VVSG so as to justify their submission. Technologies in the innovation class must meet the relevant requirements of the VVSG as well as further the general goals of holding fair, accurate, transparent, secure, accessible, timely, and verifiable elections.

A review panel process, separate from the VVSG conformance process, will review innovation class submissions and make recommendations as to their eventual conformance to the VVSG.

2.5 Open-Ended Vulnerability Testing

The goal of open-ended vulnerability testing (OEVT) is to discover architecture, design and implementation flaws in the system which may not be detected using systematic functional, reliability, and security testing and which may be exploited to change the outcome of an election, interfere with voters’ ability to cast ballots or have their votes counted during an election, or compromise the secrecy of vote. The goal of OEVT also includes attempts to discover logic bombs, time bombs or other Trojan Horses that may have been introduced in the system hardware, firmware or software for said purposes. Open-ended vulnerability testing (OEVT) relies heavily on the experience and expertise of OEVT team members, their knowledge of the system, its component devices and associated vulnerabilities, and the team’s ability to exploit those vulnerabilities.

2.6 Expanded Security Coverage

In addition to software independence and OEVT, the treatment of security in voting systems has been expanded considerably. There are now detailed sets of requirements for eight aspects of voting system functionality and features, as shown in Table 2-2.

2.7 Treatment of COTS in Voting System Testing

To clarify the treatment of components that are neither manufacturer-developed nor unmodified COTS (commercial off-the-shelf software/hardware) and to allow different levels of scrutiny to be applied depending on the sensitivity of the components being reviewed, different subdivisions of COTS have been identified, with various requirements scoped to the new terminology. For example, a COTS operating system may not require source code review, but configuration files that support the configuration of the operating system would require test lab review.

The way in which COTS is tested has also changed; the manufacturer must deliver the system to test without the COTS installed, and the test lab must procure the COTS separately and integrate it. If the integration is successful, the COTS can safely be assumed to be unmodified.

2.8 End-to-End Testing

The testing specified in previous versions of the VVSG for accuracy and reliability is not required to be end-to-end but may bypass significant portions of the system that would be exercised during an actual election, such as the touch-screen or keyboard interface. This resulted in the voting system not being tested thoroughly for reliability or accuracy, thus this practice is now prohibited in this version of the VVSG. For example, if a tabulator is specified to count paper ballots that are manually-marked with a specific writing utensil, it is not valid to substitute ballots that were mechanically marked by a printer. Devices or software that closely and validly simulate actual election use of the system are permissible.

2.9 Reliability

The metric for reliability has been changed from Mean Time Between Failure (MTBF) to a failure rate based on volume that varies by device class and severity of failure (failures are equipment breakdowns, including software crashes, such that continued use without service or replacement is worrisome to impossible). In this version of the VVSG, there are now different failure rates per device, which permits more refined testing and eliminates the previous “one size fits all” approach.

Additionally, a volume test is now included that is analogous to the California Volume Reliability Testing Protocol. This test simulates actual election conditions and will better assess overall reliability and accuracy.

Reliability, accuracy, and probability of misfeed for optical scanners are now assessed using data collected through the course of the entire test campaign, including the volume testing. This increases the amount of data available for assessment of conformity to these performance requirements without necessarily increasing the duration of testing.

2.10 Expanded Core Requirements Coverage

The general core requirements for voting systems have been expanded greatly. In addition to the already noted improvements in COTS coverage, end-to-end testing for accuracy and reliability, and the new reliability metric, the following topics in Table 2-3 have been added or expanded.