Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Software installation involves two steps:
Generating software images via Trusted Build: Trusted Build
Imaging machines with those images: Imaging Machines
Operating system: Debian 12
RAM: 8GB minimum
Chip: x86-64 / amd64 CPU architecture, Intel i5 or higher
Storage: 250GB minimum
Debian 12 will need to be installed on the build machine. You can find detailed instructions for that process under . Once that is complete, we can install the VotingWorks build system, .
Throughout the rest of this document, substitute <inventory-name> with the inventory provided by VotingWorks.
At this point, you should have two separate VMs: debian-<inventory-name>-<apt-snapshot-date>-<debian-release-name>and online. You can confirm this by running:
The VotingWorks build process consists of three distinct build phases: , , and . The build process uses an inventory definition containing all the necessary information for a Trusted Build. All three phases are executed within a separate virtual machine (VM) managed by the tool, running on a Debian 12 operating system.
In the online phase, the build process utilizes a base Debian 12 VM with network access enabled. All necessary code repositories and build tools are securely retrieved for use during the offline phase.
In the offline phase, the build process utilizes a clone of the online VM with network access disabled.
In the final configuration phase, the build process utilizes clones of the offline VM to prepare images for specific machine types, i.e., VxAdmin, VxCentralScan, VxMarkScan, and VxScan.
After completing the final configuration phase, an unlocked installation image has been created. While this image is not appropriate for production use, it can be used for testing.
For a final production image, an additional step is required to securely sign the installation image for use with Secure Boot enabled systems, described under .
sudo apt install -y git
mkdir ~/code && cd ~/code
git clone https://github.com/votingworks/vxsuite-build-system
cd ~/code/vxsuite-build-system
./scripts/tb-initialize-build-machine.sh <inventory-name>
./scripts/tb-clone-images.sh <inventory-name>sudo virsh list --allTo install Debian 12 on the build machine (VxBuild), you will need to download the latest Debian 12 amd64 installer file here: latest Debian release.
You should use a blank USB drive to create the install drive. The process described below will wipe any existing content.
To create a USB install drive with the downloaded ISO file:
Ensure that the USB drive is available to the system. By default, it tries to attach as the /dev/sda device. An easy way to verify this is via the following command:
If the USB drive is not attached as the /dev/sda device, that’s ok. Simply replace /dev/sda with the device that your USB drive did attach to.
NOTE: The command below should not include any number as part of the device path. For example, if the above command returns /dev/sda1, you must use /dev/sda as the path in the next command.
Now that you know the correct device path, you can create the USB install drive via the following command:
NOTE: At the time of this writing, the latest stable release is 12.8. Please update to the appropriate file name if you're using a newer version.
Once the above command completes, you can safely remove the USB install drive from your system.
Before powering VxBuild on, insert the USB install drive created in the previous step.
After powering VxBuild on, begin pressing F12 until it enters the boot menu. Boot the system from the USB install drive.
You will be presented with the following screen. Select "Graphical install" and press enter.
Select your preferred language and click "Continue".
Select your location and click "Continue".
Select your keyboard layout and click "Continue".
Select your network device (may differ from screenshot) and click "Continue".
If using a wired connection, it will configure automatically.
If using a wireless connection, select a network, select WPA/WPA2 PSK, and click "Continue". Enter your wireless network password and click "Continue".
Enter "VxBuild" for the hostname and click "Continue".
Leave the domain name blank and click "Continue".
Set the root user password and click "Continue".
Enter "Vx" for the full name for the new user and click "Continue".
Enter "vx" as the username and click "Continue".
Enter a password for the "vx" user and click Continue.
Select your timezone and click "Continue".
Select "Guided - use entire disk" and click "Continue".
Select the "/dev/nvme0n1" disk and click "Continue".
Select "All files in one partition" and click "Continue".
Select "Finish partitioning and write changes to disk" and click "Continue".
Select "Yes" and click "Continue".
The base OS installation will now begin. During the installation you will be asked to answer questions related to package management.
Select "United States" and click "Continue".
Select "deb.debian.org" and click "Continue".
Leave the proxy information blank and click "Continue".
The "vx" user needs to be granted sudo privileges related to configuring and initializing the build environment tools.
Open a terminal window.
As the "vx" user, you will temporarily log in to the "root" account.
You will see your terminal window prompt change to root@VxBuild. To grant the "vx" user sudo privileges, run the following command as root.
You will see your terminal window prompt change to vx@VxBuild. You are now the "vx" user instead of the "root" user. To confirm sudo privileges, run the following command:
The command should return "root". This confirms sudo privileges have been granted correctly.
By default, newly created VMs have networking enabled. To ensure the offline VM does not have access to the internet, the networking link is disabled before the VM is ever used. This is accomplished by editing the VM's configuration (found in /etc/libvirt/qemu/offline.xml) and setting the link state to down. This functionality can be seen in the vxsuite-build-system repo, in .
If an offline VM exists, the XML configuration file is updated to set the link state to down. After that, the offline VM is explicitly re-defined from this updated XML configuration file. These steps always execute, even if the network link state is already set to down.
Any VM cloned from this offline VM will also have a disabled network since those settings are inherited from the original VM.
You will be asked to configure "popularity-contest". Select "No" and click "Continue".
The installation process will continue. Once it completes, you will be presented with a final screen confirming that it was successful.
Remove the USB drive and click "Continue".
The system will reboot and automatically start Debian 12.
Log in with the "vx" user and password you created during the installation process.
Several basic configuration prompts will be displayed. Select "Next".
Select your keyboard preference and click "Next".
Turn off Location Services and click "Next".
Do not configure any online accounts. Select "Skip".
Click the "Start Using Debian GNU/Linux" button. The dialog will be closed.
Virt Manager uses a local bridge network to connect to VMs. If you receive an error when starting a VM that says "Error starting domain: Cannot get interface MTU on 'virbr0': No such device", run the following command in the terminal on the build machine:
You should now be able to return to the VM and start it without issue.
sudo virsh net-start defaultlsblk /dev/disk/by-id/usb*part* --noheadings --output PATHdd if=/path/to/debian-12.8.0-amd64-netinst.iso of=/dev/sda bs=4M && syncsu -
<Enter root user password>echo "vx ALL=NOPASSWD: ALL" > /etc/sudoers.d/vx
exitsudo whoamiTo install an image on a VotingWorks component, i.e., to image a machine, you need a vx-iso USB drive with one or more VotingWorks application images stored in the "Data" partition. There is also a "Keys" partition that can optionally contain VotingWorks Secure Boot public keys, necessary if a machine hasn't had these keys installed yet.
USB drives used as vx-iso drives must be zeroed out before first use. This step will also ensure the USB drive is empty and no longer contains any previous data prior to use as installation media. You can zero out a drive with the following command, substituting /dev/sdX with the appropriate path to the USB you are using, e.g. /dev/sda
sudo dd if=/dev/zero of=/dev/sdX bs=8M && sudo syncDrives provided by VotingWorks are already initialized with this process and contain the appropriate vx-iso release for installing an application image. You will only need to ensure the appropriate application image(s) are copied to the vx-iso USB drive. If you are creating your own vx-iso USB drive, please contact VotingWorks for assistance.
For this example, we will assume that you have a signed vxadmin image on the build machine. It will be located in the root user's home directory and be named vxadmin-signed.img.lz4
We will also assume the USB drive is automatically mounted by the vx user, with the Data partition located at:
As the vx user, first verify the USB is mounted at the path above. An easy way to test is with the following command:
That command should return without an error. It may list one or more images if they were already on the USB drive. That's ok. (You can delete any existing images from the USB drive if you like, but it is not necessary.)
Since the signed image file is located in the root user's home directory by default, we'll use that path. As mentioned earlier, we will use a vxadmin image for this example. That path would be:
To copy the image to the USB drive, run the following command as the vx user:
Once the copy and sync completes, you can eject the USB drive and remove it. It is now ready to image a machine. Repeat this process with any other vx-iso USB drives and VotingWorks image files as required.
This section walks through the steps to install a Trusted Build image on a VotingWorks component using .
Note: If you are installing a VotingWorks application for use in an election, you are required to create a software installation record. The details of this software installation record can be found here:
/media/vx/Datals /media/vx/Data/~root/vxadmin-signed.img.lz4sudo cp ~root/vxadmin-signed.img.lz4 /media/vx/Data/ && sudo syncFirst make sure that you've prepared USB drives for imaging, following the instructions under Preparing USB Drives for Imaging. Then follow these steps:
Power off the machine.
Insert the vx-iso USB drive into the system. If this is a VxMarkScan or a VxScan, connect a keyboard as well. If there aren't enough ports available, use a USB hub as provided by VotingWorks.
Power on the machine to begin booting vx-iso.
By default, VotingWorks systems should boot from the USB. If not, you will need to select the USB drive from a BIOS or Boot Menu. For central system components, you can access the Boot Menu by pressing F9 during the boot sequence. For other components, please reach out to VotingWorks for assistance.
Select "Install Image". You can navigate vx-iso with the keyboard. This option will be auto-selected after 30 seconds.
If the machine already has Secure Boot keys installed, it should not prompt you to install keys. If it does for some reason, you should reach out to VotingWorks for assistance. Only if you know the keys need to be installed should you opt to install them.
The application image(s) on the vx-iso USB drive will be displayed. Select the number that identifies the correct image. In the event there is only one image, it will be automatically selected.
The imaging process will begin automatically after 10 seconds.
Once imaging completes, remove the USB drive as prompted. The system will then automatically reboot.
After rebooting, the system will perform an automatic encryption of the var filesystem. If Secure Boot was not enabled when the image was installed, you'll see a note about needing to enable Secure Boot. The machine will auto-boot you into the BIOS. Once Secure Boot has been enabled and the system reboots, the encryption process should complete successfully.
The /var partition should encrypt and expand, followed by a reboot. If this is the first time that a VotingWorks application has been installed to a machine, you should find yourself in the . Proceed to that section. If the machine has been previously configured with a certified VotingWorks application of the same type, the will be skipped.
Note: VotingWorks system software will be installed to the directory path: /vx/code/vxsuite
The underlying storage for /vx/code/vxsuite will depend on the hardware type. By default, NVMe storage is used. In the event an NVMe storage device is not available, an eMMC storage device will be used.
To create the offline VM with networking disabled, run the following command on the build machine:
cd ~/code/vxsuite-build-system
./scripts/tb-create-offline-vm.shOpen virt-manager if not already open:
sudo virt-managerDouble click the offline VM.
Press the start button ▶️.
Once the VM has initialized, log in with username vx and password votingworks.
To ensure that the console displays correctly, select "View" > "Resize to VM".
In the VM terminal window, validate that networking is automatically disabled in the offline VM using the following command:
This command should either time out or immediately error with a failure in name resolution.
To execute the offline phase, run the following commands:
Once the offline phase completes, the base VotingWorks application has been built. You can now shut down the offline VM:
The VotingWorks build system, , utilizes many different tools to install, configure, and build VotingWorks applications. To ensure the integrity of the build system and tools used, all third-party tools are verified against known hashes, checksums, or digital signatures. This appendix provides a description of the mechanism used to verify each tool and any additional resources those tools may be responsible for providing during the build process.
In addition to verifying third-party tools against known hashes, checksums, and digital signatures during the build process, a report of COTS tools, with applicable hashes/checksums, is provided to the VSTL for independent verification and is available . These tools are defined within the for each release of VotingWorks applications, and only those tools are installed during builds.
APT
Debian's package manager, apt, is built on a secure framework described here: .
The VotingWorks build process relies on this fundamental tool to install many of the system level tools required to build our applications. As an added measure of security, all packages are pinned to specific versions to ensure a consistent build environment over time.
To further ensure that consistent versions of dependencies and transitive dependencies are used, we create our own VotingWorks-hosted apt snapshot to pull from.
If this is the first time that a VotingWorks application has been installed to a machine, the machine will boot into a basic configuration wizard. If the machine has been previously configured with a certified VotingWorks application of the same type, this wizard will be skipped.
The majority of the steps are self-explanatory, but "Step 1: Set Machine ID" and "Step 4: Create Machine Cert" require some extra clarification.
It is important that the machine ID be unique for each machine. Many machines have a physical placard on them indicating the machine ID. That is the ID that should be used here.
To finalize an image for production use, it's necessary to sign the image with the VotingWorks Secure Boot keys so that the image can be used on a Secure-Boot-enabled machine. This process requires the use of sensitive keys and a passphrase that should only ever be known to and used by VotingWorks. Since the Trusted Build process is performed by a third-party vendor (SLI), a process to securely sign the image without compromising the keys or passphrase is needed. This document describes that process.
Once SLI has created a Trusted Build image, the image and its VM definition (an XML file) must be securely provided to VotingWorks. To transfer the image and definition, SLI will upload to a shared S3 bucket only accessible to SLI and VotingWorks. This access is controlled via IAM permission policies.
NOTE: For documentation purposes, we will use a Trusted Build image named vxadmin. Replace that with the appropriate image name, as necessary.
Python's package manager, pip, does not verify package integrity by default. However, secure installation is possible via the method described here: pip: Secure installs.
The VotingWorks build process implements the above method in the tb-install-ansible.sh script. The required packages are listed, along with their hash, in unique requirements files based on the operating system version and architecture. You can see an example here: Debian 12 x86 pip requirements.
RubyGems
Ruby's package manager, gem, does not verify package integrity by default. However, the official gem repository provides SHA256 checksums for hosted packages. You can see an example here: FPM 1.15.1.
The VotingWorks build process requires the version and SHA256 checksum for any installed gems. Once the gem file has been downloaded, it is checked against this checksum to verify its integrity. You can see that checksum verification in the rubygems playbook here: rubygems.yaml.
Rust
By default, Rust is installed over an active internet connection. Since that method is not available during the offline phase of the VotingWorks build process, we use a standalone, offline installer binary provided by Rust. To verify the integrity of the installer, the downloaded file is verified against the officially published checksum that can be found in each version's official manifest. You can see that checksum verification in the rust playbook here: rust.yaml.
Cargo
Cargo is the Rust package manager. It provides secure installs via the combination of a manifest listing packages and their versions, along with a lock file containing the checksum for each package. That method is described here: Cargo.toml vs. Cargo.lock.
Node
Node is typically installed via package managers like apt, or via a direct install from a specific version. We do not install Node via apt since we explicitly version it. As a result, we download the appropriate installer from the official Node repository. That downloaded file is then checked against the officially provided checksum. You can see that checksum verification in the node playbook here: node.yaml.
npm
npm is the default Node package manager. It is installed as part of the Node install previously described.
Packages installed via npm are checked against the officially provided checksums for each package. You can see that checksum verification in the node playbook here: node.yaml.
pnpm
pnpm is another package manager for the Node ecosystem. It is installed via npm as previously described to ensure its integrity.
Packages installed via pnpm use a lockfile: pnpm-lock.yaml. This lockfile includes the version and checksum to verify integrity when packages are installed during the build process. In addition, the VotingWorks build process explicitly requires the use of --frozen-lockfile to ensure that only the packages explicitly required are installed.
Yarn
Yarn is another package manager for the Node ecosystem. It is installed via npm as previously described to ensure its integrity.
Packages installed via yarn use a lockfile: yarn.lock. This lockfile includes the version and checksum to verify integrity when packages are installed during the build process. In addition, the VotingWorks build process explicitly requires the use of --frozen-lockfile to ensure that only the packages explicitly required are installed.
ping -c2 google.comcd ~/code/vxsuite-build-system
./scripts/tb-run-offline-phase.sh <inventory-name>On VxAdmin, you'll first see a prompt to enter a jurisdiction:
Then, on all machines, you'll see this prompt:
Insert a USB drive that you designate for this purpose. From here on out, we'll refer to this USB drive as the VxCertifier USB drive.
After selecting the VxCertifier USB drive, a certificate signing request will be written to it. You'll then be prompted to:
Because you'll be certifying your machine at your own facility as opposed to a VotingWorks facility, you won't be able to take the USB drive to VxCertifier, our VotingWorks certification terminal. We'll need to use a remote certification process instead.
You'll need to remove the VxCertifier USB drive, find the "csr-<machine-id>.pem" file inside the "certs/" directory on it, and share that file with VotingWorks. This file does not contain any private information so can be shared over the internet, e.g., via email. VotingWorks will prepare a certificate given this "csr-<machine-id>.pem" file and send the certificate back to you, in the form of a "cert-<machine-id>.pem" file. This file, too, does not contain any private information so can be shared over the internet. You'll need to copy this "cert-<machine-id>.pem" file back onto the VxCertifier USB drive, placing it in the same "certs/" directory that we pulled the "csr.pem" from. Re-inserting the USB drive into the machine and pressing enter should allow you to proceed successfully.
On VxAdmin, you'll be prompted to program your first system administrator card as a last step. Remember to record the displayed PIN. On other machines, no steps remain. You'll reboot into the app after this.
Enter a jurisdiction ({state-2-letter-abbreviation}.{county-town-etc}, e.g., ca.los-angeles or vx.test for test/demo machines):Once SLI has configured their S3 access and created a Trusted Build image, they will need to upload the image and its configuration file. Additionally, hashes of each file will be generated so that SLI and VotingWorks can confirm files were not modified during the upload or download phases.
On the SLI build machine:
Once the SLI upload has completed, VotingWorks will download and verify the hash values of all files. On the VotingWorks secure build machine, while SLI observes:
Once the VotingWorks download has completed and been verified, and the VM has been successfully defined, VotingWorks can proceed with its Secure Boot signing process, while SLI observes.
VotingWorks will boot the VM, change the vendor password, attach a virtual device containing our Secure Boot signing keys, and select the option to "Lock the System Down" from the vendor menu. When prompted, VotingWorks will enter the passphrase for the Secure Boot signing keys.
When the process completes, the lock-down script displays the system hash in SHA256 and Signed Hash Validation (SHV) versions. These hashes will be provided to SLI/EAC for official verification of the image.
Now that the image has been securely signed, VotingWorks will upload the signed image for SLI to later download. While SLI observes, VotingWorks will run the following command on the VotingWorks secure build machine:
Once this step has completed, SLI can download the files at their convenience. On the SLI build machine:
Once SLI has completed the download, and hashes have been validated, the compressed image and signature can be provided to EAC for official use. At any point after submission to EAC, election officials and VotingWorks can verify the image hash as described under Verifying the Image Installed on a Machine.
sudo su -
/home/vx/code/vxsuite-build-system/scripts/sb-unsigned-upload.sh vxadminsudo shutdown -h nowInsert a USB drive into the machine. Press enter once you've done so.Remove the USB drive, take it to VxCertifier, and bring it back to this machine when prompted. Press enter once you've re-inserted the USB drive.sudo /home/admin/code/vxsuite-build-system/scripts/sb-unsigned-download.sh vxadminsudo /home/admin/code/vxsuite-build-system/scripts/sb-signed-upload.sh vxadminsudo su -
/home/vx/code/vxsuite-build-system/scripts/sb-signed-download.sh vxadminOpen virt-manager:
sudo virt-managerDouble-click the online VM.
Press the start button ▶️.
Once the VM has initialized, log in with username vx and password votingworks.
To ensure that the console displays correctly, select "View" > "Resize to VM".
In the VM terminal window, run the following commands:
You will be prompted for the sudo password.
The online phase will take awhile to complete. Once it finishes, you can shut down the online VM:
When installing VotingWorks applications that will be used in one or more elections, it is necessary to create a record of the software installation. There are multiple requirements, as described below.
3.1.4-I.1 a unique identifier (such as a serial number) for the record; 3.1.4-I.2 a list of unique identifiers of storage media associated with the record; 3.1.4-I.3 the time, date, and location of the software installation; 3.1.4-I.4 names, affiliations, and signatures of all people present; 3.1.4-I.5 copies of the procedures used to install the software on the programmed devices of the voting system; 3.1.4-I.6 the certification number of the voting system; 3.1.4-I.7 list of the software installed as well as associated digital signatures and mechanisms for installation and verification on programmed devices of the voting system; and 3.1.4-I.8 a unique identifier (such as a serial number) of the vote-capture device or election management system (EMS) which the software is installed.
To satisfy the above requirements, VotingWorks recommends following these best practices related to each requirement.
3.1.4-I.1 - We recommend using the machine ID for this record, e.g. SC-11-004 If necessary, appending the install date and time can further ensure uniqueness, e.g. SC-11-004-20250325-1300
3.1.4-I.2 - This should be the machine ID + the system drive the application was installed to. For example, a machine identified as SC-11-004 with the VotingWorks application installed to the /dev/nvme0n1 drive would use the record: SC-11-004-dev-nvme0n1
3.1.4-I.3 - As described.
3.1.4-I.4 - As described.
3.1.4-I.5 - Include physical copies of the TDP install instructions or a link to the TDP install instructions for this release.
3.1.4-I.6 - This is the EAC Certification Number, which can be found on the Certificate of Conformance for this VotingWorks application release.
3.1.4-I.7 - Include a physical copy of the COTS Report or a link to the COTS Report that can be found in the documentation, specific to each VotingWorks application release.
3.1.4-I.8 - We recommend using the serial number of the hardware device the VotingWorks application is being installed to. If that is not available, the same identifier used in 3.1.4-I.1 is recommended.
We'll now clone the offline VM to prepare images for specific machine types, i.e., VxAdmin, VxCentralScan, VxMarkScan, and VxScan. We'll create one clone/VM per machine type.
In the following steps, the vxadmin VM will be referenced, but these steps can be repeated for each machine type: vxcentralscan, vxmarkscan, and vxscan.
To clone the offline VM, run the following command on the build machine:
This command creates a byte-for-byte clone of the offline VM, along with all settings, including network functionality disabled at the VM level.
mkdir ~/code && cd ~/code
git clone https://github.com/votingworks/vxsuite-build-system
cd ~/code/vxsuite-build-system
# Optional step if VotingWorks is doing development work in
# vxsuite-build-system and the latest code in GitHub differs from the source
# code provided to the test lab
git checkout <relevant-vxsuite-build-system-tag>
./scripts/tb-run-online-phase.sh <inventory-name>sudo shutdown -h nowOpen virt-manager if not already open:
Double-click the vxadmin VM.
Press the start button ▶️.
From the menu, select "View" > "Consoles" > "Serial 1". (If you only see a blinking cursor, make the window active by clicking in it, then press Enter. You should see a login prompt.)
Once the VM has initialized, log in with username vx and password votingworks.
To ensure that the console displays correctly, select "View" > "Resize to VM".
In the VM terminal window, run the following commands:
You will be guided through several prompts.
Select the number of the machine type that you intend to build.
Type "N" when asked whether this image is for QA.
Type "y" when asked whether this is an official release image.
Set a password for the vx-vendor user. This password will not meaningfully be used as the vendor menu on a production image is only accessible via a vendor card.
After the script finishes, the VM will reboot. You will see a console login prompt. In the VM menu, select "Virtual Machine" > "Shut Down" > "Shut Down". Close the VM window once shutdown is complete.
You will now need to perform the Secure Boot Signing process with VotingWorks. Once that process is completed, the VM and corresponding image will be ready for use with Secure Boot.
At this point, you are ready to install the image. You can find those instructions in Imaging Machines.
sudo virt-clone -o offline -n vxadmin --auto-clonesudo virt-managercd ~/code/vxsuite-complete-system
./setup-machine.shWelcome to VxSuite. THIS IS A DESTRUCTIVE SCRIPT. Ctrl-C right now if you don't know for sure what you're doing.
Which machine are we building today?
1. VxAdmin
2. VxCentralScan
3. VxMarkScan
4. VxScanIs this image for QA, where you want sudo privileges, terminal access via TTY2, and the ability to record screengrabs? [y/N]Is this additionally an official release image? [y/N]Once a machine has been imaged with a signed image, you can verify the system hash against the hash of what was built and signed during the Trusted Build process. To perform this verification, VotingWorks provides Signed Hash Validation.
See Signed Hash Validation for details.
Note: On older releases (v4.0.1 and earlier), the Secure Boot signing process only output a SHA256 hash value. That hash value needs to be converted to a base64 encoded value to match what is provided by Signed Hash Validation. That can be accomplished with the following command:
echo HASH | xxd -r -p | base64The system hash can also be computed from outside the application, using a vx-iso USB drive as described under . After booting from the drive, simply select the option to "Compute System Hash".