Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Open virt-manager if not already open:
Double-click the offline VM.
Press the start button ▶️.
Once the VM has initialized, log in with username vx and password votingworks.
To ensure that the console displays correctly, select "View" > "Resize to VM".
In the VM terminal window, validate that networking is automatically disabled in the offline VM using the following command:
This command should either time out or immediately error with a failure in name resolution.
Ensure that the USB drive created in the online phase is attached to the build machine.
To make the USB available to the offline VM, select "Virtual Machine" > "Redirect USB Device". In the dialog box that opens, select the USB drive that you attached and close the dialog.
The USB drive will usually be available as /dev/sda1. To confirm, you can use:
If it’s accessible by a different path, that’s fine, just make a note of it and use that path in the following steps as appropriate.
Run the following commands:
Once the offline phase completes, the base VotingWorks application has been built. You can now shut down the offline VM:
To finalize an image for production use, it's necessary to sign the image with the VotingWorks Secure Boot keys so that the image can be used on a Secure-Boot-enabled machine. This process requires the use of sensitive keys and a passphrase that should only ever be known to and used by VotingWorks. Since the Trusted Build process is performed by a third-party vendor (SLI), a process to securely sign the image without compromising the keys or passphrase is needed. This document describes that process.
Once SLI has created a Trusted Build image, the image and its VM definition (an XML file) must be securely provided to VotingWorks. To transfer the image and definition, SLI will upload to a shared S3 bucket only accessible to SLI and VotingWorks. This access is controlled via IAM permission policies.
NOTE: For documentation purposes, we will use a Trusted Build image named vxadmin. Replace that with the appropriate image name, as necessary.
Once SLI has configured their S3 access and created a Trusted Build image, they will need to upload the image and its configuration file. Additionally, hashes of each file will be generated so that SLI and VotingWorks can confirm files were not modified during the upload or download phases.
On the SLI build machine:
Once the SLI upload has completed, VotingWorks will download and verify the hash values of all files. On the VotingWorks secure build machine, while SLI observes:
Once the VotingWorks download has completed and been verified, and the VM has been successfully defined, VotingWorks can proceed with its Secure Boot signing process, while SLI observes.
VotingWorks will boot the VM, attach a virtual device containing our Secure Boot signing keys, and select the option to "Lock the System Down" from the vendor menu. When prompted, VotingWorks will enter the passphrase for the Secure Boot signing keys.
When the process completes, the lock-down script displays the system hash. This hash will be provided to SLI/EAC for official verification of the image.
Now that the image has been securely signed, VotingWorks will upload the signed image for SLI to later download. While SLI observes, VotingWorks will run the following command on the VotingWorks secure build machine:
Once this step has completed, SLI can download the files at their convenience. On the SLI build machine:
Once SLI has completed the download, and hashes have been validated, the compressed image and signature can be provided to EAC for official use. At any point after submission to EAC, election officials and VotingWorks can verify the image hash as described under Verifying the Image Installed on a Machine.
To install Debian 12 on the build machine (VxBuild), you will need to download the latest Debian 12 amd64 installer file here: .
You should use a blank USB drive to create the install drive. The process described below will wipe any existing content.
To create a USB install drive with the downloaded ISO file:
Ensure that the USB drive is available to the system. By default, it tries to attach as the /dev/sda device. An easy way to verify this is via the following command:
If the USB drive is not attached as the /dev/sda device, that’s ok. Simply replace /dev/sda with the device that your USB drive did attach to.
NOTE: The command below should not include any number as part of the device path. For example, if the above command returns /dev/sda1, you must use /dev/sda as the path in the next command.
Now that you know the correct device path, you can create the USB install drive via the following command:
NOTE: At the time of this writing, the latest stable release is 12.8. Please update to the appropriate file name if you're using a newer version.
Once the above command completes, you can safely remove the USB install drive from your system.
Before powering VxBuild on, insert the USB install drive created in the previous step.
After powering VxBuild on, begin pressing F12 until it enters the boot menu. Boot the system from the USB install drive.
You will be presented with the following screen. Select "Graphical install" and press enter.
Select your preferred language and click "Continue".
Select your location and click "Continue".
Select your keyboard layout and click "Continue".
Select your network device (may differ from screenshot) and click "Continue".
If using a wired connection, it will configure automatically.
If using a wireless connection, select a network, select WPA/WPA2 PSK, and click "Continue". Enter your wireless network password and click "Continue".
Enter "VxBuild" for the hostname and click "Continue".
Leave the domain name blank and click "Continue".
Set the root user password and click "Continue".
Enter "Vx" for the full name for the new user and click "Continue".
Enter "vx" as the username and click "Continue".
Enter a password for the "vx" user and click Continue.
Select your timezone and click "Continue".
Select "Guided - use entire disk" and click "Continue".
Select the "/dev/nvme0n1" disk and click "Continue".
Select "All files in one partition" and click "Continue".
Select "Finish partitioning and write changes to disk" and click "Continue".
Select "Yes" and click "Continue".
The base OS installation will now begin. During the installation you will be asked to answer questions related to package management.
Select "United States" and click "Continue".
Select "deb.debian.org" and click "Continue".
Leave the proxy information blank and click "Continue".
You will be asked to configure "popularity-contest". Select "No" and click "Continue".
The installation process will continue. Once it completes, you will be presented with a final screen confirming that it was successful.
Remove the USB drive and click "Continue".
The system will reboot and automatically start Debian 12.
Log in with the "vx" user and password you created during the installation process.
Several basic configuration prompts will be displayed. Select "Next".
Select your keyboard preference and click "Next".
Turn off Location Services and click "Next".
Do not configure any online accounts. Select "Skip".
Click the "Start Using Debian GNU/Linux" button. The dialog will be closed.
The "vx" user needs to be granted sudo privileges related to configuring and initializing the build environment tools.
Open a terminal window.
As the "vx" user, you will temporarily log in to the "root" account.
You will see your terminal window prompt change to root@VxBuild. To grant the "vx" user sudo privileges, run the following command as root.
You will see your terminal window prompt change to vx@VxBuild. You are now the "vx" user instead of the "root" user. To confirm sudo privileges, run the following command:
The command should return "root". This confirms sudo privileges have been granted correctly.
We'll now clone the offline VM to prepare images for specific machine types, i.e., VxAdmin, VxCentralScan, VxMark, and VxScan. We'll create one clone/VM per machine type.
In the following steps, the vxadmin VM will be referenced, but these steps can be repeated for each machine type: vxcentralscan, vxmark, and vxscan.
To clone the offline VM, run the following command on the build machine:
This command creates a byte-for-byte clone of the offline VM, along with all settings, including network functionality disabled at the VM level.
Open virt-manager if not already open:
Double-click the vxadmin VM.
Press the start button ▶️.
Once the VM has initialized, log in with username vx and password votingworks.
To ensure that the console displays correctly, select "View" > "Resize to VM".
In the VM terminal window, run the following commands:
You will be guided through several prompts.
Select the number of the machine type that you intend to build.
Type "N" when asked whether this image is for QA.
Type "y" when asked whether this is an official release image.
Set a password for the vx-vendor user. This password will not meaningfully be used as the vendor menu on a production image is only accessible via a vendor card.
After the script finishes, the VM will reboot. You will see a white screen displaying “Card Reader Not Detected”. In the VM menu, select "Virtual Machine" > "Shut Down" > "Shut Down". Close the VM window once shutdown is complete.
You will now need to perform the process with VotingWorks. Once that process is completed, the VM and corresponding image will be ready for use with Secure Boot.
At this point, you are ready to install the image. You can find those instructions in .
The VotingWorks build process consists of three distinct build phases: online, offline, and final configuration. The build process uses an inventory definition containing all the necessary information for a Trusted Build. All three phases are executed within a separate virtual machine (VM) managed by the virt-manager tool, running on a Debian 12 operating system.
In the online phase, the build process utilizes a base Debian 12 VM with network access enabled. All necessary code repositories and build tools are securely retrieved and transferred to a USB drive for use during the offline phase.
In the offline phase, the build process utilizes a base Debian 12 VM with network access disabled. All necessary code repositories and build tools are transferred from the USB drive created during the online phase.
In the final configuration phase, the build process utilizes clones of the offline VM to prepare images for specific machine types, i.e., VxAdmin, VxCentralScan, VxMark, and VxScan.
After completing the final configuration phase, an unlocked installation image has been created. While this image is not appropriate for production use, it can be used for testing.
For a final production image, an additional step is required to securely sign the installation image for use with Secure Boot enabled systems, described under Secure Boot Signing.
The VotingWorks build system, vxsuite-build-system, utilizes many different tools to install, configure, and build VotingWorks applications. To ensure the integrity of the build system and tools used, all third-party tools are verified against known hashes, checksums, or digital signatures. This appendix provides a description of the mechanism used to verify each tool and any additional resources those tools may be responsible for providing during the build process.
APT
Debian's package manager, apt, is built on a secure framework described here: SecureApt.
The VotingWorks build process relies on this fundamental tool to install many of the system level tools required to build our applications. As an added measure of security, all packages are pinned to specific versions to ensure a consistent build environment over time.
To further ensure that consistent versions of dependencies and transitive dependencies are used, we create our own VotingWorks-hosted apt snapshot to pull from.
pip
Python's package manager, pip, does not verify package integrity by default. However, secure installation is possible via the method described here: pip: Secure installs.
The VotingWorks build process implements the above method in the tb-install-ansible.sh script. The required packages are listed, along with their hash, in unique requirements files based on the operating system version and architecture. You can see an example here: Debian 12 x86 pip requirements.
RubyGems
Ruby's package manager, gem, does not verify package integrity by default. However, the official gem repository provides SHA256 checksums for hosted packages. You can see an example here: FPM 1.15.1.
The VotingWorks build process requires the version and SHA256 checksum for any installed gems. Once the gem file has been downloaded, it is checked against this checksum to verify its integrity. You can see that checksum verification in the rubygems playbook here: rubygems.yaml.
Rust
By default, Rust is installed over an active internet connection. Since that method is not available during the offline phase of the VotingWorks build process, we use a standalone, offline installer binary provided by Rust. To verify the integrity of the installer, the downloaded file is verified against the officially published checksum that can be found in each version's official manifest. You can see that checksum verification in the rust playbook here: rust.yaml.
Cargo
Cargo is the Rust package manager. It provides secure installs via the combination of a manifest listing packages and their versions, along with a lock file containing the checksum for each package. That method is described here: Cargo.toml vs. Cargo.lock.
Node
Node is typically installed via package managers like apt, or via a direct install from a specific version. We do not install Node via apt since we explicitly version it. As a result, we download the appropriate installer from the official Node repository. That downloaded file is then checked against the officially provided checksum. You can see that checksum verification in the node playbook here: node.yaml.
npm
npm is the default Node package manager. It is installed as part of the Node install previously described.
Packages installed via npm are checked against the officially provided checksums for each package. You can see that checksum verification in the node playbook here: node.yaml.
pnpm
pnpm is another package manager for the Node ecosystem. It is installed via npm as previously described to ensure its integrity.
Packages installed via pnpm use a lockfile: pnpm-lock.yaml. This lockfile includes the version and checksum to verify integrity when packages are installed during the build process. In addition, the VotingWorks build process explicitly requires the use of --frozen-lockfile to ensure that only the packages explicitly required are installed.
Yarn
Yarn is another package manager for the Node ecosystem. It is installed via npm as previously described to ensure its integrity.
Packages installed via yarn use a lockfile: yarn.lock. This lockfile includes the version and checksum to verify integrity when packages are installed during the build process. In addition, the VotingWorks build process explicitly requires the use of --frozen-lockfile to ensure that only the packages explicitly required are installed.
Operating system: Debian 12
RAM: 8GB minimum
Chip: x86-64 / amd64 CPU architecture, Intel i5 or higher
Storage: 250GB minimum
Debian 12 will need to be installed on the build machine. You can find detailed instructions for that process under Installing Debian 12 on VxBuild. Once that is complete, we can install the VotingWorks build system, vxsuite-build-system.
Throughout the rest of this document, substitute <inventory-name> with the inventory provided by VotingWorks.
At this point, you should have three separate VMs: debian-<inventory-name>-<apt-snapshot-date>-<debian-release-name>, online, and offline. You can confirm this by running:
By default, newly created VMs have networking enabled. To ensure the offline VM does not have access to the internet, the networking link is disabled before the VM is ever used. This is accomplished by editing the VM's configuration (found in /etc/libvirt/qemu/offline.xml) and setting the link state to down. This functionality can be seen in the vxsuite-build-system repo, in playbooks/virtmanager/clone-base-vm.yaml.
If an offline VM exists, the XML configuration file is updated to set the link state to down. After that, the offline VM is explicitly re-defined from this updated XML configuration file. These steps always execute, even if the network link state is already set to down.
Any VM cloned from this offline VM will also have a disabled network since those settings are inherited from the original VM.
Virt Manager uses a local bridge network to connect to VMs. If you receive an error when starting a VM that says "Error starting domain: Cannot get interface MTU on 'virbr0': No such device", run the following command in the terminal on the build machine:
You should now be able to return to the VM and start it without issue.
Open virt-manager:
Double-click the online VM.
Press the start button ▶️.
Once the VM has initialized, log in with username vx and password votingworks.
To ensure that the console displays correctly, select "View" > "Resize to VM".
In the VM terminal window, run the following commands:
You will be prompted for the sudo password.
The online phase will take awhile to complete.
Once it finishes, you need to attach a USB drive to the build machine. This USB drive will be used to transfer all necessary tools and code to the offline VM.
To make the USB drive available to the online VM, select "Virtual Machine" > "Redirect USB Device". In the dialog box that opens, select the USB drive that you attached and close the dialog.
In the online VM terminal:
You will be prompted to select the USB drive. Once selected, all necessary code repositories and build tools will be exported to the USB drive. After the export is complete, you can shut down the online VM: