Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
To install Debian 12 on the build machine (VxBuild), you will need to download the latest Debian 12 amd64 installer file here: latest Debian release.
You should use a blank USB drive to create the install drive. The process described below will wipe any existing content.
To create a USB install drive with the downloaded ISO file:
Ensure that the USB drive is available to the system. By default, it tries to attach as the /dev/sda device. An easy way to verify this is via the following command:
If the USB drive is not attached as the /dev/sda device, that’s ok. Simply replace /dev/sda with the device that your USB drive did attach to.
NOTE: The command below should not include any number as part of the device path. For example, if the above command returns /dev/sda1, you must use /dev/sda as the path in the next command.
Now that you know the correct device path, you can create the USB install drive via the following command:
NOTE: At the time of this writing, the latest stable release is 12.8. Please update to the appropriate file name if you're using a newer version.
Once the above command completes, you can safely remove the USB install drive from your system.
Before powering VxBuild on, insert the USB install drive created in the previous step.
After powering VxBuild on, begin pressing F12 until it enters the boot menu. Boot the system from the USB install drive.
You will be presented with the following screen. Select "Graphical install" and press enter.
Select your preferred language and click "Continue".
Select your location and click "Continue".
Select your keyboard layout and click "Continue".
Select your network device (may differ from screenshot) and click "Continue".
If using a wired connection, it will configure automatically.
If using a wireless connection, select a network, select WPA/WPA2 PSK, and click "Continue". Enter your wireless network password and click "Continue".
Enter "VxBuild" for the hostname and click "Continue".
Leave the domain name blank and click "Continue".
Set the root user password and click "Continue".
Enter "Vx" for the full name for the new user and click "Continue".
Enter "vx" as the username and click "Continue".
Enter a password for the "vx" user and click Continue.
Select your timezone and click "Continue".
Select "Guided - use entire disk" and click "Continue".
Select the "/dev/nvme0n1" disk and click "Continue".
Select "All files in one partition" and click "Continue".
Select "Finish partitioning and write changes to disk" and click "Continue".
Select "Yes" and click "Continue".
The base OS installation will now begin. During the installation you will be asked to answer questions related to package management.
Select "United States" and click "Continue".
Select "deb.debian.org" and click "Continue".
Leave the proxy information blank and click "Continue".
You will be asked to configure "popularity-contest". Select "No" and click "Continue".
The installation process will continue. Once it completes, you will be presented with a final screen confirming that it was successful.
Remove the USB drive and click "Continue".
The system will reboot and automatically start Debian 12.
Log in with the "vx" user and password you created during the installation process.
Several basic configuration prompts will be displayed. Select "Next".
Select your keyboard preference and click "Next".
Turn off Location Services and click "Next".
Do not configure any online accounts. Select "Skip".
Click the "Start Using Debian GNU/Linux" button. The dialog will be closed.
The "vx" user needs to be granted sudo privileges related to configuring and initializing the build environment tools.
Open a terminal window.
As the "vx" user, you will temporarily log in to the "root" account.
You will see your terminal window prompt change to root@VxBuild. To grant the "vx" user sudo privileges, run the following command as root.
You will see your terminal window prompt change to vx@VxBuild. You are now the "vx" user instead of the "root" user. To confirm sudo privileges, run the following command:
The command should return "root". This confirms sudo privileges have been granted correctly.
Software installation involves two steps:
Generating software images via Trusted Build:
Imaging machines with those images:
Operating system: Debian 12
RAM: 8GB minimum
Chip: x86-64 / amd64 CPU architecture, Intel i5 or higher
Storage: 250GB minimum
Debian 12 will need to be installed on the build machine. You can find detailed instructions for that process under . Once that is complete, we can install the VotingWorks build system, .
Throughout the rest of this document, substitute <inventory-name> with the inventory provided by VotingWorks.
At this point, you should have three separate VMs: debian-<inventory-name>-<apt-snapshot-date>-<debian-release-name>, online, and offline. You can confirm this by running:
Open virt-manager if not already open:
Double-click the offline VM.
Press the start button ▶️.
Once the VM has initialized, log in with username vx and password votingworks.
To ensure that the console displays correctly, select "View" > "Resize to VM".
In the VM terminal window, validate that networking is automatically disabled in the offline VM using the following command:
This command should either time out or immediately error with a failure in name resolution.
Ensure that the USB drive created in the online phase is attached to the build machine.
To make the USB available to the offline VM, select "Virtual Machine" > "Redirect USB Device". In the dialog box that opens, select the USB drive that you attached and close the dialog.
The USB drive will usually be available as /dev/sda1. To confirm, you can use:
If it’s accessible by a different path, that’s fine, just make a note of it and use that path in the following steps as appropriate.
Run the following commands:
Once the offline phase completes, the base VotingWorks application has been built. You can now shut down the offline VM:
To finalize an image for production use, it's necessary to sign the image with the VotingWorks Secure Boot keys so that the image can be used on a Secure-Boot-enabled machine. This process requires the use of sensitive keys and a passphrase that should only ever be known to and used by VotingWorks. Since the Trusted Build process is performed by a third-party vendor (SLI), a process to securely sign the image without compromising the keys or passphrase is needed. This document describes that process.
Once SLI has created a Trusted Build image, the image and its VM definition (an XML file) must be securely provided to VotingWorks. To transfer the image and definition, SLI will upload to a shared S3 bucket only accessible to SLI and VotingWorks. This access is controlled via IAM permission policies.
NOTE: For documentation purposes, we will use a Trusted Build image named vxadmin. Replace that with the appropriate image name, as necessary.
Once SLI has configured their S3 access and created a Trusted Build image, they will need to upload the image and its configuration file. Additionally, hashes of each file will be generated so that SLI and VotingWorks can confirm files were not modified during the upload or download phases.
On the SLI build machine:
Once the SLI upload has completed, VotingWorks will download and verify the hash values of all files. On the VotingWorks secure build machine, while SLI observes:
Once the VotingWorks download has completed and been verified, and the VM has been successfully defined, VotingWorks can proceed with its Secure Boot signing process, while SLI observes.
VotingWorks will boot the VM, attach a virtual device containing our Secure Boot signing keys, and select the option to "Lock the System Down" from the vendor menu. When prompted, VotingWorks will enter the passphrase for the Secure Boot signing keys.
When the process completes, the lock-down script displays the system hash. This hash will be provided to SLI/EAC for official verification of the image.
Now that the image has been securely signed, VotingWorks will upload the signed image for SLI to later download. While SLI observes, VotingWorks will run the following command on the VotingWorks secure build machine:
Once this step has completed, SLI can download the files at their convenience. On the SLI build machine:
Once SLI has completed the download, and hashes have been validated, the compressed image and signature can be provided to EAC for official use. At any point after submission to EAC, election officials and VotingWorks can verify the image hash as described under Verifying the Image Installed on a Machine.
The VotingWorks build process consists of three distinct build phases: online, offline, and final configuration. The build process uses an inventory definition containing all the necessary information for a Trusted Build. All three phases are executed within a separate virtual machine (VM) managed by the virt-manager tool, running on a Debian 12 operating system.
In the online phase, the build process utilizes a base Debian 12 VM with network access enabled. All necessary code repositories and build tools are securely retrieved and transferred to a USB drive for use during the offline phase.
In the offline phase, the build process utilizes a base Debian 12 VM with network access disabled. All necessary code repositories and build tools are transferred from the USB drive created during the online phase.
In the final configuration phase, the build process utilizes clones of the offline VM to prepare images for specific machine types, i.e., VxAdmin, VxCentralScan, VxMark, and VxScan.
After completing the final configuration phase, an unlocked installation image has been created. While this image is not appropriate for production use, it can be used for testing.
For a final production image, an additional step is required to securely sign the installation image for use with Secure Boot enabled systems, described under Secure Boot Signing.
We'll now clone the offline VM to prepare images for specific machine types, i.e., VxAdmin, VxCentralScan, VxMark, and VxScan. We'll create one clone/VM per machine type.
In the following steps, the vxadmin VM will be referenced, but these steps can be repeated for each machine type: vxcentralscan, vxmark, and vxscan.
To clone the offline VM, run the following command on the build machine:
This command creates a byte-for-byte clone of the offline VM, along with all settings, including network functionality disabled at the VM level.
Open virt-manager if not already open:
Double-click the vxadmin VM.
Press the start button ▶️.
Once the VM has initialized, log in with username vx and password votingworks.
To ensure that the console displays correctly, select "View" > "Resize to VM".
In the VM terminal window, run the following commands:
You will be guided through several prompts.
Select the number of the machine type that you intend to build.
Type "N" when asked whether this image is for QA.
Type "y" when asked whether this is an official release image.
Set a password for the vx-vendor user. This password will not meaningfully be used as the vendor menu on a production image is only accessible via a vendor card.
After the script finishes, the VM will reboot. You will see a white screen displaying “Card Reader Not Detected”. In the VM menu, select "Virtual Machine" > "Shut Down" > "Shut Down". Close the VM window once shutdown is complete.
This section walks through the steps to install a Trusted Build image on a VotingWorks component using .
Open virt-manager:
Double-click the online VM.
Press the start button ▶️.
Once the VM has initialized, log in with username vx and password votingworks.
To ensure that the console displays correctly, select "View" > "Resize to VM".
In the VM terminal window, run the following commands:
You will be prompted for the sudo password.
The online phase will take awhile to complete.
Once it finishes, you need to attach a USB drive to the build machine. This USB drive will be used to transfer all necessary tools and code to the offline VM.
To make the USB drive available to the online VM, select "Virtual Machine" > "Redirect USB Device". In the dialog box that opens, select the USB drive that you attached and close the dialog.
In the online VM terminal:
You will be prompted to select the USB drive. Once selected, all necessary code repositories and build tools will be exported to the USB drive. After the export is complete, you can shut down the online VM:
You will now need to perform the process with VotingWorks. Once that process is completed, the VM and corresponding image will be ready for use with Secure Boot.
At this point, you are ready to install the image. You can find those instructions in .
The VotingWorks build system, vxsuite-build-system, utilizes many different tools to install, configure, and build VotingWorks applications. To ensure the integrity of the build system and tools used, all third-party tools are verified against known hashes, checksums, or digital signatures. This appendix provides a description of the mechanism used to verify each tool and any additional resources those tools may be responsible for providing during the build process.
APT
Debian's package manager, apt, is built on a secure framework described here: SecureApt.
The VotingWorks build process relies on this fundamental tool to install many of the system level tools required to build our applications. As an added measure of security, all packages are pinned to specific versions to ensure a consistent build environment over time.
To further ensure that consistent versions of dependencies and transitive dependencies are used, we create our own VotingWorks-hosted apt snapshot to pull from.
pip
Python's package manager, pip, does not verify package integrity by default. However, secure installation is possible via the method described here: pip: Secure installs.
The VotingWorks build process implements the above method in the tb-install-ansible.sh script. The required packages are listed, along with their hash, in unique requirements files based on the operating system version and architecture. You can see an example here: Debian 12 x86 pip requirements.
RubyGems
Ruby's package manager, gem, does not verify package integrity by default. However, the official gem repository provides SHA256 checksums for hosted packages. You can see an example here: FPM 1.15.1.
The VotingWorks build process requires the version and SHA256 checksum for any installed gems. Once the gem file has been downloaded, it is checked against this checksum to verify its integrity. You can see that checksum verification in the rubygems playbook here: rubygems.yaml.
Rust
By default, Rust is installed over an active internet connection. Since that method is not available during the offline phase of the VotingWorks build process, we use a standalone, offline installer binary provided by Rust. To verify the integrity of the installer, the downloaded file is verified against the officially published checksum that can be found in each version's official manifest. You can see that checksum verification in the rust playbook here: rust.yaml.
Cargo
Cargo is the Rust package manager. It provides secure installs via the combination of a manifest listing packages and their versions, along with a lock file containing the checksum for each package. That method is described here: Cargo.toml vs. Cargo.lock.
Node
Node is typically installed via package managers like apt, or via a direct install from a specific version. We do not install Node via apt since we explicitly version it. As a result, we download the appropriate installer from the official Node repository. That downloaded file is then checked against the officially provided checksum. You can see that checksum verification in the node playbook here: node.yaml.
npm
npm is the default Node package manager. It is installed as part of the Node install previously described.
Packages installed via npm are checked against the officially provided checksums for each package. You can see that checksum verification in the node playbook here: node.yaml.
pnpm
pnpm is another package manager for the Node ecosystem. It is installed via npm as previously described to ensure its integrity.
Packages installed via pnpm use a lockfile: pnpm-lock.yaml. This lockfile includes the version and checksum to verify integrity when packages are installed during the build process. In addition, the VotingWorks build process explicitly requires the use of --frozen-lockfile to ensure that only the packages explicitly required are installed.
Yarn
Yarn is another package manager for the Node ecosystem. It is installed via npm as previously described to ensure its integrity.
Packages installed via yarn use a lockfile: yarn.lock. This lockfile includes the version and checksum to verify integrity when packages are installed during the build process. In addition, the VotingWorks build process explicitly requires the use of --frozen-lockfile to ensure that only the packages explicitly required are installed.
To install an image on a VotingWorks component, i.e., to image a machine, you need two USB drives:
A vx-iso USB drive — vx-iso is our VotingWorks-specific ISO installer program.
An image USB drive — This is an empty USB drive with two partitions, a "Data" partition that can contain as many VotingWorks images as space allows and a "Keys" partition that can optionally contain the VotingWorks Secure Boot public keys, necessary if a machine hasn't had these keys installed yet.
Clone the vx-iso repo for the tooling necessary to prepare the above:
To create a vx-iso USB drive, you can follow these instructions: https://github.com/votingworks/vx-iso/blob/main/README.md#creating-an-install-drive
If this is SLI, we have provided you with vx-iso USB drives so that you don't need to prepare them from scratch.
To create an image USB drive, you can follow these instructions: https://github.com/votingworks/vx-iso/blob/main/README.md#creating-an-image-with-optional-secure-boot-keys-drive
You'll need access to the relevant images and optionally the VotingWorks Secure Boot public keys. Both of these are stored on a private S3 bucket, though they're not sensitive, and VotingWorks can prepare temporary links to grant access to them.
If this is SLI, you do not need to create data drives with the VotingWorks Secure Boot public keys. Secure Boot has already been configured on all your machines.
After you've imaged a machine, the machine will boot into a basic configuration wizard. The majority of the steps are self-explanatory, but "Step 1: Set Machine ID" and "Step 4: Create Machine Cert" require some extra clarification.
It is important that the machine ID be unique for each machine. Many machines have a physical placard on them indicating the machine ID. That is the ID that should be used here.
On VxAdmin, you'll first see a prompt to enter a jurisdiction:
SLI should use co.sli
Then, on all machines, you'll see this prompt:
Insert a USB drive that you designate for this purpose. From here on out, we'll refer to this USB drive as the VxCertifier USB drive.
After selecting the VxCertifier USB drive, a certificate signing request will be written to it. You'll then be prompted to:
Because you'll be certifying your machine at your own facility as opposed to a VotingWorks facility, you won't be able to take the USB drive to VxCertifier, our VotingWorks certification terminal. We'll need to use a remote certification process instead.
You'll need to remove the VxCertifier USB drive, find the "csr.pem" file inside the "certs/" directory on it, and share that file with VotingWorks. This file does not contain any private information so can be shared over the internet, e.g., via email. VotingWorks will prepare a certificate given this "csr.pem" file and send the certificate back to you, in the form of a "cert.pem" file. This file, too, does not contain any private information so can be shared over the internet. You'll need to copy this "cert.pem" file back onto the VxCertifier USB drive, placing it in the same "certs/" directory that we pulled the "csr.pem" from. Re-inserting the USB drive into the machine and pressing enter should allow you to proceed successfully.
On VxAdmin, you'll be prompted to program your first system administrator card as a last step. Remember to record the displayed PIN. On other machines, no steps remain. You'll reboot into the app after this.
First make sure that you've prepared USB drives for imaging, following the instructions under Preparing USB Drives for Imaging. Then follow these steps:
Power off the machine.
Insert both the vx-iso and image USB drives into the system. If this is a VxMark or a VxScan, connect a keyboard as well. If there aren't enough ports available, use a USB hub as provided by VotingWorks.
Power on the machine to begin booting vx-iso.
The precinct system components (VxMark and VxScan) are to auto-boot from a bootable USB drive when connected and should auto-boot to vx-iso.
The central system components (VxAdmin and VxCentralScan) require entering f9 after powering on to boot to USB. Select the USB drive corresponding to the vx-iso drive.
Select "Write an image". You can navigate vx-iso with the keyboard. This option will be auto-selected in 10 seconds.
If the machine already has Secure Boot keys installed, it should not prompt you to install keys. If it does for some reason, you should reach out to VotingWorks for assistance. Only if you know the keys need to be installed should you opt to install them.
The images on the image USB drive will be displayed. Select the number that identifies the correct image.
Enter 27 for the final expected size of the image in GB.
Confirm your selections and wait for imaging to complete.
Once imaging completes, remove the USB drives and press "Enter" to reboot.
On reboot, you should see a prompt for a passphrase. This passphrase is used to decrypt the machine's /var partition so that it can be re-encrypted via the TPM. Enter "insecure" — this passphrase is not relevant to our security architecture. If Secure Boot is not enabled, you'll instead see a note about needing to enable Secure Boot. The machine will auto-boot you into the BIOS. On reboot after that, you should see the passphrase prompt.
The /var partition should encrypt and expand, and you should then find yourself in Basic Configuration Wizard. Proceed to that section.
On VxMark, if you find yourself on an unexpected screen after the above steps, e.g., a Secure Boot error screen or booting straight into a previously installed image, you may need to manually edit the VxMark boot order. You can follow these instructions to do so:
Power off the machine.
Insert the vx-iso USB drive.
Power on the machine and auto-boot to vx-iso.
Use "Ctrl+C" to leave the main vx-iso interface and access a terminal.
Type efibootmgr to list out the boot entries. The output will look something like this:
Identify the boot entry for the recently installed image. Let's say in this case we want vxadmin-signed, Boot0002.
Run the following command, replacing the index, to make that entry the first in the boot order after the USB drive:
By default, newly created VMs have networking enabled. To ensure the offline VM does not have access to the internet, the networking link is disabled before the VM is ever used. This is accomplished by editing the VM's configuration (found in /etc/libvirt/qemu/offline.xml) and setting the link state to down. This functionality can be seen in the vxsuite-build-system repo, in .
If an offline VM exists, the XML configuration file is updated to set the link state to down. After that, the offline VM is explicitly re-defined from this updated XML configuration file. These steps always execute, even if the network link state is already set to down.
Any VM cloned from this offline VM will also have a disabled network since those settings are inherited from the original VM.
Virt Manager uses a local bridge network to connect to VMs. If you receive an error when starting a VM that says "Error starting domain: Cannot get interface MTU on 'virbr0': No such device", run the following command in the terminal on the build machine:
You should now be able to return to the VM and start it without issue.
Once a machine has been imaged with a signed image, you can verify the system hash against the hash of what was built and signed during the Trusted Build process. To perform this verification, you have two options.
This approach allows you to verify the system hash using a tool outside the system itself. vx-verifier is a modified version of vx-iso.
Power the machine down.
Insert the vx-verifier USB drive.
Power the machine on. It should auto-boot to the vx-verifier USB drive.
Navigate to the "Verify Hash" option, attaching a keyboard if necessary.
This will calculate and output the system hash of the installed image, which can be checked against the system hash recorded during the Trusted Build process.
Once you have verified the hash, you can press "enter" to reboot and remove the vx-verifier USB drive.
This approach allows you to verify the system hash from within the system. See .
