Logging

All VxSuite applications use the same framework to log user actions, application processes, and errors. The logs are persisted to disk and can be exported by election managers or system administrators.

Application Log Events

VxSuite applications use a shared logging library to capture required or otherwise important application log events. The core metadata for each application log event maps roughly to VVSG 2.0 standards, with some additional fields:

• Log Event ID - The unique identifier for each event. The full list of log event IDs with descriptions is included in the automatically generated log documentation.

• Log Event Type - Describes whether the event is a status update or an action, and whether it originated with the user, application, or larger system. The full list of log event types with descriptions is included in the automatically generated log documentation.

• User - If there was an authenticated user role at the time of a user action, describes the role as system_administrator, election_manager, poll_worker or vendor. If no user was authenticated, then unknown. If event does not originate from the user but rather automatically from the application or operating system, it will be system.

• Disposition - For actions that can fail or succeed, success or failure. Otherwise na for not applicable.

• Source - Indicates where in the system's architecture the log came from, whether it be the frontend renderer, the frontend server, the backend server, a hardware daemon, or the operating system.

• Message - The log message itself, which may not be present if the log event ID sufficiently describes the event.

• Other Metadata - Other metadata can be freely included. For example, if a user adds manual tallies on VxAdmin, the metadata of those manual tallies will be included in the logging.

• Time Written - Timestamp of when the log was created.

Log Management

The shared logging library formats the log events as JSON and pushes them to application processes' stdout. All stdout output from the application processes - both the application log events and any process output - is sent to the system's logger utility and assigned the tag votingworksapp.

VxSuite uses rsyslog, an advanced implementation of the Syslog protocol, to centralize and manage logs from votingworksapp and other sources. rsyslog has advantages over the built-in Linux syslog implementation of the Syslog protocol, including JSON structured messages, advanced filtering, and increased performance. Using rsyslog, different types of logs are directed to different log files in /var/log/votingworks/:

Log File	Contents
vx-logs.log	Key logs required by VVSG 2.0 or otherwise critical for understanding the behavior of the application or device, including: all application log events USB device connection events machine boot and shutdown events all sudo actions password events dm-verity events
auth.log	All operating system authentication-related events. Note that this will not include voting system authentication events such as logging in with a smart card - these events are in vx-logs.log
syslog	All other events emitted from the application or the operating system that do not fall into the other log files.

vx-logs.log is the most important and informative file in most cases. The other files rarely need to be referenced. All logs required by VVSG certification will exist in vx-logs.log.

Because each log file can grow very large, the application will rotate logs if necessary with the logrotate utility on each startup. Previous log files will be compressed and suffixed with a timestamp.

Including the various log files and log rotation, below is an example of the list of log files that might appear on a device:

/var/log/votingworks/vx-logs.log
/var/log/votingworks/vx-logs.log-20241104.gz
/var/log/votingworks/vx-logs.log-20241105.gz
/var/log/votingworks/auth.log
/var/log/votingworks/syslog
/var/log/votingworks/syslog-20241105.gz

Default VotingWorks Log Format

{
    "timeLogWritten":"2024-11-05T15:51:51.246232-08:00",
    "host":"VotingWorks",
    "source":"vx-scan-backend",
    "eventId":"auth-login",
    "eventType":"user-action",
    "user":"poll_worker",
    "message":"User logged in.",
    "disposition":"success",
}

The default VotingWorks log format is how logs are formatted when emitted from the application and how they appear in vx-logs.log. It is a direct mapping of theApplication Log Events fields.

CDF Log Format

Logs can be exported in CDF format, in which case the vx-logs.log file is replaced by a vx-logs.cdf.json file with the logging attributes mapped as follows:

CDF Attribute
ElectionEventLog.GeneratedTime	ISO formatted timestamp of when logs were exported
ElectionEventLog.Device	List containing only the current device
Device.Type	The CDF DeviceType matching the type of machine: VxAdmin -> "ems" VxScan -> "scan-single" VxCentralScan -> "scan-batch" VxMark -> "bmd"
Device.Id	The serial number of the device
Device.Version	The software version e.g. "v3.1.2"
Device.Event	List of all events
Event.Id	A VotingWorks defined identifier for the type of event, such as "save-election-package-complete"
Event.Disposition	"success", "failure", "na", or "other"
Event.OtherDisposition	If the disposition is "other", the details appear here
Event.Sequence	The index of the log in the list of logs, zero-indexed
Event.TimeStamp	ISO formatted timestamp of when the event was logged
Event.Type	Log event type
Event.Description	Log message
Event.Details	JSON including the log source and any additional logging metadata
Event.UserId	The user at the time of the log

Error Logs

The log export flow exposed in all VxSuite applications also allows for exporting an errors-only version of the logs. This will only export a log file for logs where the disposition is "failure" and covers all software and hardware errors that have occurred on the machine.

15.1-D Logging Event Types Table

A full list of all logs made in the system with a description of each one can be found here. For convenience a table is provided below mapping Table 15-1 from VVSG 2.0 Requirement 15.1-D to the appropriate logs in the VotingWorks system. The details and descriptions provided in Table 15-1 have been simplified for brevity in this table. Some items have been expanded into multiple rows to more precisely specify linked logs.

General System Functions

System Event	Detail	Logging Details
Device generated error and exception messages	Entries into exception handling routines	Errors can be logged with any LogEventId but are indicated by a failure Disposition. All logs contain a Source. Every error is logged when it occurs, and each log can be assumed to correspond to one instance of that error occurring unless otherwise stated in the log. Of particular note are logs with LogEventId unknown-error. These represent unexpected system errors, as opposed to user errors.
Device generated error and exception messages	Notifications of physical violations of security	VxMark physical violations of security are covered by printer-scanner cover-open logs. These will have a LogEventId of paper-handler-state-machine-transition with a body Transitioned to: "cover_opened_unauthorized" The VxScan scanner cover is intended to be opened by poll workers for maintenance, so this does not constitute a physical violation of security. Nonetheless, logs are also emitted for this case. These will have a LogEventId of scanner-state-machine-transition with body Transitioned to: "cover_open"
Device generated error and exception messages	Other exception events such as power failures, failure of critical hardware components, data transmission errors, or other types of operating anomalies	On VxMark, hardware failures are covered by logs with LogEventId paper-handler-state-machine-transition and body Transitioned to: "no_hardware" On VxScan, hardware failures are covered by logs with LogEventId scanner-state-machine-transition and body Transitioned to: "disconnected"
Device generated error and exception messages	Recovery actions taken	Recovery actions will vary for every error and can therefore map to any log. Recommended recovery actions for user-facing errors are documented in our user manual under the following pages: VxAdmin Error Messages VxCentralScan Error Messages VxMark Error Messages VxScan Error Messages
Critical system status messages	Diagnostic and status messages upon startup	machine-boot-init machine-boot-complete
Critical system status messages	The "zero totals" check	polls-opened toggled-test-mode
Critical system status messages	The initiation or termination of scanner and communications equipment operation	scanner-state-machine-event scanner-state-machine-transition mark-scan-state-machine-event paper-handler-state-machine-transition
Critical system status messages	Printer errors	printer-status-changed printer-print-complete paper-handler-state-machine-transition With a failure Disposition
Critical system status messages	Detection or remediation of malware or other malicious software	Malware or other malicious software is prevented from being added to VotingWorks systems via our System Integrity design. Rather than using a traditional "allowlist" that is only applicable to malware, we create a system-wide hash of the expected system content. If anything changes, whether via malware or some other method, the system will no longer boot and will present a dm-verity failure message. dm-verity with a failure Disposition
Critical system status messages	Cryptographic boot validation success/failure	dmverity-boot with success or failure indicated by the Disposition
Non-critical status messages		device-attached usb-device-change-detected scanner-state-machine-event scanner-state-machine-transition mark-scan-state-machine-event paper-handler-state-machine-transition Various other logs
Events that require election official intervention		device-unattached scanner-state-machine-event scanner-state-machine-transition mark-scan-state-machine-event paper-handler-state-machine-transition auth-login Various other logs
Device shutdown and restarts		machine-shutdown-init indicates the start of a shutdown or restart. If followed by a machine-shutdown-complete log, the shutdown or restart can be taken to be successful. Otherwise, it can be considered abnormal. In the case of a restart, we additionally expect a machine-boot-init and machine-boot-complete log.
Changes to system configuration settings		It is not possible to change these configuration settings, and any attempt to do so will result in a sudo-action log.
Integrity checks for executables, configuration files, data, and logs		For election package configuration, a failed integrity check will result in an election-configured log with a failure Disposition and Election package authentication erred in the message. For CVR import, a failed integrity check will result in an import-cast-vote-records-complete log with a failure Disposition and authentication-error in the error details. For the file system as a whole, a failed integrity check will result in a dm-verity log with a failure Disposition. All of the above logs, if seen with a success Disposition, indicate successful integrity checks.
The addition and deletion of files		The overwhelming majority of machine data is stored in a database, the creation and destruction of which are captured in database-create-complete and database-destroy-complete logs. The only other files stored on the file system are scan images on VxCentralScan and VxScan. On VxCentralScan, scan image file creations pair with scan-sheet-complete logs, and scan image file deletions (without a complete unconfiguration) pair with clear-ballot-data-complete and delete-cvr-batch-complete. On VxScan, scan image file creations pair with scanner-state-machine-transition logs, where the body includes Transitioned to: "interpreting", and scan image file deletions (without a complete unconfiguration) pair with toggled-test-mode-complete logs. On all machines, user-triggered exports of reports and logs to USB drives pair with file-saved logs. We don't expose functionality to delete these files individually. They can be cleared by formatting the USB drive, however, for which a usb-drive-format-complete log will be emitted.
System readiness results	System pass or fail of hardware and software test for system readiness	diagnostic-init diagnostic-complete diagnostic-error election-configured polls-opened toggled-test-mode
System readiness results	Identification of the software release, identification of the election to be processed, polling place identification, and the results of the software and hardware diagnostic tests	The software release can be verified using Signed Hash Validation, for which a signed-hash-validation-complete log will be emitted. The election, specifically ballot hash, can be found in the election-configured log. The polling place can be found in the polls-opened log as well as the precinct-configuration-changed log if a change was made from the default selection. The results of software and hardware diagnostic tests can be found in diagnostic-complete and diagnostic-error logs.
System readiness results	Pass or fail of ballot style compatibility and integrity tes	The election-configured log with a success Disposition indicates a pass. That log with a failure Disposition indicates a fail. More on integrity checks a few rows up in this table.
System readiness results	Pass or fail of system test data removal	The toggled-test-mode log with a success Disposition indicates a pass. The toggled-test-mode log with a failure disposition indicates a fail.
System readiness results	Zero totals of data paths and memory locations for vote recording	The polls-opened log with a success Disposition indicates a pass. The polls-opened log with a failure Disposition and User prevented from opening polls because ballots have already been scanned in the body indicates a fail.
Removable media events		usb-device-change-detected
Backup and restore		On VxCentralScan and VxScan, full CVR exports serve as backups. These pair with cast-vote-records-complete logs. This data is not, however, intended to be loaded back onto VxCentralScan and VxScan, so there is not an equivalent restore operation. Because VxMark does not tabulate, there are no CVRs on it to backup. On all machines, user-triggered exports of reports and logs to USB drives pair with file-saved logs. This data is also not meant to be loaded back into the system.

Authentication and Access Control

System Event	Detail	Logging Details
Authentication related events	Login/logoff events	auth-login auth-logout auth-pin-entry
	Account lockout events	auth-pin-entry-lockout auth-logout where the log's reason field contains session_expired
	Password changes	smart-card-program-complete covers all smart card PIN changes. For the vendor menu, accessed via a vendor smart card with a PIN and then additionally gated by a vendor password, changing the latter password results in a password-change log.
Access control related events	Use of privileges (such as a user running a process as an administrator)	sudo-action
Access control related events	Attempts to exceed privileges	sudo-action with a message indicating that the action was disallowed
Access control related events	All access attempts to application and underlying system resources	To access underlying system resources, one would need to either break out of kiosk mode (which is not possible) or have access to a vendor card and its PIN. One would then need to use sudo. As such, a sudo-action log would be emitted.
Access control related events	Changes to the access control configuration of the voting device	The access control configuration is controlled by the system settings file within the election package. As such, the access control configuration can only be changed by reconfiguring with a new election package. This emits an election-configured log.
User account and role (or groups) management activity	Addition and deletion of user accounts and roles	New users are created by programming smart cards, for which a smart-card-program-complete log is emitted. Users can be "deleted" by unprogramming smart cards, for which a smart-card-unprogram-complete log is emitted.
User account and role (or groups) management activity	User account and role suspension and reactivation	As noted above, smart-card-program-complete and smart-card-unprogram-complete cover this.
User account and role (or groups) management activity	Changes to account or role security attributes such as password length, access levels, login restrictions, and permissions	Security settings are controlled by the system settings file within the election package. As such, security settings can only be changed by reconfiguring with a new election package. This emits an election-configured log.
User account and role (or groups) management activity	Administrator account and role password resets	The system administrator role is itself smart card authenticated, and password changes are accomplished by reprogramming, so smart-card-program-complete covers this. For the vendor menu, accessed via a vendor smart card with a PIN and then additionally gated by a vendor password, changing the latter password results in a password-change log. All other substantive vendor actions result in a sudo-action log.

Networking

System Event	Logging Details
Enabling or disabling networking functionality	It is not possible to enable wired or wireless networking. Any attempt to bypass protections in place blocking this will result in an error and a sudo-action log.

Software

System Event	Detail	Logging Details
Installing, upgrading, patching, or modifying software or firmware		Because we update software by completely reimaging machines, all logs prior to updating are cleared. We procedurally make sure that logs are exported before beginning a software update. A sudo-action log that references an expand-var-filesystem.sh command is indicative of a freshly imaged/updated machine. The software version/hash, both before and after update, can be verified using Signed Hash Validation, for which a signed-hash-validation-complete log will be emitted. The software version/hash is also logged on every boot in a subset of the dmverity-boot logs. Searching for verity.hash in the vx-logs.log file will help you hone in on these logs.
Changes to configuration settings	Changes to critical function settings, including location of election definition file, contents of the election definition file, vote reporting, location of logs, and voting device configuration settings	The location of the election definition and logs cannot be changed; both are stored in static locations. Changes to the contents of the election definition pair with election-configured and election-unconfigured logs. The are no controls outside of election definition configuration that affect how votes are reported. Other voting device configuration settings are covered by continuous-export-toggled, double-sheet-toggled, sound-toggled, and toggled-test-mode logs.
Changes to configuration settings	Changes to device settings including, but not limited to, enabling and disabling services	As noted above, device settings are covered by continuous-export-toggled, double-sheet-toggled, sound-toggled, and toggled-test-mode logs.
Changes to configuration settings	Starting and stopping processes	For lower-level hardware daemons, we have process-started and process-terminated logs.
Abnormal process exits		unknown-error
Successful and failed database connection attempts (if a database is used)		database-connect-init database-connect-complete
Changes to cryptographic keys		A machine TPM key is created during initial configuration by VotingWorks after software imaging. This operation pairs with a sudo-action log that references a generate-key.sh command. No additional keys can be added, nor can this key be deleted. The key can however be changed by VotingWorks, via use of a vendor smart card. The change will also show up in the logs as a sudo-action that references the generate-key.sh command.

Voting Functions

System Event	Detail	Logging Details
Ballot definition and modification		election-configured election-unconfigured
Voting events	Opening and closing polls	polls-opened polls-closed voting-paused voting-resumed reset-polls-to-paused
Voting events	Casting a vote	VxMark: vote-cast VxScan: scanner-state-machine-event with the event scanStart to indicate the voter has started feeding a ballot. If the voter has to cast from adjudication there will be an accept event.
Voting events	Canceling a vote during verification	VxMark: ballot-invalidated VxScan: scanner-state-machine-event with event reject
Voting events	Success or failure of log and election results exportation	export-cast-vote-record-complete file-saved