# Logging All VxSuite applications use the same framework to log user actions, application processes, and errors. The logs are persisted to disk and can be exported by election managers or system administrators. ## Application Log Events VxSuite applications use a [shared logging library](https://github.com/votingworks/vxsuite/tree/v4.0.2/libs/logging) to capture required or otherwise important application log events. The core metadata for each application log event maps roughly to VVSG 2.0 standards, with some additional fields: * **Log Event ID** - The unique identifier for each event. The full list of log event IDs with descriptions is included in the automatically generated log [documentation](https://github.com/votingworks/docs-vxsuite-v4/blob/main/software-docs/VotingWorksLoggingDocumentation.md). * **Log Event Type** - Describes whether the event involves an action or a status update, and whether it originated with a user, the application, or the larger system. Log event types can be thought of as categories whereas log event IDs are more specific. Every log event type includes many log event IDs under it. The full list of log event types with descriptions is included in the automatically generated log [documentation](https://github.com/votingworks/docs-vxsuite-v4/blob/main/software-docs/VotingWorksLoggingDocumentation.md). * **User** - If there was an authenticated user role at the time of a user action, describes the role as `system_administrator`, `election_manager`, `poll_worker` or `vendor`. If no user was authenticated, then `unknown`. If event does not originate from the user but rather automatically from the application or operating system, it will be `system`. * **Disposition** - For actions that can fail or succeed, `success` or `failure`. Otherwise `na` for not applicable. * **Source** - Indicates where in the system's architecture the log came from, whether it be the frontend renderer, the frontend server, the backend server, a hardware daemon, or the operating system. * **Message** - The log message itself, which may not be present if the log event ID sufficiently describes the event. * **Other Metadata** - Other metadata can be freely included. For example, if a user adds manual tallies on VxAdmin, the metadata of those manual tallies will be included in the logging. * **Time Written** - Timestamp of when the log was created. ## Log Management The shared logging library formats the log events as JSON and pushes them to application processes' `stdout`. All `stdout` output from the application processes - both the application log events and any process output - is sent to the system's `logger` utility and assigned the tag `votingworksapp`. VxSuite uses `rsyslog`, an advanced implementation of the Syslog protocol, to centralize and manage logs from `votingworksapp` and other sources. `rsyslog` has advantages over the built-in Linux `syslog` implementation of the Syslog protocol, including JSON structured messages, advanced filtering, and increased performance. Using `rsyslog`, different types of logs are directed to different log files in `/var/log/votingworks/`:
Log FileContents
vx-logs.log

Key logs required by VVSG 2.0 or otherwise critical for understanding the behavior of the application or device, including:

  • all application log events
  • USB device connection events
  • machine boot and shutdown events
  • all sudo actions
  • password events
  • dm-verity events
auth.logAll operating system authentication-related events. Note that this will not include voting system authentication events such as logging in with a smart card - these events are in vx-logs.log
syslogAll other events emitted from the application or the operating system that do not fall into the other log files.
`vx-logs.log` is the most important and informative file in most cases. The other files rarely need to be referenced. All logs required by VVSG certification will exist in `vx-logs.log.` Because each log file can grow very large, the application will rotate logs if necessary with the `logrotate` utility on each startup. Previous log files will be compressed and suffixed with a timestamp. Including the various log files and log rotation, below is an example of the list of log files that might appear on a device: 
/var/log/votingworks/vx-logs.log
/var/log/votingworks/vx-logs.log-20241104.gz
/var/log/votingworks/vx-logs.log-20241105.gz
/var/log/votingworks/auth.log
/var/log/votingworks/syslog
/var/log/votingworks/syslog-20241105.gz
## Default VotingWorks Log Format ```json { "timeLogWritten":"2024-11-05T15:51:51.246232-08:00", "host":"VotingWorks", "source":"vx-scan-backend", "eventId":"auth-login", "eventType":"user-action", "user":"poll_worker", "message":"User logged in.", "disposition":"success", } ``` The default VotingWorks log format is how logs are formatted when emitted from the application and how they appear in `vx-logs.log`. It is a direct mapping of the[#application-log-events](#application-log-events "mention") fields. ## CDF Log Format Logs can be exported in CDF format, in which case the `vx-logs.log` file is replaced by a `vx-logs.cdf.json` file with the logging attributes mapped as follows:
CDF Attribute
ElectionEventLog.GeneratedTimeISO formatted timestamp of when logs were exported
ElectionEventLog.DeviceList containing only the current device
Device.Type

The CDF DeviceType matching the type of machine:

  • VxAdmin -> "ems"
  • VxScan -> "scan-single"
  • VxCentralScan -> "scan-batch"
  • VxMarkScan -> "bmd"
Device.IdThe serial number of the device
Device.VersionThe software version e.g. "v3.1.2"
Device.EventList of all events
Event.IdA VotingWorks defined identifier for the type of event, such as "save-election-package-complete"
Event.Disposition"success", "failure", "na", or "other"
Event.OtherDispositionIf the disposition is "other", the details appear here
Event.SequenceThe index of the log in the list of logs, zero-indexed
Event.TimeStampISO formatted timestamp of when the event was logged
Event.TypeLog event type
Event.DescriptionLog message
Event.DetailsJSON including the log source and any additional logging metadata
Event.UserIdThe user at the time of the log
## Error Logs The log export flow exposed in all VxSuite applications also allows for exporting an errors-only version of the logs. This will only export a log file for logs where the disposition is "failure" and covers all software and hardware errors that have occurred on the machine. ## 15.1-D Logging Event Types Table A full list of all logs made in the system with a description of each one can be found [here](https://github.com/votingworks/docs-vxsuite-v4/blob/main/software-docs/VotingWorksLoggingDocumentation.md). For convenience a table is provided below mapping Table 15-1 from VVSG 2.0 Requirement 15.1-D to the appropriate logs in the VotingWorks system. The details and descriptions provided in Table 15-1 have been simplified for brevity in this table. Some items have been expanded into multiple rows to more precisely specify linked logs. #### General System Functions | System Event | Detail | Logging Details | | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Device generated error and exception messages | Entries into exception handling routines |

Errors can be logged with any LogEventId but are indicated by a failure Disposition. All logs contain a Source. Every error is logged when it occurs, and each log can be assumed to correspond to one instance of that error occurring unless otherwise stated in the log.

Of particular note are logs with LogEventId unknown-error. These represent unexpected system errors, as opposed to user errors.

| | Device generated error and exception messages | Notifications of physical violations of security |

VxMarkScan physical violations of security are covered by printer-scanner cover-open logs. These will have a LogEventId of paper-handler-state-machine-transition with a body Transitioned to: "cover\_opened\_unauthorized"


The VxScan scanner cover is intended to be opened by poll workers for maintenance, so this does not constitute a physical violation of security. Nonetheless, logs are also emitted for this case. These will have a LogEventId of scanner-state-machine-transition with body Transitioned to: "coverOpen"

| | Device generated error and exception messages | Other exception events such as power failures, failure of critical hardware components, data transmission errors, or other types of operating anomalies |

On VxMarkScan, hardware failures are covered by logs with LogEventId paper-handler-state-machine-transition and body Transitioned to: "no\_hardware"


On VxScan, hardware failures are covered by logs with LogEventId scanner-state-machine-transition and body Transitioned to: "disconnected"

| | Device generated error and exception messages | Recovery actions taken |

Recovery actions will vary for every error and can therefore map to any log. Recommended recovery actions for user-facing errors are documented in our user manual under the following pages:

VxAdmin Error Messages

VxCentralScan Error Messages

VxMarkScan Error Messages
VxScan Error Messages

| | Critical system status messages | Diagnostic and status messages upon startup |

machine-boot-init
machine-boot-complete

| | Critical system status messages | The "zero totals" check |

polls-opened
toggled-test-mode

| | Critical system status messages | The initiation or termination of scanner and communications equipment operation |

scanner-state-machine-event

scanner-state-machine-transition

mark-scan-state-machine-event

paper-handler-state-machine-transition

| | Critical system status messages | Printer errors |

printer-status-changed and printer-print-complete with a failure Disposition cover VxScan.

paper-handler-state-machine-transition covers VxMarkScan.

For the VxAdmin printer, we emit a printer-config-added log when VxAdmin successfully registers the printer as connected, and we emit a printer-config-removed log when VxAdmin registers the printer as disconnected. If the printer is connected but the last printer log is printer-config-removed, it can be inferred that there's a problem.

For the VxCentralScan imprinter, an imprinter-status log is emitted whenever paper scanning is initiated. The log indicates in its message whether or not the imprinter is registering as attached. It can be inferred that there's a problem if the imprinter is attached, but the log indicates that it is not attached. Note that the VxCentralScan Fujitsu scanner requires you to power cycle it after you either attach or detach the imprinter. If you detach the imprinter while the Fujitsu scanner is on, an indicator on the Fujitsu scanner screen will say that the imprinter cover is open. No scans are accepted in this state. You have to power cycle the Fujitsu scanner to dismiss the warning and renable scanning. After this is done, the logs will indicate on next scan that the imprinter is detached. Vice versa, if you attach the imprinter while the Fujitsu scanner is on, the device jams. It takes a power cycle to get it to register the imprinter attachment.

| | Critical system status messages | Detection or remediation of malware or other malicious software |

Malware or other malicious software is prevented from being added to VotingWorks systems via our System Integrity design. Rather than using a traditional "allowlist" that is only applicable to malware, we create a system-wide hash of the expected system content. If anything changes, whether via malware or some other method, the system will no longer boot.

Failed cryptographic boot validation intentionally stops the boot process before logging processes can even spin up, so no log will be found in the log file for this case. A representative message will be displayed on screen, however, e.g., "Verity device detected corruption after activation."

If you encounter a failed cryptographic boot validation - since it may be indicative of malware - make note of the error message and reach out to VotingWorks.

| | Critical system status messages | Cryptographic boot validation success/failure |

dmverity-boot with a success Disposition indicates successful cryptographic boot validation.

Failed cryptographic boot validation intentionally stops the boot process before logging processes can even spin up, so no log will be found in the log file for this case. A representative message will be displayed on screen, however, e.g., "Verity device detected corruption after activation."

If you encounter a failed cryptographic boot validation - since it may be indicative of malware - make note of the error message and reach out to VotingWorks.

| | Non-critical status messages | |

device-attached
usb-device-change-detected
scanner-state-machine-event

scanner-state-machine-transition

mark-scan-state-machine-event

paper-handler-state-machine-transition

Various other logs. In the event of failure, please make note of the failure and reach out to VotingWorks.

| | Events that require election official intervention | |

device-unattached
scanner-state-machine-event

scanner-state-machine-transition

mark-scan-state-machine-event

paper-handler-state-machine-transition
auth-login

Various other logs

| | Device shutdown and restarts | |

machine-shutdown-init indicates the start of a shutdown, and machine-shutdown-complete indicates the successful completion of a shutdown.

A restart will begin with the above two shutdown logs and then be followed by a machine-boot-init log, which indicates the start of a boot, and a machine-boot-complete log, which indicates the successful completion of a boot.

Abnormal shutdowns and restarts can take many forms in the logs.

A machine-shutdown-init log without a corresponding machine-shutdown-complete log indicates a failed shutdown or restart.

A machine-boot-init log without a corresponding machine-boot-complete log indicates a failed boot or restart.

Finally, a machine-boot-init log without a previous machine-shutdown-complete log indicates that the last shutdown was abnormal and likely the result of a loss of power rather than user-initiated.

| | Changes to system configuration settings | | It is not possible to change these configuration settings, and any attempt to do so will result in a `sudo-action` log. | | Integrity checks for executables, configuration files, data, and logs | |

For election package configuration, a failed integrity check will result in an election-configured log with a failure Disposition and Election package authentication erred in the message.


For CVR import, a failed integrity check will result in an import-cast-vote-records-complete log with a failure Disposition and authentication-error in the error details.


All of the above logs, if seen with a success Disposition, indicate successful integrity checks.

| | The addition and deletion of files | |

The overwhelming majority of machine data is stored in a database, the creation and destruction of which are captured in database-create-complete and database-reset-complete logs.


The only other files stored on the file system are scan images on VxCentralScan and VxScan. On VxCentralScan, scan image file creations pair with scan-sheet-complete logs, and scan image file deletions (without a complete unconfiguration) pair with clear-ballot-data-complete and delete-cvr-batch-complete. On VxScan, scan image file creations pair with scanner-state-machine-transition logs, where the body includes Transitioned to: "interpreting", and scan image file deletions (without a complete unconfiguration) pair with toggled-test-mode logs.


On all machines, user-triggered exports of reports and logs to USB drives pair with file-saved logs. We don't expose functionality to delete these files individually. They can be cleared by formatting the USB drive, however, for which a usb-drive-format-complete log will be emitted.

| | System readiness results | System pass or fail of hardware and software test for system readiness |

diagnostic-init
diagnostic-complete
election-configured
polls-opened
toggled-test-mode

| | System readiness results | Identification of the software release, identification of the election to be processed, polling place identification, and the results of the software and hardware diagnostic tests |

The software release can be verified using signed-hash-validation, for which a signed-hash-validation-complete log will be emitted.

The election, specifically ballot hash, can be found in the election-configured log.

The polling place can be found in the polls-opened log as well as the precinct-configuration-changed log if a change was made from the default selection.

The results of software and hardware diagnostic tests can be found in diagnostic-complete logs.

| | System readiness results | Pass or fail of ballot style compatibility and integrity test | The `election-configured` log with a `success` Disposition indicates a pass. That log with a `failure` Disposition indicates a fail. More on integrity checks a few rows up in this table. | | System readiness results | Pass or fail of system test data removal |

On all machines other than VxAdmin, the toggled-test-mode log with a success Disposition indicates a pass. There is no expected circumstance under which a toggled-test-mode log with a failure Disposition will be emitted. If such an error were to occur, the system would surface an error screen indicating an abnormal/unexpected error. VxCentralScan does disallow switching from official mode to test mode if ballots have been scanned and corresponding CVRs have not yet been exported, but this is accomplished by completely disabling the toggle.

On VxAdmin, the clear-imported-cast-vote-records-complete log with a success Disposition indicates a pass. There is no expected circumstance under which a clear-imported-cast-vote-records log with a failure Disposition will be emitted. If such an error were to occur, the system would surface an error screen indicating an abnormal/unexpected error.

| | System readiness results | Zero totals of data paths and memory locations for vote recording |

On VxScan, the polls-opened log with a success Disposition indicates a pass.

A polls-opened log with a failure Disposition and User prevented from opening polls because ballots have already been scanned in the body would hypothetically indicate a fail, but this log cannot actually be emitted in practice as we prevent any ballot scanning until polls have been opened.

On VxAdmin and VxCentralScan, a data-check-on-startup log indicates in its message whether election results or scanned ballot data are present in the database at machine startup.

| | Removable media events | | `usb-device-change-detected` | | Backup and restore | |

On VxCentralScan and VxScan, full CVR exports serve as backups. These pair with cast-vote-records-complete logs. This data is not, however, intended to be loaded back onto VxCentralScan and VxScan, so there is not an equivalent restore operation. Because VxMarkScan does not tabulate, there are no CVRs on it to backup.

On all machines, user-triggered exports of reports and logs to USB drives pair with file-saved logs. This data is also not meant to be loaded back into the system.

| #### Authentication and Access Control | System Event | Detail | Logging Details | | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Authentication related events | Login/logoff events |

auth-login
auth-logout
auth-pin-entry

| | | Account lockout events |

auth-pin-entry-lockout

auth-logout where the log's reason field contains session\_expired on VxMarkScan and VxScan, and machine\_locked\_by\_session\_expiry on VxAdmin and VxCentralScan

| | | Password changes |

smart-card-program-complete covers all smart card PIN changes.

For the vendor menu, accessed via a vendor smart card with a PIN and then additionally gated by a vendor password, changing the latter password results in a
password-change log.

| | Access control related events |

Use of privileges (such as a user running a process
as an administrator)

| `sudo-action` | | Access control related events | Attempts to exceed privileges | `sudo-action` with a message indicating that the action was disallowed | | Access control related events |

All access attempts to application and underlying
system resources

| To access underlying system resources, one would need to either break out of kiosk mode (which is not possible) or have access to a vendor card and its PIN. One would then need to use sudo. As such, a `sudo-action` log would be emitted. | | Access control related events |

Changes to the access control configuration of the
voting device

| The access control configuration is controlled by the system settings file within the election package. As such, the access control configuration can only be changed by reconfiguring with a new election package. This emits an `election-configured` log. | | User account and role (or groups) management activity | Addition and deletion of user accounts and roles |

New users are created by programming smart cards, for which a smart-card-program-complete log is emitted.

Users can be "deleted" by unprogramming smart cards, for which a smart-card-unprogram-complete log is emitted.

| | User account and role (or groups) management activity | User account and role suspension and reactivation | As noted above, `smart-card-program-complete` and `smart-card-unprogram-complete` cover this. | | User account and role (or groups) management activity | Changes to account or role security attributes such as password length, access levels, login restrictions, and permissions | Security settings are controlled by the system settings file within the election package. As such, security settings can only be changed by reconfiguring with a new election package. This emits an `election-configured` log. | | User account and role (or groups) management activity | Administrator account and role password resets |

The system administrator role is itself smart card authenticated, and password changes are accomplished by reprogramming, so smart-card-program-complete covers this.

For the vendor menu, accessed via a vendor smart card with a PIN and then additionally gated by a vendor password, changing the latter password results in a
password-change log. All other substantive vendor actions result in a sudo-action log.

| #### Networking | System Event | Logging Details | | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Enabling or disabling networking functionality | It is not possible to enable wired or wireless networking. Any attempt to bypass protections in place blocking this will result in an error and a `sudo-action` log. | #### Software | System Event | Detail | Logging Details | | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Installing, upgrading, patching, or modifying software or firmware | |

Because we update software by completely reimaging machines, all logs prior to updating are cleared. We procedurally make sure that logs are exported before beginning a software update.

A sudo-action log that references an expand-var-filesystem.sh command is indicative of a freshly imaged/updated machine.

The software version/hash, both before and after update, can be verified using signed-hash-validation, for which a signed-hash-validation-complete log will be emitted.

The software version/hash is also logged on every boot in a subset of the dmverity-boot logs. Searching for verity.hash in the vx-logs.log file will help you hone in on these logs.

| | Changes to configuration settings | Changes to critical function settings, including location of election definition file, contents of the election definition file, vote reporting, location of logs, and voting device configuration settings |

The location of the election definition and logs cannot be changed; both are stored in static locations. Changes to the contents of the election definition pair with election-configured and election-unconfigured logs.

The are no controls outside of election definition configuration that affect how votes are reported.

Other voting device configuration settings are covered by continuous-export-toggled, double-sheet-toggled, sound-toggled, and toggled-test-mode logs.

| | Changes to configuration settings | Changes to device settings including, but not limited to, enabling and disabling services | As noted above, device settings are covered by `continuous-export-toggled`, `double-sheet-toggled`, `sound-toggled`, and `toggled-test-mode` logs. | | Changes to configuration settings | Starting and stopping processes |

For lower-level hardware daemons run in a separate process, we have process-started and process-terminated logs. The only machine that currently runs a lower-level hardware daemon in a separate process is VxMarkScan for the accessible controller and PAT input.

The only known circumstance under which a process-terminated log can be emitted is during VxMarkScan electrical testing. Shocking the PAT input with an ESD pulse can cause the hardware daemon to terminate and restart.

| | Abnormal process exits | |

unknown-error

There is no expected circumstance under which a process will exit abnormally. If such an error were to occur, the system would surface an error screen indicating an abnormal/unexpected error.

| | Successful and failed database connection attempts (if a database is used) | |

database-connect-init
database-connect-complete

There is no expected circumstance under which a database connection attempt will fail. If such an error were to occur, the system would surface an error screen indicating an abnormal/unexpected error.

| | Changes to cryptographic keys | | A machine TPM key is created during initial configuration by VotingWorks after software imaging. This operation pairs with a `sudo-action` log that references a `generate-key.sh` command. No additional keys can be added, nor can this key be deleted. The key can however be changed by VotingWorks, via use of a vendor smart card. The change will also show up in the logs as a `sudo-action` that references the `generate-key.sh` command. | #### Voting Functions | System Event | Detail | Logging Details | | ---------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Ballot definition and modification | |

election-configured

election-unconfigured

| | Voting events | Opening and closing polls |

polls-opened

polls-closed

voting-paused

voting-resumed

reset-polls-to-paused

These logging events are the internal tests that verify that the prescribed closing or suspension procedures have been followed.

| | Voting events | Casting a vote |

VxMarkScan: vote-cast

VxScan: scanner-state-machine-event with the event scanStart to indicate the voter has started feeding a ballot. If the voter has to cast from adjudication there will be an accept event.

| | Voting events | Canceling a vote during verification |

VxMarkScan: ballot-invalidated

VxScan: scanner-state-machine-event with event reject

| | Voting events | Success or failure of log and election results exportation |

export-cast-vote-record-complete
file-saved

|