Logging into Arlo will be required to upload audit files, download audit materials, and obtain your jurisdiction's audit report.

To login to Arlo:

1. Using Chrome or Safari as your browser, go to https://arlo.voting.works/.

2. Enter your email address.

3. Click Log in to your audit.

4. Check your email for a new email with a subject “Welcome to Arlo - Use the Code in this Email to Log In” - enter the verification code. This email comes from rla@vx.support, be sure to add this email address to your contacts, safe sender, or whitelist.

5. Select your audit (when applicable).

Arlo has three types of user roles. Depending on the audit, users may play multiple roles.

Audit Administrator

The audit administrator is responsible for setting up the audit, monitoring progress during the file upload and ballot retrieval process, starting additional rounds of auditing, and downloading the final audit results.

Audit administrator user credentials are set up by VotingWorks.

Jurisdiction Manager

The jurisdiction manager is responsible for uploading all necessary files prior to the audit and coordinating the retrieval of ballots/batches and associated data entry.

Jurisdiction manager user credentials are set up by audit administrators as participants in the audit setup process.

Audit Board

The audit board is responsible for retrieving ballots and entering their marks into Arlo.

Audit board user credentials are set up by jurisdiction managers by selecting the number of audit boards.

Tally Entry

In batch comparison audits, the audit board may not be designated to enter batch tallies into Arlo. Tally entry user account(s) can be created as needed to enter batch tallies. The Jurisdiction Manager may also enter batch tallies.

Arlo is software for managing and conducting a Risk-Limiting Audit (RLA). The code for Arlo is open-source and freely available for anyone to use and host. VotingWorks provides a hosted version of Arlo and customer support when conducting an RLA.

Arlo supports ballot polling, batch comparison, ballot comparison, and hybrid RLAs for single jurisdiction to statewide audits.

RiskLimitingAudits.org describes the history and process of conducting an RLA.

Arlo software supports a number of RLA types to provide a statistically sound method for any paper ballot jurisdiction. VotingWorks' RLA team can help you choose which RLA type is best for your jurisdiction.

Ballot Polling

Consists of the retrieval of randomly selected ballots to confirm the outcome of the election. Only requires secure storage of paper ballots. The selected voted ballots are hand tallied, and the resulting manual vote counts are checked against the total vote counts.

Batch Comparison

Consists of the retrieval of randomly selected batches of ballots. A batch may consist of all the ballots cast in a precinct, on a specific ballot scanner, or smaller tabulated batches. Those batches are hand tallied and counts are compared to the reported results, batch by batch, to confirm the outcome of the election. Requires vote counts by batch.

Ballot Comparison

Consists of very precise retrieval of randomly selected ballots. Each ballot is compared to the voting system’s cast vote record (CVR) to confirm the outcome of the election. Requires CVRs from the voting system and each ballot to be stored in the exact order it was scanned. The imprinting of a unique identifier (in no way connected to the voter) is recommended.

Hybrid

Combination of ballot polling & ballot comparison types. If some of a state’s jurisdictions have the technology to conduct ballot comparison audits and others do not, the hybrid model allows for an audit that utilizes both types.

Full Manual Tally

In some cases, an RLA may result in a full manual tally of all ballots. This may occur when the margin of the contest being audited is so close that the number of ballots to be audited is high and a full manual tally would be more efficient or when the results of multiple rounds detect anomalies in the results and require a full manual tally to confirm the outcome.

A key step in the risk-limiting audit process is building a ballot manifest. A ballot manifest is simply a list of all of the ballot containers and the number of ballots inside.

The ballot manifest is used by Arlo to build a list of every ballot in the election. Arlo takes the list of containers and number of ballots and creates a spreadsheet with a line for each ballot and the container that it is in. Later in the process, Arlo uses the list to randomly select ballots and tell the person auditing the ballots where to find each ballot selected.

Create a Ballot Manifest

Creating a ballot manifest is slightly different you are conducting. Use the tabs below to navigate to the correct instructions for your audit type.

Number of Ballots in Each Container​

The number of ballots stored in each container varies by state and jurisdiction. What’s important when creating the ballot manifest is to make sure the number being used is independent of the tabulator. The number of ballots tabulated is generally the right number but that number should have been independently verified on election night or after the election via a reconciliation process that includes comparing it to the number of voters.​​

All voted ballots should be included. Don’t forget provisional, UOCAVA, absentee, or other ballots counted after election day as required by law.

Ballot Manifest Creation

• Create a spreadsheet

• Title columns

• Enter a name for every batch of ballots tabulated, the number of ballots in that batch, and any other required information for your audit type

  • Include any late-tabulated batches (ex. provisional ballots that counted, late arriving absentees, etc.)

• Check the total number of ballots in the ballot manifest to ensure it matches the total number of ballots tabulated (do not include a totals row in the manifest)

• Save as a .csv file

• when prompted

Cast Vote Record Preparation

For ballot comparison and hybrid audits only.

• Export from voting system

Candidate Totals by Batch Creation

For batch comparison audits only.

• Create a spreadsheet

• Title column A "Batch Name" and additional columns each candidate name as provided by your audit administrator

• Verify each batch from your ballot manifest is included and the names match

• Save as a .csv file

Cast Vote Record Requirements

When retrieving cast vote records (CVRs) from your voting system, it’s important to follow your system’s instructions to ensure proper export. Most systems export groups of CVRs in a single file, but some systems export each individual ballot’s record as a separate file.

The following data must be included for each ballot (names may vary by voting system - see chart below):

• Unique ID for each CVR

• Tabulator ID/name/number

• Batch ID/name/number

• Ballot position within a batch Imprinted ID (if applicable)

• Recorded votes/marks for every contest on a particular ballot, each containing:

  • Contest name

  • Candidate name

  • Vote/mark (and override, if applicable)

NOTE: The “Tabulator name” and “Batch name” columns on your Ballot Manifest must match those found in the cast vote records.

Voting Equipment Specific Information

As mentioned above, we strongly encourage you to follow the instructions provided by your vendor when retrieving the cast vote records from your voting system. Some tips we’ve learned from other jurisdictions are provided below.

Dominion:

• The Cast Vote Record tabular box should be checked in the export menu

• Sometimes fields are exported with extra characters, like quotation marks and equal signs. To eliminate these extra characters, open the CVR file safely in Excel (open a blank worksheet, go to Data > From Text > select CVR file > Import) and use the ctrl-s function to re-save.

ES&S:

• ES&S does not provide a single CVR export with all the data needed for the audit. Instead, two exports are required:

  • Tools > Export Cast Vote Record

  • Produce Module > Ballots - Table View > set filter to “all” > Export (this file only allows 20K records to be exported at one time and may require several exports depending on your number of ballots)

• The tabulator IDs will be parsed from the CVR using the first four digits of the Tabulator CVR field, including any leading zeros. The tabulator names in the manifest must be four digits and include any leading zeros in order to match.

• If Arlo is not accepting the CVR file, change the file name to cvr.csv and re-upload.

Hart:

• A Hart CVR export is a ZIP file containing an individual XML file for each ballot.

• Either a single ZIP file can be provided or multiple, one for each tabulator. When multiple ZIP files are provided, the ZIP file names (with ".zip" removed) will be used as tabulator names.

• Separate from the ZIP files, optional scanned ballot information CSVs can be provided. If provided, the "Workstation" values in them will be used as tabulator names, and the "UniqueIdentifier" values in them will be used as imprinted IDs. Otherwise, "CvrGuid" values will be used as imprinted IDs.

• If both multiple ZIP files are provided and scanned ballot information CSVs are provided, the ZIP file names will take precedence over the "Workstation" values as tabulator names.

• Note that tabulator names are only used if batch names in the ballot manifest are not unique.

The CVR export process may be tricky the first few attempts. We are happy to review files prior to upload and encourage all users to work through this process with us well before audit day. Many jurisdictions have used their logic and accuracy testing CVR files to practice.

Field Mapping by Voting System

In order to conduct a batch comparison risk limiting audit, Arlo needs candidate vote totals for each batch.

Open a spreadsheet program and create a simple spreadsheet.

• Column A labeled “Batch Name” is for your batch name - this must match the batch names in your ballot manifest

• Additional columns should be labeled with the names of the target contest candidates or selections. Your audit administrator will provide this information to you.

Enter the data for your county or jurisdiction. When you're finished, take a second to ensure the total number of batches matches your number of batches in your ballot manifest and that the candidate totals are correct. Then save the file as a .csv file.

Once your file(s) have been created and have been saved as .csv files, they’re ready for upload into Arlo.

Login to Arlo

Click the Browse button and navigate to the ballot manifest .csv file that you created. Once selected, click Upload File.

Additional Files

Upload completion

When the upload(s) are successful, a message will appear at the top of the screen that says “1/1 or 2/2 files successfully uploaded.”

After you upload, if you find a mistake in the files, log back in and use the Replace File button to update the file.

If you are having problems uploading the file(s), verify you have saved the spreadsheet in .csv format, your columns are named correctly, and make sure there are no extra columns or rows. If you continue to have problems with your manifest, CVRs, or candidate totals by batch files, please send the file to rla@vx.support for review.

Ballot Retrieval Preparations

• Log in to Arlo when prompted

• Select number of audit boards to complete the work (one audit board can pull 20-25 ballots/hr)

• Print the ballot retrieval list

• Print the placeholder sheets on bright colored paper

• Print the placeholder labels on removable labels (Avery 6479)

• Print the audit board credentials

• Ensure each audit board retrieves the selected ballots and completes ballot entry into Arlo.

The tallies for each batch selected for audit must be entered into Arlo to conclude the audit. Tallies can be entered by audit boards, specified tally entry users, and/or jurisdiction managers.

To create tally entry accounts for audit boards or specified tally entry users, select Set Up Tally Entry Accounts on the Set Up Tally Entry Accounts screen. If the Jurisdiction Manager will be entering tallies, select Skip.

The Jurisdiction Manager should provide the Tally Entry User(s) with the login link displayed on the screen (each link is unique to the jurisdiction). Then the Tally Entry User(s) should Log In by entering their name(s) and party affiliation (if required). A login code will be provided, the Jurisdiction Manager should enter that code to authenticate the Tally Entry User(s) by selecting Enter Login Code, entering the code, and selecting Confirm to complete the login process.

Once the audit has been launched, each jurisdiction must login to download and print their audit materials.

Login to Arlo

Download and Print your Audit Materials

The following is our recommended supply list for ballot polling and ballot comparison audit boards:

• Table and chairs for each audit board

• Placeholder sheets generated from Arlo (print on bright colored paper)

• Ballot labels generated from Arlo (print on bright colored removable labels - Avery 6479)

• Audit board credentials from Arlo

• Tally sheets

• List of valid write-in candidates (where applicable)

• Voter intent guidelines

• Attention signs

• Sortkwik and/or rubber fingers

• Device to open containers

• Materials to reseal containers

• Colored pens [red is recommended]

• Device with wifi/cellular to enter ballot tallies into Arlo (laptop, smartphone, etc.)

The following is our recommended supply list for batch comparison audit boards:

• Table and chairs for each audit board

• Audit Board Batch Sheets (provided in Arlo)

• List of valid write-in candidates (where applicable)

• Voter intent guidelines

• Attention signs

• File folders or envelopes for ballots that need review (when a review board is utilized)

• Sortkwik and/or rubber fingers

• Device to open containers

• Materials to reseal containers

• Colored pens [red is recommended]

Ballot Retrieval

• Check out ballot container with ballots to audit

• Perform chain of custody tasks and open the container

• Retrieve ballots

• Close, reseal, and perform chain of custody tasks on the container

• Repeat until all ballots are retrieved

Ballot Entry into Arlo

• Record voter marks on a tally sheet (if applicable)

• Use the Audit Board Credential sheet to log in to Arlo

• Enter the audit board member names and select party affiliation (if required)

• Click Start Auditing

• Enter the ballot marks for the current ballot displayed on the screen

• Review the information entered for the ballot for accuracy

• Click Submit & Next

• Repeat until all ballots are entered

• Click Auditing Complete - Submit Results

• Enter the audit board member names exactly as you entered them at the start

• Click Sign Off

When all jurisdictions have completed their ballot/batch entry and the risk limit has been met, Arlo will provide each jurisdiction with a report that includes all of their ballot retrieval information. Click on the Download Audit Report button for a copy of the report.

Additional Sampling

If the risk limit was not met after all ballots have been audited in each participating jurisdiction, Arlo will immediately escalate to an additional round of sampling. Once the audit administrator has started the next round, a new ballot retrieval list, placeholders, labels, and audit board credentials will be generated for each jurisdiction required to retrieve more ballots.

After the audit begins, each jurisdiction with ballots selected will receive a ballot retrieval list. Generally this list is downloaded by the jurisdiction retrieving the ballots from Arlo, the audit software (see instructions above) but sometimes it may be sent to a county by the state or to a local jurisdiction by their county.

To retrieve ballots, refer to your ballot retrieval list to determine which container is needed.

Retrieve the correct container and perform required chain of custody verification steps. Then open the container. Pull the entire batch of ballots out of the storage container. They should be well organized but may need to be manipulated into a stack of ballots that is mostly neat.

Next refer back to the Ballot Number column of your retrieval list for the ballot number (or numbers) that you need. Start counting at one and stop counting when you get to the ballot number that you are retrieving.

Once you count to the ballot number on your retrieval list, pull the ballot out of the stack, place a colored sheet of paper in the ballot's place to serve as a placeholder, and stick a removable label on to the ballot. The placeholder and label can be generated from Arlo, but if they are not, use blank stock and write the batch name and ballot number on each. Put the retrieved ballot to the side until all ballots have been retrieved.

If more than one ballot is needed from the batch, keep counting until you have retrieved all of the ballots. Always double check the batch name with the retrieval list to ensure it is the same container.

Once the ballots have been retrieved from a container, reseal the container (keeping the ballots selected out) and complete any required chain of custody documentation.

The ballot mark for each ballot selected for audit must be entered into Arlo to conclude the audit. Arlo is flexible and allows the audit board to enter ballots as they are retrieved or the system allows you to skip and return to ballots that have not yet been retrieved. Many jurisdictions use a tally sheet to record the votes on the retrieved ballots prior to data entry into Arlo.

Jurisdiction Managers must print the audit board credentials for each audit board.

Using the audit board credentials provided, the audit board enters ballots into Arlo as follows:

Use the QR code or type the randomly generated url into your toolbar. This is different for everyone - do not use the url or QR code in the example.

Enter audit board member information. We recommend two people complete this task to ensure accurate data entry. Complete Party Affiliation information as required by your state.

3. Click Audit First Ballot or select a ballot and click Audit Ballot to begin entering ballots.

Ballot Card Data Entry Screen Overview

There is a Ballot Card Data Entry screen for each ballot. Under Audit Board Selections the batch name and ballot number of the ballot being audited is listed. If the ballot has not been retrieved, use the “Ballot Not Found” button. Use the “All Ballots” button to return to the full list of ballots selected for audit.

1. Select the candidate(s) marked on the ballot. Both audit board members should agree. Then select Submit Selections. No vote types: Overvote - select each candidate marked by the voter Contest not voted - select the Blank Vote button Contest not on ballot - select Not on Ballot button Invalid Write-In - select Invalid Write-In (if a contest has valid write-in candidates, a Write-In button will be listed with contest choices) If the audit board cannot agree - explain in the Add Note box.

2. On the verification screen, review the ballot information for accuracy and select Confirm Selections to go to the next ballot.

3. Once all ballots have been audited, select Submit Audited Ballots.

4. Enter auditor names exactly as they were entered at the start and click Sign Off.

Audited ballots should be resecured in a ballot container. Generally a new container is recommended but when very small numbers of ballots are retrieved, it may make more sense to return them to their original container.

After the audit begins, each jurisdiction with batches selected will receive a batch retrieval list. Generally this list is downloaded by the jurisdiction retrieving the batches from Arlo, the audit software but sometimes it may be sent to a county by the state or to a local jurisdiction by their county.

To retrieve batches, refer to your batch retrieval list to determine which container is needed. Each batch retrieval list will show the batch name and the audit board selected to tally the batch.

Retrieve the correct container and perform required chain of custody verification steps. Then open the container. Pull the entire batch of ballots out of the storage container and manipulate into a stack of ballots that is mostly neat. If the container contains more than 1000 ballots, ballots should be removed from the container and sorted in manageable stacks, leaving the rest of the ballots in the container until the previous stack is tallied.

Batch Vote Totals

Depending on the number of contests being audited, jurisdictions may use a variety of methods to tally the vote totals for each batch. The following are instructions for two common methods.

Adjudicating Ballots

Some ballots may need special attention. The following ballot types may need special review. Depending on the audit, these steps may be performed by the audit board or a special review team.

Duplicated ballots - ballots that were hand duplicated require the retrieval of the original ballot. Use the original ballot to determine the vote cast by the voter in the contest being audited.

Write-In ballots - in states where write-in ballots are required to be qualified, write-in ballots should be compared to the list of qualified write-in candidates and tallied if deemed qualified according to state rule or law.

Undetermined ballots - if an audit board can not agree on the voter’s mark(s), an additional person or designated team should review to make a determination. If a determination is not made, a remark can be made in Arlo to explain.

Final Steps

Once the batch has been tallied, return all ballots to the container, reseal the container, and complete any chain of custody documentation required. Return everything to the check-in/out station.

A state or jurisdiction administrator of the election should set up the audit in Arlo. To set up an audit in Arlo, using Chrome or Safari as your browser, go to https://arlo.voting.works/, and follow these steps:

1. Enter the name of your audit in the box provided.

2. Select the audit type.

3. Click Create Audit.

A standardized contests file tells Arlo which contests to expect and their correct names. A standardized contest file is also a two column spreadsheet. The first column titled “Contest Name” and the second column titled “Jurisdictions.” The contest name column should be populated with the contest names exactly as entered in your EMS system and the jurisdiction column should list each participant that has the contest on their ballots. If all participants have the contest, type “all.” Save the spreadsheet as a .csv file.

When the spreadsheet is ready, upload it into the Participants & Contests section of Arlo, as follows:

1. Select Browse and locate the standardized contests .csv file created.

2. Select Upload File.

3. Select Next.

Once the participant list is uploaded, the audit admin can monitor the file upload progress from the Audit Progress screen.

Below the map on the Audit Progress screen is a list of all jurisdictions and the various stages in the upload process they've completed.

Status: 0/2 files uploaded - jurisdiction has not logged in to Arlo and uploaded either file. 1/2 files uploaded - jurisdiction has only uploaded the ballot manifest.

2/2 files uploaded - jurisdiction has successfully uploaded their audit files. Upload failed - jurisdiction has attempted to upload one or more of the files but an error was found.

To review the files, select the Jurisdiction Name. Then select Download under the file name to review.

Ballots in Manifest is the total number of ballots the jurisdiction has in their manifest. This should match their number of ballots tabulated. If the expected ballots were entered into the Participant file, that colum will also be displayed with a third column showing the difference.

Information on the progress screen can be sorted in many different ways, as well as downloaded into a .csv file. Sort order will be maintained when the screen is refreshed.

Use Download Reported Results to download the reported results summed by jurisdiction. This is useful when comparing uploaded results with another data source.

Once your organization is created in Arlo by VotingWorks, using Chrome or Safari as your browser, go to https://arlo.voting.works/, and click on "Log in as an admin."

Enter your email address when prompted. Click on Forgot password? and a verification code will be sent to your email address for use as your password. Use the verification code as prompted to complete the login process. Future logins will not require the password reset.

The target contests are the required contests to be audited. Target contests will drive the ballot retrieval sample size and require the risk limit to be met.

In order to begin the audit, Arlo needs to know the participants. A participant (or Jurisdiction Manager) is usually a county or local jurisdiction responsible for creating their own pre-audit files and coordinating the retrieval of ballots. Prepare a participant list by creating a two column spreadsheet. The first column titled “Jurisdiction” and the second column titled “Admin Email.” The email address should be the email the user wants to use to retrieve login information. Save the spreadsheet as a .csv file.

An optional third column, titled Expected Number of Ballots can be added to the participant list. This number will help audit administrators compare the total number of ballots in the jurisdiction manager's ballot manifest with some other source of truth (ex. election night reporting, epollbook check-ins, voter history, etc.) quickly.

When the spreadsheet is ready, complete the steps in the Participants section of Arlo, as follows:

1. Select Browse and locate the participants .csv file created.

2. Select Upload File.

3. Select Next.

The completion of this step allows participants to begin to upload pre-audit files into Arlo. The remaining setup tasks may be completed during the upload process and before the audit begins.

The participant file can be updated as often as necessary prior to the launch of the audit.

An opportunistic contest is any other contest on the ballot the audit administrator would like to include. Some jurisdictions may choose to include each additional contest on the ballot while others may choose not to audit any additional contests. It’s important to note that an opportunistic contest with a lower margin of victory from the targeted contests is not likely to meet the risk limit with the targeted contest’s ballot retrieval sample size. Setup for opportunistic contests is exactly the same as targeted contests, see instructions above.

In the final audit settings, your state should already be set as a default but if not, select it from the drop down. The State does populate the map in the Audit Progress Screen. Then enter the Election Name, designate the desired risk limit, and enter the random seed.

The risk limit may be designated in state statute or rule. Be sure to verify your state's requirements.

Presentation mode may be used when entering the random seed to a public audience. Select Presentation Mode and a larger box only mean for random seed entry will open. Select Set Random Seed to save it after it's entered.

At this point the audit is ready to begin if all jurisdictions have uploaded the required files. Check the top of the screen for jurisdiction file upload progress before selecting Save & Next.

Creating a Batch

Many batches of ballots are created by circumstance. Ballots voted at a precinct on Election Day, vote center ballots collected at various times during early voting periods, centrally tabulated ballots by the number a scanner can tabulate in one pass, etc. Methodically storing ballots in smaller batches (when possible) will allow for easier reconcilation, ballot manifest creation, and RLA ballot retrieval. The more precise your ballot storage, the more RLA audit types you can utilize.

Ballots processed in a precinct or vote centers on Election Day are generally all stored together in one batch and keeping the ballots in the order they were scanned is not possible. The number of ballots tabulated should be compared to the number of voters issued ballots (excluding ballots not tabulated like provisionals). If multiple tabulators were used in a precinct, batching and storing the ballots by tabulator provides a way to store the ballots in smaller batches. They can be placed in the same container as long as the batches are clearly bundled and labeled (internally and externally).

Ballots processed in early voting sites over a period of time are generally stored when tabulator bins are full or at certain points in time. Batching and storing the ballots by tabulator and date will allow for regular reconciliation and smaller batches. They can be placed in the same container (re-securing each time a batch is added) as long as the batches are clearly bundled and labeled (internally and externally).

Ballots processed in a central location can be managed and stored in well defined batches. Some jurisdictions choose to pre-sort mail ballots by precinct, ballot style, or date returned. Batching as ballots are returned and processing as a batch from check-in, signature verification, envelope opening, through to tabulation.

Documenting a Batch

Precincts, vote centers, and early voting sites should at a minimum document and reconcile the number of ballots tabulated with the number of voters checked in. Use a simple precinct reconciliation form like the one displayed below. When ballots are able to be further batched by tabulator or date, this form can be modified to include multiple public counter boxes and tabulator ID numbers or set up in a format that allows for daily reconciliation. Use the information from your reconciliation forms to build your ballot manifest.

Centrally counted ballots go through multiple processing steps and managing the batches along the way with a Batch Cover Sheet is helpful. In the example provided below, you'll see two key sections.

First, a section for the processing team to name the batch, provide physical envelope counts, and a final ballot count. Noting differences as they process in smaller batches. The batch name could be a precinct name, ballot style, date, or combination of these elements.

The second section is for the tabulation team to verify the number of ballots tabulated matches the processing team's physical count and record any differences. If the tabulator provides a batch number, that number should be recorded as it ties the batch back to CVR records and batch totals.

The batch cover sheet should follow the batch of ballots throughout the process. We recommend printing on colored paper so they stand out from the white ballots.

Storing a Batch

Documenting the storage of each batch is important. While batch cover sheets should be stored with the batch, the information from the batch cover sheet needs to be available to build a ballot manifest as well as to facilitate the retrieval of ballots.

Use a Batch Tracking Sheet to record the information needed for each batch and the container it is stored inside. The tabulator batch number is only necessary for batch comparison, ballot comparison, and hybrid RLAs.

Finally, be sure to properly label the ballot container with the batch names and/or numbers stored inside.

After all setup screens are completed and all jurisdictions have uploaded their files, it is time to launch the audit.

First, take a moment to review all of the audit settings for accuracy. Arlo provides all of the details for final review.

The sample size for each contest will be displayed on the screen.

At this point, there are two options. The first is Preview Sample. This allows for a quick check to ensure the sample appears correct. A list of jurisdictions and their sample size will be displayed.

Select Launch Audit to officially begin the audit.

Once the audit is launched, the audit admin can monitor the audit progress from the Audit Progress screen.

Status: Not started - jurisdiction has not logged in to Arlo and downloaded the ballot retrieval list. In progress - jurisdiction has downloaded the ballot retrieval list. Complete - all ballots have been audited.

There are slight display differences depending on the audit type.

The Count unique sampled ballots toggle displays the raw number of ballots to be retrieved by each jurisdiction when toggled on. When toggled off, the number of ballots represents the entire sample size, including ballots sampled more than once that only need to be retrieved once.

All but ballot polling audits will have a discrepancy column. This highlights discrepancies that the audit administrator may want to review prior to the completion of the audit. Discrepancies will only display after a jurisdiction manager or all audit boards have finalized their data entry. To review the discrepancies, select the flag for a quick look or select the Download Discrepancy Report button for a full report.

The discrepancy will be displayed on screen or in the report and will identify the batch, the results from the audit, the reported results, the change in results, and the change in margin.

When all audit boards or jurisdiction managers have completed their ballot or batch entry and the risk limit has been met, the audit administrator finishes the round by selecting the Finish Round x button. This allows for discrepancy review before officially completing the round.

If the round completes the audit, the Audit Progress screen will display the message “Congratulations - the audit is complete!” Select the Download Audit Report button for a report that includes all of the audit data, including the risk measurement.

Additional Sampling

If the risk limit was not met after all ballots have been audited, Arlo will immediately escalate to an additional round of sampling. A new ballot retrieval list, placeholders, labels, and audit board credentials will be generated for each jurisdiction required to retrieve more ballots. Use the Start round x button to start the next round.

A room or facility where ballots can be reviewed securely is important.

At minimum, the room will need a 6’ table and chairs for each of the following:

• Ballot container check-in/out station (2 chairs)

• Each audit board (2 chairs)

• Any additional review stations as required

• A sign-in station for public and credential observers (1 chair)

Supplies needed for the room:

• Signage for credentialed observers to sign in

• Credentialed observers sign-in sheets

• Name tags for staff

• Tape for signs

• Pens

Below are various roles election administrators have found to be most helpful when conducting an audit. The sample size of ballots and/or batches to be audited will generally determine if these roles can be combined or if multiple people should be employed for each role.

Role #1 - Supervisor

The role of the Supervisor is to ensure the audit process is running smoothly and answer questions about the process. The Supervisor should not be assigned to one of the roles below, to ensure that they are free to troubleshoot and assist others as needed. In jurisdictions with many audit boards, more than one Supervisor may be helpful.

Role #2 - Check-In/Out Station

At a minimum, two people are needed to check ballots in/out to audit boards to maintain chain of custody as well as keep inventory of the ballot containers audited. One person should be stationed with the ballot containers at all times while another person delivers containers to and from the audit boards (a “runner”). This keeps the audit boards stationary and provides a methodical process for keeping track of each container. If you have more than a few audit boards, be sure to add an additional runner for every 5 audit boards to ensure the delivery and pick-up of ballot containers is not a bottleneck in the process.

Role #3 - Audit Boards

Audit boards must be comprised of at least two people and will be responsible for retrieving ballots for a ballot polling or ballot comparison audit or sorting and tallying the votes in a batch for a batch comparison audit. Be sure to keep tables clear of anything (especially food and beverage) other than ballots and required documentation. Audit boards are generally responsible for entering ballots into Arlo as well and will need access to a wifi or cellular enabled device.

Role #4 - Tally Entry User

For batch comparison audits, the tally entry process may be performed by someone other than the audit board. This may be a user who only enters batch tallies or combined with the check-in/out station, supervisor, or review board role. We recommend an additional person assist the person in this role to ensure accurate data entry.

Role #5 - Review Board

Some jurisdictions have rules requiring various types of ballots be escalated to a review team when an audit board can't agree or for more complicated determinations like write-in candidate adjudication or comparing duplicated ballots with their original ballots. Follow your state procedures to determine if this role is necessary.

Auditing of election results is necessary to ensure effective election administration and to build public confidence in the election results. Risk-limiting audits provide a more efficient manner of conducting audits than traditional audit methods and shall be used in every election in this state where there are state or federal contests.

The state’s chief election official, in consultation with local election officials, shall establish rules for conducting a risk-limiting audit after every election containing a state or federal contest.

The rules shall include:

1. Procedures to conduct a risk-limiting audit;

2. Criteria for which contest(s) must be audited;

3. Criteria to determine the scope of the risk-limiting audit; and

4. Criteria for public observation and the publication of audit results.

As used in this section, “risk-limiting audit” means an election audit that utilizes statistical principles and methods and is designed to limit the risk of certifying an incorrect election outcome.

If a risk-limiting audit of a contest leads to a full manual tally of all ballots cast, the vote counts according to that manual tally shall become the official contest results.

At a minimum, two people are needed to check ballots in/out to audit boards for chain of custody as well as keep inventory of the ballot containers audited. One person should be stationed with the ballot containers at all times while another person delivers containers to and from the audit boards (a “runner”). This keeps the audit boards stationary and provides a methodical process for keeping track of each container. If you have more than a few audit boards, be sure to add an additional runner for every 5 audit boards to ensure the delivery and pick-up of ballot containers is not a bottleneck in the process.

Check-Out

When a check-in/out station member delivers a container to an audit board, the batch name and audit board number should be recorded on a Ballot Container Inventory Sheet (example below).

Check-In

When an audit board indicates it has completed their tasks, the check-in/out station member should:

• verify proper completion of the required paperwork (Audit Board Batch Tally Sheet, Ballot Tally Sheet, etc.)

• ensure the container is resealed

• return the container and tally sheet to the check-in/out station

• note the return of the container on the Ballot Container Inventory Sheet

• deliver any necessary ballots/envelopes to any station that may provide further review (duplicates, write-ins, needs further review)

• enter the candidate totals for the batch into Arlo and mark the batch sheet as “entered” (for batch comparison audits)

Supply List

• Ballot Container Inventory Sheet

• Organizers for paperwork (tally sheets, ballots for review, etc.)

• Device with wifi/cellular to enter ballot tallies into Arlo (laptop, smartphone, etc.)

• Device to open containers

• Materials to reseal containers

• Colored pens [red is recommended]

RLA Basics

Audit Preparation

Audit Setup

Ballot Retrieval

Tally the Votes

Batch Retrieval

Batch Tally Entry

Audit Board Batch Tally Process - Sort & Stack

The audit report generated by Arlo after the completion of a risk-limiting audit contains all of the audit data. This guide explains the data found in each section.

Election Info Section

This election info section provides the basic election information for the audit.

Organization - name of jurisdiction conducting audit Election Name - name of election being audited State - State of jurisdiction conducting audit

Contests Section

The contests section provides the tabulated results for each contest subject to audit.

Contest Name - name of contest audited Targeted? - indicates whether a contest was targeted and subject to the risk limit or opportunistic and not subject to the risk limit Number of Winners - number of winners in contest audited Votes Allowed - number of votes allowed in contest audited Total Ballots - total number of ballots in contest audited. For hybrid audits, total votes will also be listed for each CVR and Non-CVR segment. Tabulated Votes - tabulated vote totals for each candidate in contest audited. For hybrid audits, tabulated votes will also be listed for each CVR and Non-CVR segment.

Audit Settings Section

The audit settings section provides all of the settings that were selected to drive the audit.

Audit Name - name of the audit Audit Type - risk-limiting audit type selected. Audit types include:

• Ballot Comparison

• Ballot Polling

• Hybrid (Ballot Comparison & Ballot Polling)

• Batch Comparison

Audit Math Type - math type selected - most audit types have one math type. Math types include:

• SUPERSIMPLE - Ballot Comparison

• BRAVO - Ballot Polling type

• Minerva - Ballot Polling type

• SUITE - Ballot Comparison & Ballot Polling Hybrid

• MACRO - Batch Comparison

Risk Limit - risk limit set by the audit administrator Random Seed - random number generated to start the audit Online Data Entry? - indicates whether an audit administrator selected online data entry, requiring audit boards to enter data for each ballot individually, or offline data entry, allowing the jurisdiction manager to enter sample totals

Audit Boards Section

This audit boards section provides the names and party affiliation (when required) of each audit board.

Jurisdiction Name - name of the audit board’s jurisdiction Audit Board Name - name of the audit board Member 1 & 2 Name - names of the audit board members Member 1 & 2 Affiliation - party affiliation of audit board members (when required)

Rounds Section

The rounds section includes the data completed for each round of the audit.

Round Number - number of the round Contest Name - name of the contest being audited Targeted? - indicates whether a contest was targeted and subject to the risk limit or opportunistic and not subject to the risk limit Sample Size - number of ballots retrieved for that contest during the round specified Risk-Limit Met? - Yes/No indicator if the sample met the risk limit P-Value - risk measurement for the round and contest specified Start Time - time the round was launched End Time - time the last audit board signed off in the round specified Audited Votes - audited sample vote totals for the round and contest specified

Sampled Ballots/Batches Section

The sampled ballots/batches section includes data on each ballot/batch sampled.

Jurisdiction Name - name of the ballot’s jurisdiction Container - container the ballot was retrieved from (if applicable) Tabulator - tabulator used to scan the ballot (if applicable) Batch Name - name of the batch the ballot was retrieved from (may include container name) Ballot Position - position in the batch of ballots the ballot was retrieved from (not applicable in batch comparison audits) Imprinted ID - tabulator imprint ID according to the voting system (applicable in ballot comparison and hybrid audits) Ticket Numbers - order in which the ballot was randomly selected Audited? - ballot audited status. Audited status includes:

• AUDITED - Ballot was audited

• NOT_AUDITED - Ballot was not audited

• NOT_FOUND - Ballot audit was attempted, ballot was not found.

Audit Result - vote(s) recorded from that ballot/batch for that contest. In addition to candidate/proposal votes, audit result may includes:

• BLANK - No votes were cast in that contest

• CONTEST_NOT_ON_BALLOT - Contest was not on the ballot audited

Reported Results - votes as reported in the contest totals by batch file (applicable to batch comparison audits) CVR Result - vote(s) as found in the voting system’s Cast Vote Record (CVR) for the ballot audited (applicable to ballot comparison and hybrid audits) Change in Results - differences in the audit and reported results (if applicable) Change in Margin - differences in vote margin between the audit and reported results (if applicable) Last Edited By - Jurisdiction Manager or Audit Board member who entered the audit results