Automated tests

Unit & end-to-end tests

All the projects of the Bulletin Board are comprehensively covered by unit tests, while the Bulletin Board server is also end-to-end tested with Cypress exploiting the sandbox.

You can execute the whole test suite by running make test from the root folder of the repository.

If you want to only run the tests of one project, you can do it by running make test_server, make test_client, make test_dummy, make test_electionguard singularly.

All tests are run in the CI process.

Load tests

The aim of load tests is to make sure the system can handle the expected load and to spot possible performance leaks that only become apparent with high loads of traffic.

Load tests have to be tailored to the event we are preparing for, and as such we need to configure and run them manually.

In our case, we will primarily focus on load testing the Bulletin Board during election time.

Test plan

We divide the test phase in 4 steps:

  1. Prepare a production-like Bulletin Board environment to execute the test

  2. Generate random, encrypted votes

  3. Send the votes to the Bulletin Board with JMeter

  4. Gather and evaluate the results

1. Prepare the environment

  • The Bulletin Board comes with a sandbox that ships some tools to help with the load tests. Make sure you set the SANDBOX environment variable to any value to activate it;

  • The web server and the background worker server shall run in separate machines in order for web traffic and background jobs not to affect each other;

  • Use Sidekiq as queue adapter;

  • In bulletin_board/server/config/settings.yml, set iat_expiration_minutes to few hours/days. This is the amount of time a vote is considered valid from its emission - since the vote generation phase could take a few hours, you don’t want it to be too low and have the Bulletin Board to reject the votes when you start the load test.

2. Generate the votes

As mentioned above, the sandbox provides some utilities to help us to run a load test. Between them, there’s the vote generation.

The vote generation utility emulates what happens in the voters' browser when they encrypt the vote, but it does it on the server and in bulk. Notice that the votes are randomly generated - from the selected answers to the voter ids.

To generate the votes, follow these steps:

  • Access the sandbox at /sandbox/elections/ - make sure the sandbox is activated;

  • Set up a new election. Notice that the more questions and answers, the slower the encryption and decryption process will be. It is hence important to have a similar election setup to the one we are preparing for to have valid results from this test. Also, make sure you select electionguard as voting scheme name;

  • Perform the Key Ceremony following the wizard and start the voting period;

  • You can now click on stress tests to access the vote generation utility.

    Votes generation in testing

  • Input the number of votes you want to generate and hit Generate votes.

The vote generation process is heavy and can lead to high memory consumption. With 1Gb of RAM you could run out of memory if you generate more than 2000 votes in one go. You can launch several consecutive generations and merge the results at the end.

The vote generation process can take a long time (proportionally to the number of votes). Avoid launching other generation processes when one is already running. You can check Sidekiq’s queues at /sidekiq to see whether the process is still running.

  • The generation process uploads a CSV with the votes data to transfer.sh. Once the process is finished, you can download the CSV, hitting Download votes. Notice that you’re also prompted the time when the file has been uploaded. That is for you to know when the new file is available in case you want to launch another votes generation right away.

    Download votes in testing

  • If you generated the votes in more than one process, merge the CSVs into one.

3. Send the votes to the Bulletin Board

We are now ready to start the actual load test.

We will employ Apache JMeter 5.4.1 and feed it with the CSV we just generated.

Download and unarchive JMeter. Open its UI by running:

./bin/jmeter

Open the following file to use as a template for the load test configuration:

jmeter-bb-load-test-config.jmx

You will need to adjust some configs to your needs:

  • In HTTP Request Defaults put the hostname/IP of your BB installation in Server Name or IP;

  • In CSV Data Set Config select the CSV with the generated votes in Filename;

  • In Thread Group you can choose:

    • How many votes to cast (Number of Threads);

    • What span of time should the votes be sent (Ramp-up period).

      This configuration will send 1 vote every 0

      This configuration will send 1 vote every 0,5 seconds

To check if the configuration is correct, send a low number of votes (e.g. Number of Threads = 2). Hit the play button and check in HTTP Request → View Results Tree if there is any error.

Once you are happy with the configuration, hit the save button and close the JMeter UI. To run the actual test, we will use JMeter in the more lightweight CLI mode.

Here is the command:

./bin/jmeter -n –t jmeter-bb-load-test-config.jmx -l testresults.jtl

Every 30 seconds, JMeter will prompt some stats in the terminal, so you can follow along the execution of the test. It also logs the results in testresults.jtl, that you can later analyse.

While JMeter is running the test, you’ll want to check the Sidekiq queues state on its dashboard at /sidekiq.

4. Gather the results

There are three main metrics we can measure:

  1. Roundtrip time for a request from JMeter;

  2. Votes processing time;

  3. Server resources consumption.

Roundtrip time: we can measure the requests roundtrip time over time to check if the web server can handle the predicted volume of traffic over a long period of time. Remember that the votes are not processed synchronously: every request enqueues a job that is processed in the background, so this only covers the web server load. This metric is provided by JMeter itself using the testresults.jtl generated during the test (see https://jmeter.apache.org/usermanual/generating-dashboard.html)

Votes processing time: the Bulletin Board sandbox also provides a tool to gather some basic metrics of the vote processing. Click on Load test stats to gather the stats of the votes processed during the last 2 hours.

Server resources consumption: we can check how the server(s) reacted to the load by checking the resource consumption over the time - memory, CPU, response time.