Real Time Test Dashboard using Prometheus and Grafana
In this article I’ll be sharing Prometheus and Grafana integration for Selenium tests real time report which in this example is written in Java.
Prometheus is a software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using a HTTP pull model, with flexible queries and real-time alerting. Meanwhile, Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.
Pre-requisites:
- Docker Desktop: https://docs.docker.com/desktop/windows/install/
- Prometheus: https://prometheus.io/download/ (this example uses Prometheus for Windows)
- Grafana: https://grafana.com/grafana/download (this example uses Standalone Windows Binaries)
Part 1 — Setup Prometheus Pushgateway
The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway. The Pushgateway then exposes these metrics to Prometheus.
One way to deploy the Pushgateway is using the prom/pushgateway Docker image, Pushgateway will be running in port 9091here, pull and run the image as follows:
> docker pull prom/pushgateway
> docker run -d -p 9091:9091 prom/pushgateway
Part 2 — Setup Selenium Test
In this example the metrics that I’m trying to gather is Total of Executed Tests and Total of Failing Tests . Here’s how it’ll will work:
- The steps in @BeforeAll and @AfterAll will allow pushing the contents of CollectorRegistry to Pushgateway in localhost:9091.
requests.inc();will increase theTotal of Executed Testscount at the end of each tests.
@ExtendWith(ExecutionWatcher.class)
public class MyTest {
WebDriver driver;
static CollectorRegistry registry;
static Counter requests;
public static Counter failedRequests;
public static Counter passedRequests; @BeforeAll
public static void setUp() {
registry = new CollectorRegistry();
requests = Counter.build()
.name("total_tests")
.help("Number of tests run.")
.register(registry);
failedRequests = Counter.build()
.name("failed_tests")
.help("Number of failed tests.")
.register(registry);
passedRequests = Counter.build()
.name("passed_tests")
.help("Number of passed tests.")
.register(registry);
} @BeforeEach
public void startTest(TestInfo testInfo) {
ExtentTestManager.startTest(testInfo.getDisplayName());
driver = new ChromeDriver();
} @Test
@DisplayName("Navigate to google and verify page title")
void myTest() {
driver.get("https://www.google.com");
Assertions.assertTrue(driver.getTitle().contains("Google"));
} @Test
@DisplayName("Navigate to GitHub and verify page title")
void myTest() {
driver.get("https://www.github.com");
Assertions.assertTrue(driver.getTitle().contains("GitHub"));
} @AfterEach
public void teardown() {
driver.quit();
requests.inc();
} @AfterAll
public static void tearDown() throws IOException {
PushGateway pg = new PushGateway("127.0.0.1:9091");
pg.pushAdd(registry, "my_batch_job");
}
}
For Total of Failing Tests I’ve used TestWatcher that will perform increment on the counter depending on the test result whether it has passed or failed. More information on TestWatcher can be found in this JUnit documentation.
public class ExecutionWatcher implements TestWatcher {
@Override
public void testFailed() {
failedRequests.inc();
} @Override
public void testSuccessful();
passedRequests.inc();
}
}
Part 3 — Setup Prometheus
Download Prometheus installation file, once you’ve extracted the .zip file you’ll find the a file called prometheus.yml
Set targets to where you’d like Prometheus to run on.
global:
scrape_interval: 15s
evaluation_interval: 15salerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093rule_files:
# - "first_rules.yml"
# - "second_rules.yml"scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090","localhost:9091"]
Run the prometheus.exe file using the configured .yml file as follows:
> prometheus --config.file=prometheus.yml
Part 4 — Setup Grafana
Before setting up Grafana using Prometheus, installation guidelines can be found here.
- Add Prometheus into Grafana Data Source through Configuration > Data sources > Add Data Source > Select
Prometheus> And setURLandAccessas follows. Click onSave and Testand ensure that the data source is working.
2. Create new dashboard and add empty panel
3. Set and save the configuration of the new panel as follows:
- Data source: Prometheus
- Add new Queries search for
total_testsandfailed_testswhich you have specified as counter name in your Selenium Test
4. Run the Selenium test and the metrics will be displayed in Grafana once all the tests are completed.
Conclusion
This integration was quite a challenge for me as this was my first time working with Prometheus and all the articles I found for Selenium Test Reports in Grafana would generally use InfluxDB instead of Prometheus. In the future I would love to work on more type of metrics besides test counter to identify metrics such as:
- Which category of tests are the ones that most often fail/flakey
- Whether there are specific times a test would be fail due to server / 3rd party which will also let us know how we can improve our system reliability
- Most common reason a test would fail based on the test logs
References
Besides using Prometheus, Selenium real-time report can also be generated using Grafana and InfluxDB which you can read in the following articles:
- https://medium.com/@kishansk26/real-time-selenium-reports-using-grafana-influxdb-4606ad80781c
- https://www.vinsguru.com/selenium-webdriver-real-time-test-metrics-using-grafana-influxdb/
The following documentations has helped me greatly in this learning process:
