Examples of Performance metrics

Metrics in Performance testing

As with any other kind of testing, metrics are critical in Performance testing. They enable the tester, developers and stakeholders understand how well their system is performing against what they expect.

You should note that the metrics collected for different performance tests will differ based on varying factors but the common ones are listed as below:

1. Technical Environment

Performance metrics will vary by the type of technical environment the test is carried out on. Examples of these environments include:

• Web-based
• Mobile
• Desktop devices
• Server side processing
• Databases
• Networks

The metrics collected from a technical environment can include any of the following:

• Response time (e.g per transaction, per concurrent user, page load times)
• Resource utilisation (CPU, Memory, network bandwidth, latency, I/O rate, available disk space, idle and busy threads)
• The number of transactions that can be performed in a given period of time
• Number of errors impacting performance
• Completion times for CRUD operations
• Background load on shared resources such as virtual environments.

2. Business environment

From the business perspective of the tests, the metrics will differ and include but are not limited to the following:

• Business process efficiency — The speed of performing an overall business process including the normal, alternate and exceptional use case flows.
• Throughput of data, transactions and other units of work performed such as orders processed per hour.
• Service Level Agreement compliance or violation rates.
• Scope of usage — How many users are performing transactions at a given time and can they be accommodated for?
• The number of users concurrently performing a task
• The number of orders processed during peak load time(when the number of users is max)

3. Operational environment

The operational aspect of performance testing focusses on tasks that are not considered to be user-facing in nature and includes the following metrics:

• Operational processes — The time required for environmental startups, shutdowns, backups and resumption.
• System restoration — The time required to restore data from a backup.
• The time needed to issue an alert or warning.

Please note that collecting more metrics than required is not a good practice. Each metric chosen will require a means for consistent collection and reporting. It is therefore important to define an obtainable set of metrics that suit your performance objectives.

Now that we know what types of metrics we should be collecting across the various aspects of Performance testing, let’s take a look at the ways in which we can collect these metrics.

Sources of performance Metrics

Before you proceed to chose a way to collect your metrics, you should be aware of the ‘Probe effect’.

The Probe effect is an unintended impact in the system’s performance during the process of metric collection. We should aim to minimise this impact as much as possible when collecting metrics.

Performance metrics can be collected from the following sources:

• Performance Test Tools

These tools provide measurements and metrics as the result of a test. They include load generators such as WebLOAD and LoadNinja

• Performance Monitoring Tools

These tools are used to supplement the Performance Test Tools. In addition, they are used to monitor system performance on an ongoing basis and alert the administrators in case of any errors, alerts or security attacks. An example is TraceView.

• Log Analysis Tools

These tools scan the server logs and compile metrics from them. Some of these tools such as Stackify can create charts to provide a graphical view of the data.

Errors, alerts and warnings are stored in server logs. They include:

1. High resource utilisation
2. Memory errors and warnings
3. Deadlocks and multi-threading problems
4. Database errors such as SQL timeouts.

That brings us to the end of this informative article. This should give you an understanding on what results to look for when carrying out your performance tests and what type of tools to use to get these results. I hope you found this useful!

Cheers.