It’s one of these questions, that happen for a reason. And usually not a good one. Something broke, our regression tests sound alarm. It’s time for test case analysis – the faster the better. baangt just got a lot more useful in helping you to find your answers quickly! Let’s dive into how a good workflow to answer this question looks.
Houston, we have a problem in one of our test cases. Analysis please!
We receive the result of one of the test runs, which shows red – the indicator for status “failed”. We have a problem, but using baangt’s built-in capabilities we also have a clear path to resolution.
Identify what the problem is
The test case is in status “failed”. To find out, which error is thrown exactly, we simply open the result file from whichever channel we received it, head to the sheet “output” and look at the column “TCErrorLog”. Here we will find a meaningful description of the error. Just to the right of this column – in case of Web-Tests – we will find a screenshot, that shows exactly where the problem occurred. That’s usually already two very helpful sources of information, that can lead us into the right direction:
Analysis of test case errors is easy with baangt test automation software
Test Case Analysis: Did the application change or did we change or both?
Now we have a few major questions, which we will answer using a few smart steps:
- When did this test case work properly for the last time?
- Does this test case have a history of flakyness?
- When was the test data last changed?
- When was the test case last changed?
Step 1: Use built-in reporting capabilities help to narrow down the reason
One way to get this information is using the built in reporting capabilities of baangt:
In the second tab of this sheet you can filter for e.g. one of the test cases and see in a clear sequence, which values of which attributes have changed between the different executions. No other open source test automation tool offer anything similar!
This image shows for each test case and test data combination in which test run and on which stage it was successful. Seeing this information can help us in test case analysis to identify flaky test cases or see a tendency or a trend, of when a test case works well and when it fails.
Step 1: Quicker info with less depth – test case execution history
!Sic. This is simply another way to execute step 1. Which one you prefer in your current test case analysis depends on your setup. You will chose the way that works best and fastest for your current situation.
Sometimes we’re more interested in a specific test case data combination. Now, of course we always have the chance to execute this one testcase in manual mode, with or without debug and with or without slow mode, where we would see each and every click and interaction, that happens during this test case. This usually takes longer time than a quick sanity check directly in the test case.
Exactly for this quick check you can open the cloned test data definition file and see all necessary details within just a few seconds:
Here we see for each of our test data definitions how it performed in the previous executions. For each line we see the result, the duration and in case of errors the exact error text. A second, convenient and very fast way for an analysis
When was the latest change of test data?
Another important question in test case analysis. Answering it fast can save us and colleagues many hours of tedious work time. As a result we built the change logs into each and every file! Whenever baangt uses a file, not only will we create a copy before saving anything. We also analyze the contents and write each and every change into the change log. Here an example for test data:
When did the test case definition change last?
That’s the last question to give us a good overview, where to start looking and how we might solve the situation with minimum effort for all participants. You guessed it already – also in the test case definition (simple format and complex format) we write a change log, of course with timestamp, previous value, new value.
Test case analysis might be a bad idea after all
Depending on our setup and use case, don’t forget to check, if you have set the proper test case version for the stage, that you are currently testing. Especially when neither test case nor test data definition was changed, the test case worked on lower stages but refused to work on e.g. final quality, you might save yourself and the developers some headache by double-checking the version number in the test run. If you encounter that more often than usual (e.g. because you work in a very complex system environment) consider adding this step as the first item in your checklist.
That’s all for today, thanks for reading!
Recent Comments