JIRA is popular among project managers, developers and QA experts for its project management capabilities and issue-tracking features. For QA professionals, JIRA provides solid features of bug tracking and requirements management that are deeply customizable, alongside great team collaboration so everyone can benefit throughout the dev cycle.
What's the problem then? Well, one outstanding shortcoming of JIRA is the lack of decent test case management. Test case management such as test case creation, execution, and tracking are non-existent within this tool, yet an add-on called Zephyr makes a valiant attempt to bridge this gap. This add-on brings many good features to meet the need as well as Zephyr reports to feedback on this activity, however, as we shall see, it still has its shortcomings.
In this article, we will examine the pros and cons of Zephyr.
Zephyr for JIRA
Zephyr was built for test case management and it leverages JIRA’s existing bug-tracking features to manage test cases and their execution. It facilitates work in real time between developers, QA, and project/product managers, thus allowing for close monitoring and updates of the build for overall improved software quality assurance. Although Zephyr has limited reporting ability within JIRA, test case planning and execution is easy and integrated with the JIRA bug tickets; thereby saving time by identifying the execution of previously failed test cases when a bug has been fixed.
Zephyr integrates seamlessly with JIRA as seen in the above screen shot of the main project screen - after installation of the plugin a new ‘Tests’ menu option is included in the main navigation bar, and when expanded displays the list of test case management actions available. The options include creation, execution and management of tests, plus test summary and test metrics for monitoring the QA process. Additional options offer help on the use of this tool.
Creating Tests with JIRA
Test cases are written in the same way in JIRA as all other tickets and issues are created - using the ‘Create’ button on the top right hand side of the screen. Test cases, indicated by a new icon - - are a unique type of issue, with no change to the other standard types of Epic, Story, Improvement and Task. Writing test cases is frankly much easier and more efficient in a spreadsheet tool like Excel than in Zephyr primarily because of the ability to navigate in Excel via keyboard - ie. using the arrow and tab keys. On the browser screen in Zephyr the QA engineer must tab through multiple fields in order to get from one test case to another, so when creating the initial test plan with potentially dozens or even hundreds of tests, this can quickly become tedious.
It would therefore be natural to ask - why would a QA team opt to work with this tool? Because it does offer some important and useful test case management features that a simple spreadsheet does not, such as the following:
- While creating a test case, comments relating to it can be added by the user to provide more information about it. Other team members can also provide helpful inputs by using the ‘Comments’ tab, enhancing collaboration and teamwork.
- Test cycles can be created and attached to specific versions of a release.
- Test cases can be assigned to specific test cycle(s) during its creation, ensuring timely execution.
- Files like screenshots and login information can be attached to the test case using the ‘Attach Files’ tab.
- Zephyr allows a test case to be linked to one or more other issues like Epic, Story or even to another test case. This ensures that the linked test cases are kept under consideration while the developers and the QA team work on them. By linking a test case to an Epic, the QA team has the ability to easily run a full regression on a particular part of the product.
- Test cases can be easily cloned, creating a duplicate copy which has the same test details. The cloned test is a separate item and not connected to the original test.
- Entire test cycles can also be cloned, allowing an easy method to roll an entire test plan from one version to the next, or perhaps from one type of test plan to another (for example, from a sanity test plan into the regression plan).
- Zephyr leverages the existing Projects, Components, Labels or Versions in JIRA to make it easier for categorizing test cases in a flexible manner. One test can be part of more than one version of your application. All the tests grouped under one Project or one ‘Label’ can be viewed easily. For instance, if a messenger app is being tested, test cases will have to be written relating to the Chat feature. They can all be grouped under the label ‘Chat’ and’ can be viewed together.
- Search Tests: Like other issues in Zephyr, tests are easy to search for by using the ‘Search Tests’ option in the ‘Tests’ tab.
- Helpful tip: use Excel to perform initial rapid test case creation and modification; once the test plan is stable import these test cases into Zephyr for management, execution, and reporting.
Planning the Test Cycle
Zephyr allows the user to create multiple test cycles and view them under their respective versions. Tests can be added singly to an existing test cycle or cloned as a group from another test cycle (also, one test can be part of more than one test cycle). Once all the tests are added to your test cycle the execution phase is clearly organized and tracking of results is easy. During execution a progress tracking bar displays the tests executed and the ones remaining as a percentage. After execution is complete for the current test cycle, create the next test cycle (typically associated with the next version), and clone your desired tests into this test cycle and you're ready to go when the developers next release to your QA environment.
Executing the Tests
This option allows the user to execute any test case after accessing its particular test cycle; the user can even run the test immediately without including it in any test cycle (this is considered an ad hoc execution). The execution and tracking of tests becomes easier if they are grouped under test cycles. This may result in the user raising a new issue (e.g. a bug ticket), attaching the test to an already existing issue, attaching visual evidence of the bug and even inserting comments.
Test Summary and Metrics with a Zephyr report
Zephyr has a simple and basic reporting scheme as seen below. For a particular project, the number of total tests executed and the tests remaining can be seen in the test summary. The tests can also be seen based on the specific Version or any particular Label under which they are grouped.
The ‘Test Metrics’ option in the ‘Tests’ tab displays the current status of your project with the help of pie charts and graphs, listing the execution details by day, by test cycles or by the name of a particular tester so that their output can be monitored. The test metrics can be customized to reflect the test execution status of a particular version or for a specific time period.
Limitations of Zephyr
We've discussed many of the benefits of this add-on to JIRA, but Zephyr has some important limitations which must be taken into consideration when evaluating whether this is the proper test case management tool to adopt for your efforts:
Excel wins for efficiency
As mentioned previously, writing test cases is far more efficient in Excel. One workaround is to create in Excel and import into Zephyr for life cycle management. The test cases can be edited within the tool as updates occur and test execution status changes.
Issues with rapid editing
Rapid editing is inefficient - Each edit must be individually saved and the user attempting to use their keyboard must tab multiple times to get to the next/previous row. No real workaround is available for this; and once the test cases are in Zephyr, it doesn't make sense to export the data to excel, edit it there, and re-import since you'll lose all useful history.
There is no facility in Zephyr to group together and list all the test cases which are linked to the same issue. If you have a bug ticket related to a handful of tests, there’s not a simple method to view and execute each of these tests again; this would be a useful way for a team lead to assign work to QA members if it were available.
Historical reporting lacks some granularity
The status of a single test case execution across multiple application versions cannot be displayed in any sort of report or query. For example, if you have a regression test effort across multiple app versions, you cannot view the results of these individual tests (and entire test cycles) across all these versions. I as the QA lead cannot easily tell that minor test case TC-131 went from pass-pass-pass over the last 3 application versions, whereas critical test TC-223 had a history of pass-fail-fail for those same versions. Having this sort of historical reporting allows the QA lead deeper understanding of the test results and comprehensive risk analysis when reporting to management prior to a production release.
Visualization and new cycles
Although there is a way to see a specific list of tests in a previous app version ( for example, all failed tests in the previous version), there is no way to add them as a group to a test cycle for future execution. The QA engineer must manually note each test and individually add them to another cycle. And while writing test cases, images cannot be linked to individual test steps.
Field values change in the reporting when updated
Field values are tied to the actual test case ticket and not to the test ticket in relation to a given test run or test cycle, therefore any history reporting will display the latest value regardless of what that value was in the historical cycle. For example, if TC-133 was changed from a low priority in an earlier version to a higher priority in a later one, if I run a report on that earlier execution results I'll see the current priority of that test case and not the actual value of that field when it was executed. This can cause confusion and potential miscommunication.
So is Zephyr worth it?
In a software testing service such as what OnPath Testing provides, using the Zephyr plug-in for JIRA can be a great way for QA, project managers, and developers to manage and monitor tests and execution status in real time. It is a useful tool, however the limitations as discussed above must be taken into account and understood in order for a QA team to get the most benefit out of their test effort, while supplying the necessary details when reporting to management.
Good luck, and if you have also used this tool please let us know your thoughts!