JIRA is popular among project managers, developers and QA experts for its project management capabilities and issue-tracking features. For QA professionals, JIRA provides solid bug tracking and requirements management features that are deeply customizable. These sit alongside great team collaboration features that allow everyone to benefit throughout the dev cycle.
What's the problem then? Well, one outstanding shortcoming of JIRA is the lack of decent test case management. Test case management such as test case creation, execution, and tracking are non-existent within this tool, yet an add-on called Zephyr makes a valiant attempt to bridge this gap. This add-on brings plenty of features to address the lack of test management in JIRA, as well as 'Zephyr reports' to feed back on this activity. With this said, Zephyr for JIRA it still has its shortcomings - as we shall see.
Zephyr for JIRA
Zephyr was built for test case management. It leverages JIRA’s existing bug-tracking features to manage test cases and their execution. It facilitates work in real time between developers, QA, and project/product managers, thus allowing for close monitoring and updates of the build for overall improved software quality assurance. Although Zephyr has limited reporting ability within JIRA, test case planning and execution is easy and integrated with the JIRA bug tickets; thereby saving time by identifying the execution of previously failed test cases when a bug has been fixed.
Zephyr integrates seamlessly with JIRA, as seen in the above screen shot of the main project screen - after installation of the plugin a new ‘Tests’ menu option is included in the main navigation bar, and when expanded displays the list of test case management actions available. The options include creation, execution and management of tests, plus test summary and test metrics for monitoring the QA process. Additional options offer help on the use of this tool.
Creating Tests with JIRA
Test cases are written in the same way in JIRA as all other tickets and issues are created - using the ‘Create’ button on the top right hand side of the screen. Test cases, indicated by a new icon - - are a unique type of issue, with no change to the other standard types of Epic, Story, Improvement and Task. Writing test cases is frankly much easier and more efficient in a spreadsheet tool like Excel than in Zephyr primarily because of the ability to navigate in Excel via keyboard - ie. using the arrow and tab keys. On the browser screen in Zephyr the QA engineer must tab through multiple fields in order to get from one test case to another, so when creating the initial test plan with potentially dozens or even hundreds of tests, this can quickly become tedious.
It would therefore be natural to ask - why would a QA team opt to work with this tool? It does offer some important and useful test case management features that a simple spreadsheet does not, such as:
- While creating a test case, comments relating to it can be added by the user to provide more information. Other team members can also provide helpful inputs by using the ‘Comments’ tab, enhancing collaboration and teamwork.
- Test cycles can be created and attached to specific versions of a release.
- Test cases can be assigned to specific test cycle(s) during its creation, ensuring timely execution.
- Files like screenshots and login information can be attached to the test case using the ‘Attach Files’ tab.
- Zephyr allows a test case to be linked to one or more other issues like Epic, Story or even to another test case. This ensures that the linked test cases are kept under consideration while the developers and the QA team work on them. By linking a test case to an Epic, the QA team has the ability to easily run a full regression on a particular part of the product.
- Test cases can be easily cloned, creating a duplicate copy which has the same test details. The cloned test is a separate item and not connected to the original test.
- Entire test cycles can also be cloned, allowing an easy method to roll an entire test plan from one version to the next, or perhaps from one type of test plan to another (for example, from a sanity test plan into the regression plan).
- Zephyr leverages the existing Projects, Components, Labels or Versions in JIRA to make it easier for categorizing test cases in a flexible manner. One test can be part of more than one version of your application. All the tests grouped under one Project or one ‘Label’ can be viewed easily. For instance, if a messenger app is being tested, test cases will have to be written relating to the Chat feature. They can all be grouped under the label ‘Chat’ and’ can be viewed together.
- Search Tests: Like other issues in Zephyr, tests are easy to search for by using the ‘Search Tests’ option in the ‘Tests’ tab.
- Helpful tip: use Excel to perform initial rapid test case creation and modification; once the test plan is stable import these test cases into Zephyr for management, execution, and reporting.
Planning the Test Cycle
Zephyr allows the user to create multiple test cycles and view them under their respective versions. Tests can be added singly to an existing test cycle or cloned as a group from another test cycle (also, one test can be part of more than one test cycle). Once all the tests are added to your test cycle the execution phase is clearly organized and tracking of results is easy. During execution a progress tracking bar displays the tests executed and the ones remaining as a percentage. After execution is complete for the current test cycle, create the next test cycle (typically associated with the next version), and clone your desired tests into this test cycle and you're ready to go when the developers next release to your QA environment.
BONUS: Configuring JIRA for mobile
Since JIRA was built primarily to cater to the development needs of the web and desktop applications market, leveraging JIRA for managing mobile application development involves some level of customization.
For managing mobile applications in JIRA, these custom fields can be added:
Mobile applications are routinely designed to have the same user functionality for different Operating Systems (OS). But, based on the platform on which they are applied, their implementation can be different.
The solution is the creation of a custom field specifying the exact OS; different tickets can be raised for the different developer teams working on Android and iOS.
Developers need their own environment in which they can code and run unit and integrated testing. Since the development environment is dynamic and unstable, it has to be kept separate from the QA (or staging) environment. The production environment is kept exclusive from the development and testing phase so that the end-users and other software interfaces are not affected by the dynamics of development and testing.
A ticket should only be close in the environment in which it was created. This environment data is not included in JIRA by default. It must be specified as a custom field in the issue documentation as, say, 'Affected Build'.
Affected Device OS
Mobile manufacturers are continuously coming up with newer versions of their OS. For example: Kitkat is a version of the Android OS and it, in turn, has multiple versions. If a ticket has been raised on a device running Kitkat, the developers will need to reproduce it on that same OS.
They can do so most effectively if this is communicated consistently by adding a custom field on the JIRA issue screen to specify the affected device OS. Otherwise, they may end up trying to reproduce the issue on a different version where the same ticket may not be valid.
As developers release newer builds, testing of application components continues in real-time. It may happen that at one time, a QA engineer might have two versions of the same component for testing. Obviously, they will want to test the latest version. A custom field entitled 'Tested Version' should be created.
The fix version and the tested version of the component may be the same - '1.1', for example. But, if the fixed version is 1.1 and the tested version is, say, 1.2, having a custom field for the latter will provide more clarity and prevent confusion.
Executing the Tests
This option allows the user to execute any test case after accessing its particular test cycle; the user can even run the test immediately without including it in any test cycle (this is considered an ad hoc execution). The execution and tracking of tests becomes easier if they are grouped under test cycles. This may result in the user raising a new issue (e.g. a bug ticket), attaching the test to an already existing issue, attaching visual evidence of the bug and even inserting comments.
Reporting using Zephyr and JIRA
Test Summary and Metrics
Zephyr has a simple and basic reporting scheme as seen below. For a particular project, the number of total tests executed and the tests remaining can be seen in the test summary. The tests can also be seen based on the specific Version or any particular Label under which they are grouped.
The ‘Test Metrics’ option in the ‘Tests’ tab displays the current status of your project with the help of pie charts and graphs, listing the execution details by day, by test cycles or by the name of a particular tester so that their output can be monitored. The test metrics can be customized to reflect the test execution status of a particular version or for a specific time period.
Issue reporting with JIRA formatting features
JIRA makes the task of issue reporting easier for QA engineers by including some nifty formatting features. Let's take the example of an issue to show how these features help when documenting bugs.
The following screenshot shows the issue screen in JIRA. Formatting features are available in the “Description” field to help make it more organized and presentable:
While documenting the issue in the description field, the QA engineer must clearly convey the following information:
- The issue.
The table looks like it does because of the syntax behind it. For a header, use the h4. command at the beginning of the line, followed by the header text (such as the word ‘Environment’ here). Headers can be placed in the description by writing hn. at the beginning of the line followed by the header text (where n is a number from 1 to 6 in decreasing order of font size_.
To add tables to a description, use vertical bars (||) to enclose table headers and to enclose column text.
In the previous example, the syntax for the table is:
|Device A|iPhone 6|User A|
|Device B|Samsung S5|User B|
- How to reproduce the issue.
To fix the issue, the developers will need to reproduce it. You can use numbered lists to outline those steps and make it easier for them to do so.
To create numbered list, use the pound or hash symbol (#) in the first space of the syntax. And, if you need to insert any step into the list later on, JIRA helps you out by automatically re-adjusting the order of list numbers.
In the example, the syntax for the steps is:
# Given User A has uploaded video in his private library
# On Device A, send this video to User B as self-destruct
# On Device B, look at the photo preview inside message thread
If you want to create a bulleted list, you can use an asterisk (*) in the first space of the line. For further indentation, use two asterisks (**).
- What is the expected behavior?
As a QA engineer, you should describe what the application component is expected to do as part of the issue documentation. Whenever possible, provide a link to the relevant section in the specification documentation.
- What is the actual behavior?
To draw attention to an attribute, you can use syntax to highlight it in bold using an asterisk (*) or italics , use underscore (_) or even choose to display it in a different color.
Whenever possible, you should attach test evidence like an image or a video to show the actual behavior of the component. JIRA provides an image-capturing tool called JIRA Capture for web application testing. To use this tool, the URL address of the image is enclosed within ! in the syntax within the Description.
The URL address is further enclosed within the vertical bar (|) symbol so that the image is displayed within a box conforming to its dimensions. The syntax can be something like this :
What you see in the Description is:
As you have seen, the syntax of these features is easy to comprehend and use. The clearer your documentation, the better the developers can understand and resolve the issue.
Reporting metrics on the JIRA dashboard
The JIRA dashboard makes it very easy to view and analyze statistics for different fields such as projects, versions, and users, all filtered and expressed graphically with the help of pie charts, graphs, and other data displays. JIRA refers to these charts and data items as 'gadgets', which can be customized both by their placement on the page as well as their appearance and behavior. You can also set an auto-refresh time period for a gadget, so that there is no need to manually query or refresh in order to see the latest results on your efforts and team.
The gadgets available on the JIRA dashboard can help the QA lead prepare a variety of reports:
- Average Age Report: This report is expressed in the form of a bar chart showing the average duration (in days) for which issues remained unresolved at particular points in time. Looking at the average age report gives the QA team and the management a very good idea of the health of the project, and raise appropriate concerns if necessary.
- Pie Chart Report: This report displays the issues of a project filtered by a query in the form of a pie chart. It is useful as a high-level overview of the current build, helping to keep track of progress, assess workloads and even the performance of team members.
- Heat Maps: This is a dashboard gadget which is easy on the eyes. What the user sees are the values of an issue key grouped according to a query. They are grouped on the Dashboard according to the number of the issues under each Label. This results in a larger font size for a Label with a higher count, and progressively smaller font sizes for Labels with smaller counts. Along with visual graphs, heat maps are helpful in quickly determining that responsibility is fairly distributed among the team members as new sprints and projects are taken up.
- Created Vs. Resolved Report: With this report, you can view the number of issues created - and those which were resolved - during a specific time period. The health and progress of the project can be easily assessed using this report, and is ideal for giving feedback to management about the pace of work on both the QA and developer end.
Activity Stream Gadget: This dashboard gadget displays all the recent activity in all active projects by default. It can be customized to display information filtered by project, issue key or username to narrow the data displayed. The activity of more than one user also can be monitored by specifying their names in the “Apply Filters” field.
Whether it's cloud testing, automated testing, functional testing or regression testing, software quality assurance work takes significant time, effort and attention to detail. With some basic knowledge of metrics to watch for, though, there is every reason why any team should be successful at it.
Limitations of Zephyr
We've discussed many of the benefits of this add-on to JIRA, but Zephyr has some important limitations which must be taken into consideration when evaluating whether this is the proper test case management tool to adopt for your efforts:
Excel wins for efficiency
As mentioned previously, writing test cases is far more efficient in Excel. One workaround is to create in Excel and import into Zephyr for life cycle management. The test cases can be edited within the tool as updates occur and test execution status changes.
Issues with rapid editing
Rapid editing is inefficient - Each edit must be individually saved and the user attempting to use their keyboard must tab multiple times to get to the next/previous row. No real workaround is available for this; and once the test cases are in Zephyr, it doesn't make sense to export the data to excel, edit it there, and re-import since you'll lose all useful history.
There is no facility in Zephyr to group together and list all the test cases which are linked to the same issue. If you have a bug ticket related to a handful of tests, there’s not a simple method to view and execute each of these tests again; this would be a useful way for a team lead to assign work to QA members if it were available.
Historical reporting lacks some granularity
The status of a single test case execution across multiple application versions cannot be displayed in any sort of report or query. For example, if you have a regression test effort across multiple app versions, you cannot view the results of these individual tests (and entire test cycles) across all these versions. I as the QA lead cannot easily tell that minor test case TC-131 went from pass-pass-pass over the last 3 application versions, whereas critical test TC-223 had a history of pass-fail-fail for those same versions. Having this sort of historical reporting allows the QA lead deeper understanding of the test results and comprehensive risk analysis when reporting to management prior to a production release.
Visualization and new cycles
Although there is a way to see a specific list of tests in a previous app version ( for example, all failed tests in the previous version), there is no way to add them as a group to a test cycle for future execution. The QA engineer must manually note each test and individually add them to another cycle. And while writing test cases, images cannot be linked to individual test steps.
Field values change in the reporting when updated
Field values are tied to the actual test case ticket and not to the test ticket in relation to a given test run or test cycle, therefore any history reporting will display the latest value regardless of what that value was in the historical cycle. For example, if TC-133 was changed from a low priority in an earlier version to a higher priority in a later one, if I run a report on that earlier execution results I'll see the current priority of that test case and not the actual value of that field when it was executed. This can cause confusion and potential miscommunication.
So, is Zephyr worth it?
In a software testing service such as what OnPath Testing provides, using the Zephyr plug-in for JIRA can be a great way for QA, project managers, and developers to manage and monitor tests and execution status in real time. It is a useful tool, however the limitations as discussed above must be taken into account and understood in order for a QA team to get the most benefit out of their test effort, while supplying the necessary details when reporting to management.
Good luck, and if you have also used this tool please let us know your thoughts!