JIRA is popular among project managers, developers and QA experts for its project management capabilities and issue-tracking features. For QA professionals, JIRA provides solid bug tracking and requirements management features that are deeply customizable. These sit alongside great team collaboration features that allow everyone to benefit throughout the dev cycle.
What's the problem then? Well, one outstanding shortcoming of JIRA is the lack of decent test case management. Test case management such as test case creation, execution, and tracking are non-existent within this tool, yet an add-on called Zephyr makes a valiant attempt to bridge this gap. This add-on brings plenty of features to address the lack of test management in JIRA, as well as 'Zephyr reports' to feed back on this activity. With this said, Zephyr for JIRA it still has its shortcomings - as we shall see.
Zephyr was built for test case management. It leverages JIRA’s existing bug-tracking features to manage test cases and their execution. It facilitates work in real time between developers, QA, and project/product managers, thus allowing for close monitoring and updates of the build for overall improved software quality assurance. Although Zephyr has limited reporting ability within JIRA, test case planning and execution is easy and integrated with the JIRA bug tickets; thereby saving time by identifying the execution of previously failed test cases when a bug has been fixed.
Zephyr integrates seamlessly with JIRA, as seen in the above screen shot of the main project screen - after installation of the plugin a new ‘Tests’ menu option is included in the main navigation bar, and when expanded displays the list of test case management actions available. The options include creation, execution and management of tests, plus test summary and test metrics for monitoring the QA process. Additional options offer help on the use of this tool.
Test cases are written in the same way in JIRA as all other tickets and issues are created - using the ‘Create’ button on the top right hand side of the screen. Test cases, indicated by a new icon -
It would therefore be natural to ask - why would a QA team opt to work with this tool? It does offer some important and useful test case management features that a simple spreadsheet does not, such as:
Zephyr allows the user to create multiple test cycles and view them under their respective versions. Tests can be added singly to an existing test cycle or cloned as a group from another test cycle (also, one test can be part of more than one test cycle). Once all the tests are added to your test cycle the execution phase is clearly organized and tracking of results is easy. During execution a progress tracking bar displays the tests executed and the ones remaining as a percentage. After execution is complete for the current test cycle, create the next test cycle (typically associated with the next version), and clone your desired tests into this test cycle and you're ready to go when the developers next release to your QA environment.
Since JIRA was built primarily to cater to the development needs of the web and desktop applications market, leveraging JIRA for managing mobile application development involves some level of customization.
For managing mobile applications in JIRA, these custom fields can be added:
OS
Mobile applications are routinely designed to have the same user functionality for different Operating Systems (OS). But, based on the platform on which they are applied, their implementation can be different.
The solution is the creation of a custom field specifying the exact OS; different tickets can be raised for the different developer teams working on Android and iOS.
Affected Build
Developers need their own environment in which they can code and run unit and integrated testing. Since the development environment is dynamic and unstable, it has to be kept separate from the QA (or staging) environment. The production environment is kept exclusive from the development and testing phase so that the end-users and other software interfaces are not affected by the dynamics of development and testing.
A ticket should only be close in the environment in which it was created. This environment data is not included in JIRA by default. It must be specified as a custom field in the issue documentation as, say, 'Affected Build'.
Affected Device OS
Mobile manufacturers are continuously coming up with newer versions of their OS. For example: Kitkat is a version of the Android OS and it, in turn, has multiple versions. If a ticket has been raised on a device running Kitkat, the developers will need to reproduce it on that same OS.
They can do so most effectively if this is communicated consistently by adding a custom field on the JIRA issue screen to specify the affected device OS. Otherwise, they may end up trying to reproduce the issue on a different version where the same ticket may not be valid.
Tested Version
As developers release newer builds, testing of application components continues in real-time. It may happen that at one time, a QA engineer might have two versions of the same component for testing. Obviously, they will want to test the latest version. A custom field entitled 'Tested Version' should be created.
The fix version and the tested version of the component may be the same - '1.1', for example. But, if the fixed version is 1.1 and the tested version is, say, 1.2, having a custom field for the latter will provide more clarity and prevent confusion.
This option allows the user to execute any test case after accessing its particular test cycle; the user can even run the test immediately without including it in any test cycle (this is considered an ad hoc execution). The execution and tracking of tests becomes easier if they are grouped under test cycles. This may result in the user raising a new issue (e.g. a bug ticket), attaching the test to an already existing issue, attaching visual evidence of the bug and even inserting comments.
Zephyr has a simple and basic reporting scheme as seen below. For a particular project, the number of total tests executed and the tests remaining can be seen in the test summary. The tests can also be seen based on the specific Version or any particular Label under which they are grouped.
The ‘Test Metrics’ option in the ‘Tests’ tab displays the current status of your project with the help of pie charts and graphs, listing the execution details by day, by test cycles or by the name of a particular tester so that their output can be monitored. The test metrics can be customized to reflect the test execution status of a particular version or for a specific time period.
JIRA makes the task of issue reporting easier for QA engineers by including some nifty formatting features. Let's take the example of an issue to show how these features help when documenting bugs.
The following screenshot shows the issue screen in JIRA. Formatting features are available in the “Description” field to help make it more organized and presentable:
While documenting the issue in the description field, the QA engineer must clearly convey the following information:
The table looks like it does because of the syntax behind it. For a header, use the h4. command at the beginning of the line, followed by the header text (such as the word ‘Environment’ here). Headers can be placed in the description by writing hn. at the beginning of the line followed by the header text (where n is a number from 1 to 6 in decreasing order of font size_.
To add tables to a description, use vertical bars (||) to enclose table headers and to enclose column text.
In the previous example, the syntax for the table is:
|Device A|iPhone 6|User A|
|Device B|Samsung S5|User B|
To fix the issue, the developers will need to reproduce it. You can use numbered lists to outline those steps and make it easier for them to do so.
To create numbered list, use the pound or hash symbol (#) in the first space of the syntax. And, if you need to insert any step into the list later on, JIRA helps you out by automatically re-adjusting the order of list numbers.
In the example, the syntax for the steps is:
# Given User A has uploaded video in his private library
# On Device A, send this video to User B as self-destruct
# On Device B, look at the photo preview inside message thread
If you want to create a bulleted list, you can use an asterisk (*) in the first space of the line. For further indentation, use two asterisks (**).
As a QA engineer, you should describe what the application component is expected to do as part of the issue documentation. Whenever possible, provide a link to the relevant section in the specification documentation.
To draw attention to an attribute, you can use syntax to highlight it in bold using an asterisk (*) or italics , use underscore (_) or even choose to display it in a different color.
Whenever possible, you should attach test evidence like an image or a video to show the actual behavior of the component. JIRA provides an image-capturing tool called JIRA Capture for web application testing. To use this tool, the URL address of the image is enclosed within ! in the syntax within the Description.
The URL address is further enclosed within the vertical bar (|) symbol so that the image is displayed within a box conforming to its dimensions. The syntax can be something like this :
http://jira.onpathtesting.com:8080/secure/attachment/12747/Screenshot_2015-04-30-18-08-31.png|height=300|width=150!|
What you see in the Description is:
As you have seen, the syntax of these features is easy to comprehend and use. The clearer your documentation, the better the developers can understand and resolve the issue.
The JIRA dashboard makes it very easy to view and analyze statistics for different fields such as projects, versions, and users, all filtered and expressed graphically with the help of pie charts, graphs, and other data displays. JIRA refers to these charts and data items as 'gadgets', which can be customized both by their placement on the page as well as their appearance and behavior. You can also set an auto-refresh time period for a gadget, so that there is no need to manually query or refresh in order to see the latest results on your efforts and team.
The gadgets available on the JIRA dashboard can help the QA lead prepare a variety of reports:
Activity Stream Gadget: This dashboard gadget displays all the recent activity in all active projects by default. It can be customized to display information filtered by project, issue key or username to narrow the data displayed. The activity of more than one user also can be monitored by specifying their names in the “Apply Filters” field.
Whether it's cloud testing, automated testing, functional testing or regression testing, software quality assurance work takes significant time, effort and attention to detail. With some basic knowledge of metrics to watch for, though, there is every reason why any team should be successful at it.
We've discussed many of the benefits of this add-on to JIRA, but Zephyr has some important limitations which must be taken into consideration when evaluating whether this is the proper test case management tool to adopt for your efforts:
As mentioned previously, writing test cases is far more efficient in Excel. One workaround is to create in Excel and import into Zephyr for life cycle management. The test cases can be edited within the tool as updates occur and test execution status changes.
Rapid editing is inefficient - Each edit must be individually saved and the user attempting to use their keyboard must tab multiple times to get to the next/previous row. No real workaround is available for this; and once the test cases are in Zephyr, it doesn't make sense to export the data to excel, edit it there, and re-import since you'll lose all useful history.
There is no facility in Zephyr to group together and list all the test cases which are linked to the same issue. If you have a bug ticket related to a handful of tests, there’s not a simple method to view and execute each of these tests again; this would be a useful way for a team lead to assign work to QA members if it were available.
The status of a single test case execution across multiple application versions cannot be displayed in any sort of report or query. For example, if you have a regression test effort across multiple app versions, you cannot view the results of these individual tests (and entire test cycles) across all these versions. I as the QA lead cannot easily tell that minor test case TC-131 went from pass-pass-pass over the last 3 application versions, whereas critical test TC-223 had a history of pass-fail-fail for those same versions. Having this sort of historical reporting allows the QA lead deeper understanding of the test results and comprehensive risk analysis when reporting to management prior to a production release.
Although there is a way to see a specific list of tests in a previous app version ( for example, all failed tests in the previous version), there is no way to add them as a group to a test cycle for future execution. The QA engineer must manually note each test and individually add them to another cycle. And while writing test cases, images cannot be linked to individual test steps.
Field values are tied to the actual test case ticket and not to the test ticket in relation to a given test run or test cycle, therefore any history reporting will display the latest value regardless of what that value was in the historical cycle. For example, if TC-133 was changed from a low priority in an earlier version to a higher priority in a later one, if I run a report on that earlier execution results I'll see the current priority of that test case and not the actual value of that field when it was executed. This can cause confusion and potential miscommunication.
In a software testing service such as what OnPath Testing provides, using the Zephyr plug-in for JIRA can be a great way for QA, project managers, and developers to manage and monitor tests and execution status in real time. It is a useful tool, however the limitations as discussed above must be taken into account and understood in order for a QA team to get the most benefit out of their test effort, while supplying the necessary details when reporting to management.
Good luck, and if you have also used this tool please let us know your thoughts!