A Test Set is a way to logically group Tests. Basically, it is a list of Tests and is the basis for Test organisation in Xray. Have you discovered its full potential?
1. Requirements coverage
You can cover your requirements with Tests by linking them using the “tested by” issue link. You can do the same using Test Sets. It enables a dynamic coverage of requirements, meaning, if you have a requirement covered by a Test Set, that requirement is covered by all the Tests belonging to that Test Set.
This also means that you can add Tests to your Test Set and they will automatically appear as covering the related requirement. This is particularly useful if you have similar requirements that you need to validate using the same Tests.
Another use case would be companies providing testing services that use Test Sets to identify which test cases to build. In this scenario, they start with an empty Test Set linked to the requirement. Once that is done, they will actually build and add the Tests to the Test Set.
2. Ranking Tests within a Test Set
As mentioned earlier, a Test Set is a list of Tests. To be a bit more precise, it is an ordered list of Tests. This order can be applied whenever you “Add Test Sets” to an existing Test Execution. Whenever you do it, you’re in fact adding the Tests to the Test Execution, taking into account their ranking in the Test Set.
This is useful if you want to clearly define the “priority” of the Tests within the Test Set. When you schedule Test Executions based on this, that “priority” will be taken into account in the ranking of the Tests.
Let’s say you have a “smoke testing” Test Set, which contains the 3 Tests below:
If you use the “Add Test Sets” option in the issue screen of a new or existing Test Execution, then the Tests will be added to the Test Execution in the same order/ranking they have in the original Test Set.
3. “Test Set Status”
A Test Set does not directly reflect execution-related aspects, but you may want to quickly check the status of the Tests that are grouped within the same Test Set.
Let’s say you have a Test Set containing crucial Tests related to security checking:
By adding this Test Set Status custom field to the screen, you can see the status of the related Tests in a specific version. This status (i.e., the version for which it shows the results for) can be defined in Xray’s Custom Fields Preferences.
It is also particularly informative if included in a listing of some relevant Tests Sets using the Filter Results tool:
4. Export to PDF or Word
If you need to export the Tests of your Test Set to a fully customizable Word or PDF document, Xporter for Jira provides the free template “Xray Test Set” that you can install and customize to your needs.
5. “Tests Association with a Test Set“
This custom field provides a list of the Tests associated with a Test Set. You can use it, for example, in the Issues search screen to quickly see the associated Tests keys.
It can also be used to create a Test Set with the Tests you want when you import data from CSV.
We believe that Test Sets are a strong feature that can improve the organization in your testing process. Ready to put these tips into practice?
hi, i’m trying to use the test sets
but i want to see them under my test plan
so i’ll see what i haven’t wrote about yet
what i see now under the test plan is the entire tests I’ve written
how can i see it more organized?
Hi Roni,
For you to organize your tests on the Test Plan, the best suggestion is to use the Test Plan Board.
If you need further support, feel free to contact us through our Support Portal. https://jira.xpand-it.com/servicedesk/customer/portal/2
Best Regards,
Xray Team
Hello,
I have the same test case in three different test sets.
When I put the three test sets in the same test execution, it’s not possible to execute the same test case three times as in the test sets.
Is there a tip for this situation that I can?
Thank you in advance,
Zaineb.
Hi Zaineb,
What we suggest is that you add one test and execute them in different Test Environments, where the tests will be in the respective attribute. A Test Environment is similar to a label, but Xray has special logic to deal with it. If you use Test Environments, you may reuse the same Test for multiple test environments and create Test Execution for each one.
Take a look at the documentation here.
For any further questions please contact our technical support to help you out more specifically.
Best Regards,
Xray Team
Hello,
I have an application with 14 modules. Each module has the following features: sections, actions, search, and report. Ideally, each feature must be covered by a set of tests. A difficulty appears here because the same section (a single functionality, a component) can be used in several modules. The idea is to define tests sets for each module and several tests cases for each section. These latter tests cases can be associated to several tests sets more than once? Thank you very much. JAD
Hi JAD,
Yes, every Test case can be associated with more than one Test Set.
If the question for “more than once?” is for having repeated Test cases on a Test Set, then the answer is no.
For any further questions please contact our technical support to help you out more specifically.
Best Regards,
Xray Team
how could i use a column in the test run and the test run report? More specifically I want to add 2 new columns (planned start date and due date) for every test case in a test environment. In addition, right now when I add the actual start date column to the test run, it is blank! Why? When I run the Test Run report it is fine.
Hi Ted,
Currently, there is no way to add columns to the reports. Regarding your specific question, reach out to our wonderful support team who will be happy to help you with configurations. 🙂
https://jira.xpand-it.com/servicedesk/customer/portal/2
Best regards,
Xray Team
How can I import automated Cucumber test results from Jenkins into a test set? Right now we have the integration working but only showing execution results in the individual tests. I want the results to be automatically associated with a test set. Any way to do this?
Hi Sara,
You are able to import the execution results of your tests from an automated Jenkins job, using our plugin for Jenkins as described here: https://confluence.xpand-it.com/display/public/XRAY/Integration+with+Jenkins#IntegrationwithJenkins-Importingtheexecutionresults
However, you’re not able to import the Execution results directly to a Test Set. The purpose of the Test Set issue type is only to organize the Tests, like a folder. Maybe, you want to use Test Plans instead: https://confluence.xpand-it.com/display/public/XRAY/Test+Plan 🙂
Best regards,
Team Xray
Hello,
I’m having a problem with creating an automation rule that would change the priority of a single xTest belonging to a given Test set once the priority of this Test set is changed accordingly. I tried using smart values but I’m not sure they’re supported by xRay. Is there a smart value for “Tests Association with a Test Set“?
Thank you in advance!
Kirill
Hi Kirill,
Please contact our support on Service Desk. They will be able to help you address your question. 🙂
Best regards,
Team Xray
We create few test sets according to the module we have. However, we are wondering should we reuse the test (test case) if we re-test the same module in different Sprint?
Hi BY,
Test cases are reusable entities; they’re templates in essence.
A test set is just a group of test case templates.
Best regards,
Team Xray
Is there a way that tests can be sent to a Reviewer and he can Approve those – Approval workflow
Hi Mandar,
Yes, it’s possible to implement approval workflows.
Some usage examples are detailed in this article: https://docs.getxray.app/display/XRAY/Using+Jira+workflows+for+testing+purposes
Best regards,
Team Xray
Hi Team,
We have recently started using the xray. We have created the test set and linked it to a Story. So we have a mapping between a story and Test set.
Is there any way if we can get the linking between the test cases and story as well without linking the test cases manually?
Hi Abhijit,
If you have linked the Test Set to the user story using the issue link type “tests”, then implicitly the Tests of that Test Set will cover that user story. You don’t need to have explicit links between these Tests and the user story, as Xray can automatically calculate the coverage based on the linked Test Set.
Note: Whenever you link a Test Set to a user story, no explicit issue link is created between the respective Tests and that user story.
Hope this helps!
Best regards,
Team Xray
Thanks for the above tips.
I am not sure how to reuse the same few hundred tests on different stories.
In my scenario each story represents the conversion of a given file, which will then be validated by my tests:
New story -> new file -> same tests on the new file.
As soon as one test passes for one story, its status will affect the coverage/ requirement status of all other stories. What I need instead is a user story specific requirement status even though all stories share the same test.
Is the parameterized test (mentioned in your roadmap) a solution ?
When will it be available ?
Thanks for your advice
Uwe
Hi Uwe,
We don’t recommend linking one Test to multiple requirements; usually that may be a signal that we need to improve something.
What you’re trying to achieve can be seen as data-driven testing, which will be released soon.
In that case you would have one Test executed multiple times, against multiple input rows. However, it would still be linked to a given Story.
Feel free to reach our support so we can discuss this in more detail with you.
Best regards,
Team Xray
Hi,
I am not seeing the JIRA field option to link the test set to the test during upload.
‘Tests association with a Test set’ is not visible. Can anyone help? Thanks
Hi Gareth,
Please reach out to our support team who’ll be happy to help you resolve this issue.
Service Desk
Best regards,
Team Xray