Blog / DevOps & Agile

Automated tests with Jenkins

Xray Solution Architect

5 min 31 comments

Xray for Jira automated tests with Jenkins

Following the segment initiated with Bamboo, in this article we will show you how you can automate your tests using Jenkins and our Xray for Jira.

In Xray, Automated Tests are executed by an external tool, which controls the test executions and the comparison of actual outcomes and the predicted ones. They can automate some repetitive tasks in a formalized testing process already in place, or add additional testing that would be difficult to perform manually.

As the testing tool, we will use Cucumber because it allows you to execute plain-text functional descriptions as automated tests. You can find more information about how automated tests are created in Xray, as well as more information about Cucumber, in Xray documentation and Cucumber’s website, respectively.

It is also important to point out that this is just possible since Xray makes also available it’s Rest API and, with that, we will be able to export Test definitions and import execution results.

Xray for JIRA add-on for Jenkins aims to simplify the process of continuous integration. With the setup of some Xray specific tasks, users can now easily export Cucumber features, report automated testing results for many different automated testing frameworks and associate them with new or existing Tests, Test Executions, Test Plans, Test Environments, and more.

Examples

Let’s get to know the add-on by running two examples. For each example we will create two plans, both taking into account the specific testing framework and with the common final goal of importing the execution results to a specific Xray endpoint.

We will assume that you already know how to pull your test project from your repository and how to run the tests in the Continuous Integration environment. As such we will only focus on how to configure the Xray specific tasks.

JUnit

Apart from supporting Cucumber natively, Xray enables you to take advantage of many other testing frameworks like JUnit. In this sense, Xray for Jenkins lets you import results in other formats besides Cucumber JSON.

  1. Pulling the JUnit project;
  2. Executing the tests in the CI environment;
  3. Importing the execution results, including Tests, to Jira.

Importing the execution results to Jira

To start the configuration, add the post-build action Xray: Results Import Task.

JUnitTask

After that, configure it.

In this example, we have a configuration where the JUnit XML format is chosen.

JUnitImportTaskConfig

After running the plan, the expected result is a new Test Execution issue created in the Jira instance.

ExpectedResultJUnit

Cucumber

In a typical Cucumber Workflow, after having created a Cucumber project and the Cucumber tests specified in Jira, you may want to have a project that exports the features from Jira, executes the automated tests on a CI environment and then imports back its results.

  1. Pulling the Cucumber project;
  2. Exporting Cucumber features from Jira to your Cucumber project;
  3. Executing the tests in the CI environment;
  4. Importing the execution results back to Jira.

Exporting Cucumber features

To start the configuration, add the build step Xray: Cucumber Features Export Task.

ExportTask

After that, configure it.

In this example, we configured the task to extract the features from a set of issues (PROJ-78 and PROJ-79) to the folder that holds the Cucumber project.

ExportTaskConfig

Importing the execution results

To start the configuration, add the post-build action Xray: Results Import Task.

JUnitTask

After that, configure it.

In this example, we configured the task to import the Cucumber JSON results back to Jira.

CucumberImportTaskConfig

Once all configurations are done, click Save at the bottom of the page.

After running the job, the expected result is a new Test Execution issue created in the Jira instance.

ExpectedResultCucumber

Importing the execution results with user-defined field values

For Cucumber, Behave, JUnit, Nunit and Robot, Xray for Jenkins allows you to create new Test Executions and have control over newly-created Test Execution fields. You can send two files, the normal execution result file and a JSON file similar to the one Jira uses to create new issues. More details regarding how Jira creates new issues here.

For this scenario and example, the import task needs to be configured with the Cucumber JSON Multipart format. When selecting this option, you can additionally configure the Test Execution fields in one of two ways:

  • Insert the relative path to the JSON file containing the information, or
  • Insert the JSON content directly in the field.

In this example, we configured the following object:
{
"fields": {
"project": {
"key": "PROJ"
},
"summary": "Test Execution for Cucumber results (Generated by job: ${BUILD_TAG})",
"issuetype": {
"id": "10102"
}
}
}

And configured the task to import the Cucumber JSON Multipart results back to Jira.

CucumberMPartImportTaskConfig

Once all configurations are done, click Save at the bottom of the page.

After running the job, the expected result is a new Test Execution issue created in the Jira instance, with the Test Execution fields as specified in the Jenkins build step configuration.

ExpectedResultCucumberMPart

In sum…

Importing automated execution results and exporting Cucumber features is now easier with Xray for Jira add-on for Jenkins. With some simple parametrization, you can take advantage of the Xray’s REST API and every available endpoint for importing results, to “Continuously Integrate” and thus continuously improve your product quality, while having on-time feedback that will help you promptly fix any issue.

Good luck in “Continuous Integrating” your Tests!

Editor’s Note: This post was originally published in August 2016 and has been completely revamped and updated for accuracy and comprehensiveness in 2018.

FacebookTwitterLinkedIn

Recommended for you

Xray-logo-white

News 1 min

test-1-a1

6 most popular test automation frameworks

DevOps & Agile 7 min

A test automation framework helps your team scale automation with reusable code and aligns all team members under a set of common guidelines. Learn the 6 most popular frameworks and the pro’s and con’s of each.

31 comments

  • Hi,

    I have integrated xray with jenkins. That works fine. But xray create always new test executions. We have a lot test executions result. Can xray update a test executions instead of create new test execution?

    Best regards

    Tien Hanh Pham

    Reply

  • Hi,

    How to import multiple JUnit test execution results using XRay Jenkins plugin? (JUnit result file names are generated randomly)

    Thanks,
    Valery

    Reply

  • Hi everybody,

    as a newbie I would like to now where I can find the Jenkins plugin for XRAY?

    I now that the plugin is named like “xray-for-jira-connector.hpi”.

    Thank you for any answeres

    Best regards,
    Frank

    Reply

  • Hi ,

    What ID value should be passed in JSON for execution type ?how can I get this ID?
    I am configuring the JUNIT results in Jenkins with XRAY using JUNIT xml multipart import format .

    I want to Xray to create the execution issue each tyime when the Jenkins completes the build with name given by me in JSON

    ERROR: Step ‘Xray: Results Import Task’ failed: Unable to confirm Result of the upload….. Upload Failed! Status:400 Response:issuetype: issue type is required

    Thanks in advance

    Reply

    • Hi Mangala,
      Thank you for your comment!

      To get the ID’s place the mouse point over the Edit option of the Issue Type and you may find the ID in the URL.
      You are getting the error because the issue type must be send in the Json data file.

      If you still need any assistance please open a ticket with this question here?
      Our technical team will be glad to help you.

      Best regards,
      Xray Team

      Reply

  • Hi,
    I integrated Xray with Jenkins.
    Type of my test results is JUnit XML.
    I can send to Xray JUnit XML reports.
    But I don’t see anything about JUnit XML output at Xray.

    Thank you for any answeres
    Best regards.

    Reply

  • Hello,
    We integrated Xray with Jenkins.
    I would like to know how to adding the “Test Environment” to the test result imported to Jira?
    Thank you very much.
    Best Regards,

    Reply

  • Great plugin. Been using it import existing feature files from SCM into XRay.
    Scenario level tags convert to labels in the issue but feature level tags are not imported. Is the intentional? Am I missing something?

    Reply

    • Hi Steven,
      Thank you for your kind words. 😀
      Concerning importing of “features”, what Xray provides is an endpoint for importing Cucumber based “Scenario” / “Scenario Outline” and related “Background” as Tests and Pre-Conditions, respectively.
      The main purpose is to import tests and not the “Feature(s)” (i.e. requirements) themselves; it is assumed that those “Feature(s)” already exist and are already managed in Jira side, as user stories/epics or a similar issue type.
      You may see how the import process works in more detail here https://confluence.xpand-it.com/display/XRAY/Importing+Cucumber+Tests+-+REST
      The overall workflow assumes that someone wrote the user stories/features in Jira and that later on you will write tests (i.e. Cucumber Scenarios) in order to validate them.

      Best regards,
      Xray Team

      Reply

  • We are getting an error message configuring the Jenkins plugin, after adding the configuration and clicking on ‘Test Connection’ we get { Could not establish connection }

    Reply

  • When I use Jenkins to run an automated test case in Katalon which also exists in Jira with the same Xray key, Xray creates a new test.

    Reply

  • Hello,
    I want to write Automation test cases and Manual test cases in the same XRAY TC task. but once I select Manual TC to Automation , Written manual test cases deleted and vice a versa.

    Please assist me to write Automation and MAnual test case together.

    Thanks in Advance!!

    Reply

  • Hi
    I do an integration process with the XRAY tool for the CI process automatically
    Is there a way to link the steps in each test?
    That means XRAY will have the business steps and connect each step to the code lines in the automated script
    And at the end of the process an import to xray will be made and there every step is displayed whether passed or not

    Thanks

    Reply

    • Hi Nizan,

      Please reach out to our support team on Service Desk .

      They’ll ask you some additional questions to best answer your request. 🙂

      Best regards,
      Team Xray

      Reply

  • Hello to all,

    Import test results with multipart option is great but how to import tests from your testing framework in Xray (update existing one) by updating no just steps but labels and custom fields too?

    Is there any mechanism similar to multipart from Import Test results to update existing tests with e.g (environment name, tenant name-custom field, or other custom fields)

    Thanks in advance for the answer

    Reply

    • Hi Stanislav,

      You can achieve this by importing CSV issue files, either through our Test Case Importer or via Jira default CSV import.

      Please notice that they are similar but have different features and, due to that, we strongly advise that you review our documentation for them.

      You can find information on the Default CSV importer and our Test Case Importer on the link Importing Test Cases

      Also, notice that for Cucumber Tests you may need a different approach; for that, please check the documentation Importing Cucumber Tests – REST

      Again, we strongly advise you review all the documentation as you may need to use different approaches depending on which Test type you are using or which data you want to update.

      I hope this brings some clarification on the topic but please let us know if there is anything else we can help you with.

      Best regards,
      Team Xray

      Reply

  • Hi all,

    Is there a way to trigger a Jenkins job from Test Plan view (in Jira)? This trigger will pass arguments such as OS, Test cases to execute etc.

    Thanks in advance!

    Reply

  • Thanks for the reply, I’m interested in this case: Trigger a Jenkins project build from a Test Plan, for the Tests contained in the Test Plan. This part is not fully described how Jenkins can fetch the Test Cases/Test Sets that are listed in Plan. Additional question, what is the best practise for storing the source code of those tests in order to assure the single point of truth (between Jira and Bitbucket)?

    Thanks!

    Reply

  • Hi Team,
    We have recently started using X-Ray and the automation using Selenium + Java.
    What I want to achieve is —
    1. I have automation tests created in X-Ray as “General”.
    2. I have automation scripts created in Java for the above test cases.
    3. I have X-Ray test execution created for few of the above test cases.

    I want to integrate this with Jenkins so that I should be able to execute the test cases included in the X-Ray test execution. Do we have any plugin through which I can achieve this? So once my Jenkins job executed it should execute the test cases included in it? In turn it should search for a test case in my Java suite (may be some ID is provided in Java code to identify it) and then execute it. Sends back the result to X-Ray test execution and it updated the execution with Pass/Fail etc.

    Reply

  • Hi,
    When running Xray: Results import task, how is it possible to associate new Test Execution to existing Test Plan?

    Thanks

    Reply

Leave a reply

Categories

Subscribe our Newsletter

Receive the best articles, latest news, premium content and events information.

Subscribe

Popular Posts

Tags

Automation BDD CI/CD Continuous testing Coverage Analysis Dashboards & Gadgets Data-driven testing DevOps DevSecOps jenkins Jira-tuning Management Performance metrics performance testing Quality assurance Risk-based testing SDLC Shift-left testing Testing in Jira Traceability