I would like to thank Arnaud Heritier who gave me plenty of tips and advices to write this blog. In fact if you want to endorse someone on Maven in LinkedIn, he is the man, not me ;o)
If you use Maven in your projects you might wonder what “Let’s Turn Integration Tests with Maven into a First-Class Citizen” means. A bit of introduction on that. Maven is based around the central concept of a build lifecycle and if you look at the default lifecycle bindings you see the following phases :
As you can see, unit tests are central in Maven. Package your project with mvn package or deploy it with mvn deploy and this will kick the test phase. Why ? Because in the default lifecycle bindings of Maven, the test phase is automatically called through the Surefire plugin. The maven-surefire-plugin is designed for running unit tests and if any of the tests fail then it will fail the build immediately.
So where are the integration tests ? Not in the default lifecycle, that’s for sure. They are handled by a different plugin : Failsafe. The maven-failsafe-plugin is a fork of Surefire designed to run integration tests (after the package phase, on the integration-test phase). But on the contrary of Surefire, Failsafe is not activate by default, you need to explicitelly add it to you pom.xml and attach a phase to it :
[sourcecode language=”xml” highlight=”8,9″]
The Maven lifecycle has four phases for running integration tests:
- pre-integration-test: on this phase you can start any required service or do any action (starting a database, a webserver…)
- integration-test: failsafe will run the test on this phase
- post-integration-test: time to shutdown all services
- verify: failsafe runs another goal that interprets the results of tests here
The Failsafe Plugin has only 2 goals:
- failsafe:integration-test : runs the integration tests of an application.
- failsafe:verify : verifies that the integration tests of an application passed.
By default, the Surefire plugin executes **/Test*.java, **/*Test.java, and **/*TestCase.java test classes. The Failsafe plugin will look for **/IT*.java, **/*IT.java, and **/*ITCase.java. So if you are using unit test and integration tests, make sure to put them all in src/test/java and use this naming convention.
How does this work ? You run mvn test : unit test are executed. You run mvn verify : unit test and then integration tests are executed. But if unit test fail, integration tests are not passed. Meaning that, for Maven, unit tests are more important than integration test.
How to seperate unit and integration test ?
What I really want is to be able to, either run UT or IT or none of them. Something like :
- mvn install -DskipUTs : Skips Unit tests
- mvn install -DskipITs : Skips Integration tests
- mvn install -DskipTests : Skips both Unit and Integration Tests
Here is a little trick you need to do in your pom.xml (again, thanks Arnaud)
[sourcecode language=”xml” highlight=”8,9″]
Unit testing is at the heart of Maven… but not integration testing. With this little trick you can put UT and IT at the same level : you can run either both tests, UT only, IT only or none. But it would be good to have this behaviour by default, without any configuration. That would mean adding the integration-test phase to the default lifecycle bindings of Maven.
What do you think ? Shall I start a discussion on the Maven Mailing List ? Create a JIRA issue ? Shall we put integration tests at the same level of unit test ? Any other idea of how we could integrate both more closely ?
10 thoughts on “Let’s Turn Integration Tests with Maven to a First-Class Citizen”
Great article – Thanks! I plan to implement it today
hmm, wondering why maven-failsafe-plugin was to have both skipTests and skipITs set? Isn’t it enough just to set
Thanks Antonio for pointing out this Maven feature.
We should also notice that it is IDE independant and can support other great helpers such as Cargo or DBUnit.
You can see an implementation here : https://github.com/turiot/meccano, a small Tomcat oriented framework to split web and services layers into independant wars with no classloading troubles.
Not sure I see the point in running integration tests if unit tests have failed… Unit tests should be very fast to run, and if any of them fail there is probably a bug you should fix before running the integration tests even makes sense.
“Unit tests should be very fast to run, and if any of them fail there is probably a bug you should fix before running the integration tests even makes sense”
Yes I agree 100%.
“Not sure I see the point in running integration tests if unit tests have failed”
I don’t think the point is to run integration tests if unit tests have failed. The point is to be able to run unit tests and integration tests independently of each other.
Once use case here would be a build pipeline: the first stage compiles and unit tests, and if that succeeds, passes the results to the next stage. The second stage would run the integration tests, it does not need to run the unit tests because we know they already succeeded in the previous stage.
Another use case would be if a developer is writing integration tests, there’s only need to run the new integration tests, not to run the unit tests.
“Once use case here would be a build pipeline: the first stage compiles and unit tests, and if that succeeds, passes the results to the next stage. The second stage would run the integration tests, it does not need to run the unit tests because we know they already succeeded in the previous stage.”
Yes that is exactly how the maven build pipeline works already.
If unit- and integration test source files lie alongside each other in src/test/java in the same Maven artifact, the builtin eclipse JUnit-Integration cannot distinguish between them and will execute all @Test-annotated classes it finds when you run do a “Run as …/JUnit-Test” on the topmost package node. The integration tests might be failing in your IDE and you might not be able to run the entire suite as a unit anymore. Thus, collateral damage might be concealed. Also, code coverage is reported wrong because integration tests contribute to it while your build and code quality environment might not execute integration tests.
By convention in maven, unit tests are named xxxTest and integration tests are named xxxIT.
When the test phase executes the maven-surefire plugin will run all tests in all classes named xxxTest.
When the integration-test phase executes the maven-filesafe plugin will run all xxxIT tests