Nearly two years ago (time flies), when Java EE 6 came out, I wrote a post about application servers where I did some micro benchmarking (basically, startup time). I had plenty of comments and recently I had many people asking for some updates. Witht Java EE 6 booming, with some cloud vendors moving to Java EE 6, it was time to update this microbenchmark and focus on Java EE 6 application servers. BTW, if you want to know what Java EE 6 is, you can check the slides of a presentationI gave a few times.
The benchmark
Same disclaimer as last time : This is not a real benchmark ! so I’ll copy paste the paragraph I wrote last time :
In this test I’m just concerned about the usability of an application server for a developer. The idea is to download it, install it, start it and take some measurements : size of download, ease of installation, size of disk, startup time, size of RAM... That’s all. No deployment of an application, no fancy twists to gain performance… Because some of these application servers are resource consuming, I’m doing all my tests on a Windows XP SP3 virtual machine (running on Virtual Box 4.1). It is a fresh install of Windows XP sp3 with 1Gb of RAM and not too many software installed. So when I boot there are 27 processes running and using 230 Mb of RAM (leaving 770 Mb free). Virtualizing can be slower, so keep in mind that startup time can be a bit faster that what I’m giving you here. I use JDK 1.6.0_27 (when it’s not bundled with the server). No optimization at all is made (I haven’t twisted the JVM, or antyhing application server parameter, everything comes out of the box).
To calculate the startup time, I don’t do any fancy rocket science either. I just start the server a few times, check how long it takes and use the best startup time. Also remember that some servers do not load any container at startup, making them very fast to start. That’s why I first trigger the web admin console (when there’s one) so I’m sure at least one web application is deployed (I did a print screen of the admin console when there’s one). Then I just stop and start the server a few times and get the best startup time (by reading the logs). To calculate the memory footprint, I just use the Windows task manager and check the size of the java.exe process.
Which application servers ?
If you look at the Java EE Compatibility page at Oracle, you’ll see that there are a bunch of application servers that actually have passed the Java E E6 TCK (Oracle GlassFish Server 3.1.1, Caucho Resin 4.0.17, JBoss Application Server 7, Apache TomEE 1.0.0-beta-1, TMAX JEUS 7, IBM WebSphere Application Server 8.0, Fujitsu Interstage Application Server) and some still haven’t (Geronimo 3, JOnAS 5.3, Siwpas). On this list some application servers implement the full Java EE 6 and others just implement the Web Profile 1.0.
GlassFish 3.1.1
GlassFish 3 is the open source reference implementation for Java EE 6. The 3.1 release has added clustering capabilities. Being the reference implementation, GlassFish 3.x implements both Java EE 6 and the Web Profile 1.0, that’s why you have two different bundles you can download (one of 80.6 Mb and the other 45.8 Mb).
| Version | 3.1.1 |
| What do you get ? | Java EE 6 & Web Profile 1.0 |
| Who is behind? | Oracle |
| Open source? | Dual license CDDL and GPL |
| Documentation | Documentation |
| Download | Zipfile, executables |
| Size of the downloaded file | 80.6 Mb (Java EE 6) 45.8 Mb (Web Profile) |
| Installation mode | Unzip or exe |
| Size once installed on drive | 96.1 Mb (Java EE 6) 57.8 Mb (Web Profile) |
| How to start it | Execute %GLASSFISH_HOME%\bin\asadmin start-domain |
| Location of log files | %GLASSFISH_HOME%\domains\domain1\logs |
| Best startup time | 2 953ms (Java EE 6) 1 914ms (Web Profile) |
| Size of the process in RAM at startup | 83 MB (Java EE 6) 69 MB (Web Profile) |
| Welcome page | http://localhost:8080/ |
| Admin console | http://localhost:4848 |
Resin 4.0.23
Resin 4 was one of the first web server (such as Jetty or Tomcat) to move to the Web Profile 1.0. For me it’s the success story that confirmed that profiles in Java EE were needed. It’s really difficult to go from a Web Server to a full Java EE 6 application server (remember that there’s still Entity CMPs in EE 6) but much easier to implement the Web Profile. Resin is very CDI centric and based on the Caucho implementation called CanDI.
| Version | 4.0.23 |
| What do you get ? | Web Profile 1.0 |
| Who is behind? | Caucho |
| Open source? | GPL License |
| Documentation | Documentation |
| Download | Zip, tgz, deb |
| Size of the downloaded file | 24.2 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 32 Mb |
| How to start it | Execute %RESIN_HOME%\java -jar lib/resin.jar start |
| Location of log files | %RESIN_HOME%\log |
| Best startup time | 1 845ms |
| Size of the process in RAM at startup | 2 processes : 68.5 MB + 39 MB |
| Welcome page | http://localhost:8080/ |
| Admin console | http://localhost:8080/resin-admin/ |
JBoss 7.0.2-Final
It looks like JBoss 7 is, finally, a nice millésime. For those of you who waited so long for JBoss 5 to get certified and then didn’t see any JBoss 6, JBoss 7 is what you need with a fantastic startup time compared to the older versions (in my previous benchmardJBoss 5 was the slowest app servers, slower than WebSphere 7). Another novelty is an admin console (sexier than the good old JMX Console).
| Version | 7.0.2-Final |
| What do you get ? | Web Profile 1.0 |
| Who is behind? | JBoss/RedHat |
| Open source? | LGPL |
| Documentation | Documentation |
| Download | Zipfile |
| Size of the downloaded file | 70.7 Mb |
| Installation mode | Unzip |
| Size once installed on drive | 77.9 Mb |
| How to start it | Execute %JBOSS_HOME%\bin\standalone |
| Location of log files | %JBOSS_HOME%\standalone\log |
| Best startup time | 2 782ms |
| Size of the process in RAM at startup | 70.7 MB |
| Welcome page | http://localhost:8080/ |
| JMX console | http://localhost:9990/console |
TomEE 1.0.0-beta-1
Like Resin, Apache TomEEis the perfect success story for the Web Profile. TomEE is no more than Tomcat + OpenWebBeans + OpenEJB + OpenJPA + MyFaces + other bits. It really shows that Java EE is a jigsaw puzzle where you can take open standards, bundle them together and become a certified Web Profile application server.
| Version | 1.0.0-beta-1 |
| What do you get ? | Web Profile 1.0 |
| Who is behind? | Apache |
| Open source? | Apache Software License |
| Documentation | Documentation |
| Download | Zip, tar.gz |
| Size of the downloaded file | 24.2 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 29.2 Mb |
| How to start it | Execute %TOMEE_HOME%\bin\startup.bat |
| Location of log files | %TOMEE_HOME%\logs |
| Best startup time | 3 288ms |
| Size of the process in RAM at startup | 65 MB |
| Welcome page | http://localhost:8080/ |
| Admin console (*) | http://localhost:8080/manager/html |
(*) Looks like to be able to log on to the admin console, you need to change the %TOMCAT_HOME%\conf\tomcat-users.xml configuration files and add a manager-script role to a user. But I couldn’t make it work.
Geronimo v3.0-M1
Geronimo is not already Java EE 6 ceritified (still in Miletone 1). Geronimo comes in 2 flavours, one with Jetty the other one with Tomcat. And like TomEE, Geronimo bundles several open source implementationsof Java EE.
| Version | 3.0-M1 |
| What do you get ? | Not yet certified |
| Who is behind? | Apache |
| Open source? | Apache Software License |
| Documentation | Documentation |
| Download | Zip, tar |
| Size of the downloaded file | 66.9 Mb (Geronimo + Tomcat) |
| Installation mode | Unzip the file |
| Size once installed on drive | 76.4 Mb |
| How to start it | %GERONIMO_HOME%\bin\geronimo start |
| Location of log files | %GERONIMO_HOME%\var\log |
| Best startup time | 12 467ms |
| Size of the process in RAM at startup | 159 Mb |
| Welcome page | http://localhost:8080/ |
| Admin console (*) | http://localhost:8080/console-base |
(*) user/pwd : system/manager
JOnAS 5.3.0-M4
JOnASis till not Java EE 6 certified. It provides a full Java EE 5 execution stack plus some Java EE 6 previews (JPA 2.0, CDI, …), supporting EJB3 (and not EJB 3.1 yet).
| Version | 5.3.0-M4 |
| What do you get ? | Not yet certified |
| Who is behind? | OW2 |
| Open source? | LGPL |
| Documentation | Documentation |
| Download | Zip, tar.gz |
| Size of the downloaded file | 211 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 231 Mb |
| How to start it | %JONAS_HOME%\bin\jonas start |
| Location of log files | %JONAS_HOME%\logs |
| Best startup time | 11 036ms |
| Size of the process in RAM at startup | 107 Mb |
| Welcome page | http://localhost:9000/ |
| Admin console (*) | http://localhost:9000/jonasAdmin |
(*) user/pwd : jonas/jonas
Siwpas Enterprise 2.0.0-GA
Siwpas (Simple Web Profile Application Server) is quite new compared to the good old application servers. But like most of them, it embeds well known open source stacks, most from Apache (Tomcat, OpenEJB, OpenWebBeans, MyFaces, OpenJPA, Apache BeanValidation). It’s open source but you’ll still need a 30 days trial license to startup the enterprise version of the server.
| Version | 2.0.0-GA |
| What do you get ? | Web Profile 1.0 (Not yet certified) |
| Who is behind? | Mechsoft |
| Open source? | |
| Documentation | Documentation |
| Download | Zip, exe, tar.gz |
| Size of the downloaded file | 34.2 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 38.2 Mb |
| How to start it | %SIWPAS_HOME%\bin\startup |
| Location of log files | %SIWPAS_HOME%\logs |
| Best startup time | 8 598ms |
| Size of the process in RAM at startup | 72.8 Mb |
| Welcome page | http://localhost:8080/ |
| Admin console (*) | http://localhost:8080/console |
(*) user/pwd : admin/admin
Websphere 8.0.0.1
Websphere 8 is the Java EE 6 application server of IBM. Like the previous versions, you need a big bandwith to download one gigabyte of modules that get installed in nearly 2 hours. I noticed that there is a developer edition but once installed it looks like the 2.2 Gb of the full server (I might got lost in the maze of the IBM website). There is also a Community Edition and it looks like it’s based on Geronimo (and the executable to install it is also 116 Mb). IBM, please, make it simple.
Thanks to @sebsto I found a gem : it looks like IBM is working on a version WAS 8.5 Alpha that would look like the other app servers (light and easy to install). I’ll stay tuned on this one.
| Version | 8 |
| What do you get ? | Java EE 6 |
| Who is behind? | IBM |
| Open source? | Commercial product |
| Documentation | Documentation |
| Download | Exe file for WAS & WAS for developers |
| Size of the downloaded file | 116 Mb |
| Installation mode | Execute + Wizard install (with all the defaults) which downloads more stuff (took nearly 1:40 hour) |
| Size once installed on drive | 2.2 Gb |
| How to start it | Execute %WEBSPHERE_HOME%\AppServer\profiles\AppSrv01\bin\startServer.bat server1 -profileName AppSrv01 |
| Location of log files | %WEBSPHERE_HOME%\AppServer\profiles\AppSrv01\logs\server1 |
| Best startup time | 42 s |
| Size of the process in RAM at startup | 155 Mb |
| Admin console | https://localhost:9043/ibm/console |
Jeus 7
A bit unknown but in version 7.0 (so it shows this application server has been around for long time), Jeus 7from TMaxSoft was a bit difficult to startup (create an account on the web site, download, install, run a wizard and read the README.txt to make it started). Then I discovered that there is no Jeus 7 for Windows so I downloaded the Linux version and installed it on Ubuntu (so the results of the microbenchmark are not very accurate compare to the other made on Windows).
| Version | 7 |
| What do you get ? | Java EE 6 |
| Who is behind? | TmaxSoft |
| Open source? | Commercial |
| Documentation | Documentation |
| Download | bin file (only available on Linux) |
| Size of the downloaded file | 126 Mb |
| Installation mode | Execute the file and answer a wizard |
| Size once installed on drive | 198 Mb |
| How to start it | Execute $JEUS_HOME/bin/jeus -u administrator -p <pwd you declared in the wizard> |
| Location of log files | $JEUS_HOME/domains/jeus-domain1/logs |
| Best startup time | 23 783ms (but again that was on Ubuntu, not in Windows) |
| Size of the process in RAM at startup | 147 Mb |
| Welcome page | http://localhost:8088/ (but looks like there’s no page) |
| Admin console | Not sure there’s a admin console, but jeusadmin tool helps you to manage the server |
Interstage
I couldn’t even find where to click and download the server, so if anybody knows, I’ll be ready to give it a try.
Where the hell is Tomcat ?
Tomcat 7 is not a Java EE 6 application server nor does it implement the Web Profile 1.0. It is just a Servlet 3.0 container, so it’s shouldn’t appear in this blog post. But, everybody likes Tomcat, everybody feels Tomcat is way much faster than any server on earth (even faster than Neutrinos or maybe not). So I had to put it in this benchmark just to be a source of comparaison. And, indeed, Tomcat is the fastest application server of the list (3 times faster than GlassFish or 4 times faster than JBoss). But it only does Servlet 3 while the others implement between 12 (Web Profile) or 28 (Java EE 6) specifications.
| Version | 7.0.22 |
| What do you get ? | Servlet 3 |
| Who is behind? | Apache |
| Open source? | Apache Software License |
| Documentation | Documentation Index |
| Download | Zip, tar.gz |
| Size of the downloaded file | 7.5 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 11.3 Mb |
| How to start it | Execute %TOMCAT_HOME%\bin\startup.bat |
| Location of log files | %TOMCAT_HOME%\logs |
| Best startup time | 694 ms |
| Size of the process in RAM at startup | 34.6 Mb |
| Welcome page | http://localhost:8080/ |
| Admin console | http://localhost:8080/manager/html |
Summary
As a summary I will show you two graphs with startup time and resource consumption so it’s easier to compare.
Startup time
When you are a developer, startup up time matters (it doesn’t when you are in production). Thanks to tools like JRebel or the hot deployment of our IDEs, we tend to restart our app servers less and less. But, sometimes it happens (PermGens, classpath mess up, memory leaks, refreshing app server cache…). I didn’t put WAS 8 nor Jeus 7 in the graph below because they would just break the scale (42 and 23 seconds). It’s interesting to notice that 4 app servers (Resin, GlassFish, JBoss and TomEE) start in less that 5 seconds (Jonas & Geronimo are still in Milestone). Resin is 3 times slower than Tomcat, but again, it comes with the Web Profile.
Size once installed on the drive & size of process in RAM
The size of the application installed on your disk is not very accurate : some app servers come with documentation, samples… others no. But still, that gives you an overview of how big an app server can be (again, I couldn’t put WAS in this diagram as it will break the scale with its 2.2 Gb). The memory consumption is more interesting. Without any JVM or server twists, when you startup, they nearly all take less that 100Mb of RAM(see how close TomEE, GlassFish, JBoss and Siwpas are).
Conclusion
First of all, as you can see there is no Weblogic. Unfortunatelly Weblogic is the last big player to be certified Java EE 6. I don’t know the roadmap, hopefully I’ll soon be able to update this blog entry with the latest version of Weblogic. Second, this is not a real benchmark : I didn’t deploy any application nor I did some stress tests. One task I would like to do is to deploy a Java EE 6 application on all these servers and see how portable the code is. But that’s for another blog.
What did I want to show in this blog ? Well, that application servers have changed. Except for WAS, Weblogic or Jeus, all the others are fast at startup and take few resources (memory).
And why did I want to show this ? Because I’m fed up to still hear in 2011 : “Tomcat is light, Java EE app servers are not”. As you can see, in terms of startup and memory, the gap between Tomcat and Resin, GlassFish or JBoss is really small.
How is it going to affect your development life ? Do you remember when 7/8 years ago we had to mock our database accesses because we didn’t want to install a fat oracle database on our laptop ? These days are over now because we use in-memory databases such as H2 or Derby. Same thing is happening with application servers : soon app servers will be running in memory and we won’t be mocking anything. We will fire a test case, app servers and databases in memory, and that’s it. And with tools like Arquilian, it looks like this future is already happening.
Long live Java EE 6 and Java EE 6 application servers !
thanks for this,
jsut a “detail” note: for tomee the webadmin could be http://localhost:8080/openejb/ too
Do you plan to do an embedded test? (i wonder where could be tomee-embedded)
Thanks. I hope I’ll have time to deploy an application (Web Profile) on all of these app servers, and why not use the EJB embedded container to do some unit tests maybe
Nice article!
even the not certified ones, had you the chance to test JAX-RS on ecah server? that could produce another excellent article.
The application I would like to deploy would follow the Web Profile (which doesn’t have JAX-RS), but I’ll try to include it.
Good post, great work! I think startup times are very important in development. There has been a lot of improvement over the past years. I wish it could have been a priority in the WL8 days :).
I’ve run similar measurements about a year ago ( http://dow.ngra.de/2010/12/14/jee-oss-container-startup-times-apples-vs-oranges/ ). There also we thought that next step is to actually deploy an application and measure the startup time then. Deployment will trigger a longer startup time because more services are used and will give a better picture. I wish there was a good app that could be used on all app servers (or maybe even on servlet containers) not just app server specific admin consoles.
and please don’t use the task manager to check mem consumption but rather do it via yourkit profiler or jvisualvm plz – txs!
LieGrue,
strub
The other problem with including tomcat in the comparison is that it is not running any applications, where the appservers are all (except Jeus 7) running (and have accessed) a fairly complex web application (their console). The startup and memory times for tomcat are not really comparable in that sense either.
I had the same problem with TomEE: could not create a user with admin console access. I figured out that there are some incosistencies between tomee documentation and example XML’s. A correct entry in tomcat-users.xml goes like:
Two important things here:
1. the “username” keyword (instead of “name” which can be found in examples)
2. The “manager-gui” role
Anyway, I found out that TomEE cannot run a simple servlet without web.xml, which is incorrect… Sad thing which makes me doubt in all the certification processes like web profile cert given to TomEE…
Bare with us on TomEE. The TCK uses JSR-88 for deployment and it’s become clear that this is just different enough from Tomcat’s drop-and-wait webapps dir approach to cause issues. We’re hacking up a second JSR-88 impl for the drop-and-wait approach and that will help us work out these issues. All should be well before TomEE goes final.
My last comment has missing XML which should be (without left and right border):
user username=”admin” password=”” roles=”standard,manager-script,manager-gui”
@KCiesielski: TomEE allows servlet 3 deployment without a web.xml on trunk, did you try the SNAPSHOT version?
Well, probably they fixed it. I tried version available on website (1.0.0-beta-1). It was the same website that stated loudly that TomEE was JEE6-WebProfile-certified, hence my disappointment.
Well, just saw your post while reading the announcement of the launch of Weblogic 12.
Interested to see how it will compare after the long long wait.
Awesome blogpost, keep up the good work!
http://dhartford.blogspot.com/2011/12/javaee-6-app-servers-compared.html
Using latest weblogic 12c, i got really good startup times between 5 seconds and 7 seconds with an empty domain.
Time to add WebLogic 12c to the list – http://blogs.oracle.com/theaquarium/entry/oracle_weblogic_12c_does_full
Time for a WebLogic and TomEE update!
Yes, please. update with WebLogic 12c and new JBoss 7.1 and GlassFish 3.1.2 would be great!
Hi,
what about plain Jetty?
WebSphere now has beta of the Liberty profile (to be shipped in WAS 8.5 in a couple of months) and it starts in under 3 seconds. You can test drive it now: http://wasdev.net/
This is sort of useless data. I really don’t care how big the executable is, or how much RAM it takes or how fast it is to startup or how “easy” or “difficult” the installer is. Once a server is up, it stays up for a LONG time. Often times, when a server stars up it runs a bunch of initialization code, warms-up caches, loads data from persistent storage, initializes state, etc… That stuff take WAY longer than the server actually starting up and the data you hold in RAM dwarfs the actual base server cost. So, I really don’t understand the point of this exercise, it provides no really relevant data for a real-world scenario.
What is relevant is the performance characteristics of the various servers in terms of load, caching, messaging, DB management, etc… I want to know how does Glassfish do vs. Geronimo when we throw a couple hundred requests at it per second with a complex distribution in terms of various services, DB access, logging, event passing, etc…
Couldn’t agree more Micheal
Micheal, I don’t agree with you at all.
What you’re asking matters, don’t get me wrong, but there are already benchmarks for that. SPEC for instance, you’ve probably heard about them.
Those benchmarks are needed when you are a (very) large enterprise that has a multi-million line application, serving an established base of demanding customers and that needs to deal with a lot of traffic and gives out SLA guarantees, QoS, etc.
For small shops, indy developers, people new to the platform trying out things, and I dare even say developers in general, the stuff Antonio tests here is of the *utter most importance*!
In development, not even with JRebel, my server DOES NOT stay up for a LONG time. I continuously restart and redeploy. The size of the executable matters A LOT when you have different workspaces in say Eclipse, and assign each workspace an individual server to keep things simple. If all that’s needed to “install” a new server is to unpack a 25MB tarball (TomEE), then the ease with which I can create new workspaces for some radical experiments is unparalleled (and yes, developers need to experiment a lot).
WebSphere here is atrocious. As a simple developer trying out things, I don’t care about uptimes, or requests per second. All I need is a Java EE runtime, so I can deploy my sample code and see how it behaves. Having to download gigabytes of data and sit through installs that take hours, is just undoable. From an operations point of view, where you installed WebSphere 6 in 2004 and let it sit there untouched for 10+ years, it might indeed not matter, but for an agile and dynamic developer it’s a total disaster. And I’m not even talking about the IBM JDK requirement and having to run both Eclipse and WebSphere with that JDK (which thus means I can’t run on OS X).
Exactly your idea, that difficult installs and gigabytes of enterprise cr*p don’t matter, is EXACTLY what has been driving so many developers to alternative platforms. If you can’t grab the mindshare of new developers, then eventually your platform will just die out. It’s as simple as that.
Luckily, not everyone thinks as you, since GlassFish, JBoss AS, TomEE, etc have been addressing exactly this requirement. Even former heavyweight WebLogic is almost there (it still requires an install instead of just an unpack, but they’re trying). And surprisingly, even IBM is getting the message as Liberty is exactly the kind of lightweight setup developers grave for. Unfortunately Liberty doesn’t support all Java EE APIs yet, but if it eventually does (even if only the web profile) IBM might have a winner there again.