Updated
Thanks for your comments. I’ve added Undertow to the test ( “Servlet-Only Distribution” on the WildFly download page), corrected some mistakes, but more important, I’ve changed the memory benchmark: I now take a memory usage measure at startup, I then perform GC and wait a bit for memory to stabilize. This gives a nice min and max memory usage.
Damn, I’ve been waiting so long to publish this blog (which is the successor of the same post on Java EE 6 app servers). The idea is to do some “benchmarking”(basically, startup time, disk and memory usage) on application servers implementing Java EE 7. Java EE 7 came out in may 2013 and we have now many application servers that have passed the TCK. So let’s see where we are in terms of Java EE 7 implementations and production support.
State of Java EE Application Servers
Before diving into each implementation, let’s quickly see what happened in the application server land in these last 3 years.
Geronimo is no longer supported and will not run the race of Java EE 7. Not sure about JonAS but it hasn’t been updated since. GlassFish is still developped by Oracle and still is the Java EE reference implementation, but has no longer production support. You will have to look into Payara which is the supported version of GlassFish. Instead, Oracle is focusing on Weblogic which has Java EE 7 production support. JBoss 7 EAP is still in Beta but WildFly (the community edition of JBoss AS) has been supporting Java EE 7 since version 8. Websphere has now a Liberty and a Classic version. Jeus, from TmaxSoft and uCosminexus from Hitachi are still running the Java EE 7 race but are not as known as the other app servers.
The main editors also have a Java EE 7 Web Profile implementation (GlassFish, Payara, Websphere) and some only implement the Web Profile such as Resin, Siwpas, or TomEE. Last but not least, the Servlet containers also run the Java EE 7 race as they implement Servlet 3.1, such as Jetty, Tomcat or Undertow.
The Benchmark
Disclamer: This is not a real benchmark !
The idea of this benchmark is to download a Java EE 7 application server, install it, start it, launch the admin console if any, and take some measures : size of download, ease of installation, size on disk once installed, startup time, memory usage... That’s all. I do not deploy any application, I don’t do fancy twists to gain performance… I’m just concerned about the usability of an application server for a developer in 2016. I’m doing all my tests on a Retina OS X (16 Gb or RAM, SSD), no Docker! I use JDK 1.8.0_66 (when it’s not bundled with the server). No optimization at all is made (I haven’t twisted the JVM, or any application server parameter… everything comes out of the box).
To calculate the startup time, I don’t do any fancy rocket science either. I just start the server a few times, check the logs to see how long it takes and use the best startup time. Also remember that some servers do not load any container at startup, making them very fast to start. That’s why I trigger the web admin console (when there’s one) so I’m sure at least one web application is deployed. To calculate the memory footprint, I use JConsole and take two measures: memory used at server startup, and memory used after few seconds (after the GC calms down a bit).
Java EE 7 Application Servers
Java EE 7 comes with a Web profile and a Full profile. Full profile means that all the Java EE 7 specifications are supported. Not all the application servers support the Full profile. Here I look at GlassFish, Payara, JBoss EAP, WildFly, Weblogic and Websphere Liberty.
GlassFish 4.x
GlassFish 4 is the open source reference implementation for Java EE 7. Being the reference implementation, GlassFish 4.x implements both Java EE 7 and the Web Profile 7, that’s why you have two different bundles you can download. Oracle has dropped commercial support for GlassFish and Payara is now the supported version.
| Version | GlassFish 4.1.1 | GlassFish Web 4.1.1 |
| What do you get ? | Java EE 7 Full Profile | Java EE 7 Web Profile |
| Who is behind? | Oracle | |
| Open source? | Dual license CDDL and GPL | |
| Documentation | Documentation | |
| Download | Zipfile | Zipfile |
| Size of the downloaded file | 109.7 Mb | 60.7 Mb |
| Installation mode | Unzip | |
| Size once installed on drive | 136.9 Mb | 82.4 Mb |
| How to start it | $GLASSFISH_HOME/bin/asadmin start-domain | |
| Location of log files | $GLASSFISH_HOME/glassfish/domains/domain1/logs | |
| Best startup time | 2 821 ms | 1 991 ms |
| RAM at startup (later) | 140 Mb (44 Mb) | 120 Mb (33 Mb) |
| Welcome page | http://localhost:8080/ | |
| Admin console | http://localhost:4848 | |
Payara
Payara is the commercial support of GlassFish (so it also comes with a Full and Web profile). It is optimized for production environment and also has a micro-service distribution called Payara Micro.
| Version | Payara 4.1.1.154 | Payara Web 4.1.1.154 |
| What do you get ? | Java EE 7 Full Profile | Java EE 7 Web Profile |
| Who is behind? | C2B2 | |
| Open source? | Dual license CDDL and GPL | |
| Documentation | Documentation | |
| Download | Zipfile | Zipfile |
| Size of the downloaded file | 116 Mb | 66.6 Mb |
| Installation mode | Unzip | |
| Size once installed on drive | 142 Mb | 87.6 Mb |
| How to start it | $PAYARA_HOME/bin/asadmin start-domain | |
| Location of log files | $PAYARA_HOME/glassfish/domains/domain1/logs | |
| Best startup time | 3 001 ms | 1 953 ms |
| RAM at startup (later) | 144 Mb (43 Mb) | 118 Mb (33 Mb) |
| Welcome page | http://localhost:8080/ | |
| Admin console | http://localhost:4848 | |
JBoss 7 EAP
Unfortunatelly JBoss 7 EAP is still not final at the time of writing this blog, so no commercial support at RedHat. JBoss 7 EAP is based on WildFly, which is the community edition and evolves at a faster pace.
| Version | JBoss 7 EAP Beta 1 |
| What do you get ? | Java EE 7 Full Profile |
| Who is behind? | JBoss/RedHat |
| Open source? | LGPL |
| Documentation | Documentation |
| Download | Zipfile |
| Size of the downloaded file | 157.8 Mb |
| Installation mode | Unzip |
| Size once installed on drive | 184.2 Mb |
| How to start it | $JBOSS_HOME/bin/standalone |
| Location of log files | $JBOSS_HOME/standalone/log |
| Best startup time | 2 106ms |
| RAM at startup (later) | 337 Mb (41 Mb) |
| Welcome page | http://localhost:8080/ |
| Admin console | http://localhost:9990/console |
WildFly
WildFly is the community name for the JBoss application server. It has been supporting Java EE 7 since version 8.x. Both WildFly and EAP support the Full and the Web profile but there is no separate bundles, only one.
| Version | WildFly 8.2.1.Final | WildFly 9.0.2.Final | WildFly 10.0.0.Final |
| What do you get ? | Java EE 7 Full Profile | ||
| Who is behind? | RedHat | ||
| Open source? | LGPL | ||
| Documentation | WildFly 8.x | WildFly 9.x | WildFly 10.x |
| Download | Zipfile | Zipfile | Zipfile |
| Size of the downloaded file | 136.9 Mb | 136.7 Mb | 138.2 Mb |
| Installation mode | Unzip | ||
| Size once installed on drive | 140.4 Mb | 161.8 Mb | 163.4 Mb |
| How to start it | $WILDFLY_HOME/bin/standalone | ||
| Location of log files | $WILDFLY_HOME/standalone/log | ||
| Best startup time | 1 718 ms | 1 887 ms | 1 989 ms |
| RAM at startup (later) | 97 Mb (27 Mb) | 112 Mb (26 Mb) | 127 Mb (28 Mb) |
| Welcome page | http://localhost:8080/ | ||
| Admin console | http://localhost:4848 | ||
Weblogic 12.x
If Oracle still uses GlassFish as the Java EE reference implementation, its work and support goes into Weblogic. Weblogic 12 has recently been Java EE 7 certified.
| Version | Weblogic 12.2.1.0.0 |
| What do you get ? | Java EE 7 Full Profile |
| Who is behind? | Oracle |
| Open source? | No |
| Documentation | Documentation |
| Download | Zipfile + installer |
| Size of the downloaded file | 221.3 Mb |
| Installation mode | Unzip + install |
| Size once installed on drive | 749.9 Mb |
| How to start it | $WLS_HOME/user_projects/domains/base_domain/startWebLogic |
| Location of log files | %WLS_HOME/standalone/log |
| Best startup time | 8 411 ms* |
| RAM at startup (later) | 182 Mb (71 Mb) |
| Admin console | http://localhost:7001/console |
*Weblogic doesn’t show milliseconds in the logs so I had to change the starting script to add a few date +%s%3N.
Websphere and WebSphere Liberty 8.5.x
Today IBM has two versions of Websphere 8.5 : Classic and Liberty, both implementing Java EE 7. As for the Classic, you need to get lost in the IBM website maze and ask for help on Tweeter, to understand that you can’t install it on Mac OS X. So I just concentrated on WebSphere Liberty which is a completely different beast: download, unzip, run. There is no administration console but.
| Version | WebSphere Liberty 8.5.5.8 | WebSphere Liberty Web 8.5.5.8 |
| What do you get ? | Java EE 7 Full Profile | Java EE 7 Web Profile |
| Who is behind? | IBM | |
| Open source? | Commercial product | |
| Documentation | Documentation | |
| Download | Zipfile | Zipfile |
| Size of the downloaded file | 99.8 Mb | 67.5 Mb |
| Installation mode | Unzip | |
| Size once installed on drive | 122 Mb | 83.4 Mb |
| How to start it | $WLP_HOME/bin/server start | |
| Location of log files | $WLP_HOME%/usr/servers/defaultServer/logs | |
| Best startup time | 3 471 ms | 2 093 ms |
| RAM at startup (later) | 265 Mb (63 Mb) | 49 Mb (39 Mb) |
| Welcome page | http://localhost:9080/ | |
Java EE Web Profile 7 Application Servers
If GlassFish, Payara and Websphere have two different distributions (Full and Web profile), some application servers only implement the Web Profile. That’s the case of TomEE and Resin.
TomEE 7.x
Apache TomEE is the perfect success story for the Web Profile. TomEE is no more than Tomcat + OpenWebBeans + OpenEJB + OpenJPA + MyFaces + other bits. It really shows that Java EE is a jigsaw puzzle where you can take open standards, bundle them together and become a certified Web Profile application server. This TomEE version is bundled with Tomcat 8.0.29.
| Version | TomEE 7.0.0-M1 |
| What do you get ? | Java EE 7 Web Profile |
| Who is behind? | Apache |
| Open source? | Apache Software License |
| Documentation | Documentation |
| Download | Zip, tar.gz |
| Size of the downloaded file | 35.5 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 42.1 Mb |
| How to start it | $TOMEE_HOME/bin/startup.sh |
| Location of log files | $TOMEE_HOME/logs |
| Best startup time | 505 ms |
| RAM at startup (later) | 50 Mb (24 Mb) |
| Welcome page | http://localhost:8080/ |
| Admin console (*) | http://localhost:8080/manager/html |
(*) To be able to log on to the admin console, you need to change the $TOMCAT_HOME/conf/tomcat-users.xml configuration files and add a manager-gui role to a user.
Resin 5.x
Resin was one of the first servlet containers (such as Jetty or Tomcat) to move to the Web Profile. Again, it’s another success story that confirmed that profiles in Java EE were needed. Resin is very CDI centric and based on the Caucho implementation called CanDI.
| Version | Resin 5.0.2-alpha |
| What do you get ? | Java EE 7 Web Profile |
| Who is behind? | Caucho |
| Open source? | GPL License |
| Documentation | Documentation |
| Download | Zip, tgz, deb |
| Size of the downloaded file | 10.7 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 11.7 Mb |
| How to start it | $RESIN_HOME/bin/resin start |
| Location of log files | /tmp/resin/log/ |
| Best startup time | 1 303 ms |
| RAM at startup (later) | 2 processes : 61 Mb + 39 Mb |
| Welcome page | http://localhost:8080/ |
Servlet 3.1 Servers
You can see Java EE 7 application servers implementing everything (the Full profile), a bit (Web profile) or just the web server portion (Servlet 3.1). That’s the case of Tomcat, Jetty or Undertow. Unfortunately I do not benchmark Undertow because it doesn’t have a standalone installation: you need to start it up in your code.
Tomcat
Tomcat has been implementing Servlet 3.1 since version 8.
| Version | Tomcat 8.0.30 | Tomcat 9.0.0.M1 |
| What do you get ? | Servlet 3.1 | |
| Who is behind? | Apache | |
| Open source? | Apache Software License | |
| Documentation | Documentation | Documentation |
| Download | Zip, tar.gz | Zip, tar.gz |
| Size of the downloaded file | 9.7 Mb | 9.9 Mb |
| Installation mode | Unzip the file | |
| Size once installed on drive | 14.6 Mb | 14.5 Mb |
| How to start it | $TOMCAT_HOME/bin/startup.bat | |
| Location of log files | $TOMCAT_HOME/logs | |
| Best startup time | 401 ms | 404 ms |
| RAM at startup (later) | 45 Mb (17 Mb) | 40 Mb (15 Mb) |
| Welcome page | http://localhost:8080/ | |
| Admin console | http://localhost:8080/manager/html | |
Jetty
Jetty started to implement Servlet 3.1 from Jetty 9.
| Version | Jetty 9.3.6.v20151106 |
| What do you get ? | Servlet 3.1 |
| Who is behind? | Eclipse |
| Open source? | Apache License 2.0 and Eclipse Public License 1.0 |
| Documentation | Documentation |
| Download | Zip, tgz, deb |
| Size of the downloaded file | 11.7 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 13.4 Mb |
| How to start it | $JETTY_HOME/bin/jetty.sh start |
| Location of log files | $JETTY_HOME/logs |
| Best startup time | 479 ms |
| RAM at startup (later) | 52 Mb (8 Mb) |
| Welcome page | http://localhost:8080/ |
Undertow
JBoss Application Server used to use Tomcat as the web container. Since version 8 it uses Undertow that can be downloaded on its own.
| Version | WildFly Servlet 10.0.0.Final |
| What do you get ? | Servlet 3.1 |
| Who is behind? | RedHat |
| Open source? | LGPL |
| Documentation | Documentation |
| Download | Zip (unser Servlet-Only Distribution) |
| Size of the downloaded file | 29.2 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 35.8 Mb |
| How to start it | $WILDFLY_HOME/bin/standalone.sh |
| Location of log files | $WILDFLY_HOME/standalone/log |
| Best startup time | 1 252 ms |
| RAM at startup (later) | 93 Mb (16 Mb) |
| Welcome page | http://localhost:8080/ |
Summary
As a summary I will show you two graphs with startup time and memory consumption so it’s easier to compare.
Startup time
When you are a developer, startup up time matters (even when you move your app server to the Cloud). Thanks to tools like JRebel, or hot deployment in our IDEs, we tend to restart our app servers less and less. But still, it’s important to have fast startup time. Something to stress out, is that most of the app servers bootstrap the minimum set of services and then, lazy load services when needed.
From left to right we have Full profile, Web profile and Servlet containers. Of course, Servlet containers start faster. But it’s interesting to notice that most app servers boot in less that 3 seconds, and half, in less than 2 seconds (remember that these measures are taken on a Mac OS X 16Gb RAMP SSD). Weblogic is by far the slowest (> 8 seconds) and WildFly the fastest of the Java EE Full profile app servers. TomEE 7 (Web profile) is slightly slower than Tomcat (Servlet).
Memory Footprint
The memory consumption is also important. In this era of Micro Services, we want fast startup time and low resource consumption. Without any JVM or server twists, most app servers use less than 150Mb of RAM. Here I take the measure at startup, perform a GC and wait a few seconds so it stabilize. So you get a min and a max memory usage.
Conclusion
First of all, as you can see there is no Websphere Classic. Despite being a big player, it is difficult to find, difficult to install, and has no Mac OS X support. I hope one day I’ll be able to add it to the benchmark.
What did I want to show in this blog ? Well, that application servers have changed. Most app servers take few resources, are modular and have nice tricks to start up fast (like lazy loading services).
And why did I want to show this ? Just to show that the “Tomcat is light, Java EE app servers are not” line is outdated. Use the app server that fits your need. You need Servlet ? Go for Jetty or Tomcat. You need some JAX-RS with CDI and JPA ? Choose TomEE, WildFly or Websphere Liberty. You need the full monty ? Use WildFly, Weblogic or GlassFish. You need production support ? Use JBoss EAP (you’ll have to wait a few extra month). You need to create Micro Services ? Wait for my next blog ;o)
References