Updated
Thanks for your comments. I’ve added Undertow to the test ( “Servlet-Only Distribution” on the WildFly download page), corrected some mistakes, but more important, I’ve changed the memory benchmark: I now take a memory usage measure at startup, I then perform GC and wait a bit for memory to stabilize. This gives a nice min and max memory usage.
Damn, I’ve been waiting so long to publish this blog (which is the successor of the same post on Java EE 6 app servers). The idea is to do some “benchmarking”(basically, startup time, disk and memory usage) on application servers implementing Java EE 7. Java EE 7 came out in may 2013 and we have now many application servers that have passed the TCK. So let’s see where we are in terms of Java EE 7 implementations and production support.
State of Java EE Application Servers
Before diving into each implementation, let’s quickly see what happened in the application server land in these last 3 years.
Geronimo is no longer supported and will not run the race of Java EE 7. Not sure about JonAS but it hasn’t been updated since. GlassFish is still developped by Oracle and still is the Java EE reference implementation, but has no longer production support. You will have to look into Payara which is the supported version of GlassFish. Instead, Oracle is focusing on Weblogic which has Java EE 7 production support. JBoss 7 EAP is still in Beta but WildFly (the community edition of JBoss AS) has been supporting Java EE 7 since version 8. Websphere has now a Liberty and a Classic version. Jeus, from TmaxSoft and uCosminexus from Hitachi are still running the Java EE 7 race but are not as known as the other app servers.
The main editors also have a Java EE 7 Web Profile implementation (GlassFish, Payara, Websphere) and some only implement the Web Profile such as Resin, Siwpas, or TomEE. Last but not least, the Servlet containers also run the Java EE 7 race as they implement Servlet 3.1, such as Jetty, Tomcat or Undertow.
The Benchmark
Disclamer: This is not a real benchmark !
The idea of this benchmark is to download a Java EE 7 application server, install it, start it, launch the admin console if any, and take some measures : size of download, ease of installation, size on disk once installed, startup time, memory usage... That’s all. I do not deploy any application, I don’t do fancy twists to gain performance… I’m just concerned about the usability of an application server for a developer in 2016. I’m doing all my tests on a Retina OS X (16 Gb or RAM, SSD), no Docker! I use JDK 1.8.0_66 (when it’s not bundled with the server). No optimization at all is made (I haven’t twisted the JVM, or any application server parameter… everything comes out of the box).
To calculate the startup time, I don’t do any fancy rocket science either. I just start the server a few times, check the logs to see how long it takes and use the best startup time. Also remember that some servers do not load any container at startup, making them very fast to start. That’s why I trigger the web admin console (when there’s one) so I’m sure at least one web application is deployed. To calculate the memory footprint, I use JConsole and take two measures: memory used at server startup, and memory used after few seconds (after the GC calms down a bit).
Java EE 7 Application Servers
Java EE 7 comes with a Web profile and a Full profile. Full profile means that all the Java EE 7 specifications are supported. Not all the application servers support the Full profile. Here I look at GlassFish, Payara, JBoss EAP, WildFly, Weblogic and Websphere Liberty.
GlassFish 4.x
GlassFish 4 is the open source reference implementation for Java EE 7. Being the reference implementation, GlassFish 4.x implements both Java EE 7 and the Web Profile 7, that’s why you have two different bundles you can download. Oracle has dropped commercial support for GlassFish and Payara is now the supported version.
| Version | GlassFish 4.1.1 | GlassFish Web 4.1.1 |
| What do you get ? | Java EE 7 Full Profile | Java EE 7 Web Profile |
| Who is behind? | Oracle | |
| Open source? | Dual license CDDL and GPL | |
| Documentation | Documentation | |
| Download | Zipfile | Zipfile |
| Size of the downloaded file | 109.7 Mb | 60.7 Mb |
| Installation mode | Unzip | |
| Size once installed on drive | 136.9 Mb | 82.4 Mb |
| How to start it | $GLASSFISH_HOME/bin/asadmin start-domain | |
| Location of log files | $GLASSFISH_HOME/glassfish/domains/domain1/logs | |
| Best startup time | 2 821 ms | 1 991 ms |
| RAM at startup (later) | 140 Mb (44 Mb) | 120 Mb (33 Mb) |
| Welcome page | http://localhost:8080/ | |
| Admin console | http://localhost:4848 | |
Payara
Payara is the commercial support of GlassFish (so it also comes with a Full and Web profile). It is optimized for production environment and also has a micro-service distribution called Payara Micro.
| Version | Payara 4.1.1.154 | Payara Web 4.1.1.154 |
| What do you get ? | Java EE 7 Full Profile | Java EE 7 Web Profile |
| Who is behind? | C2B2 | |
| Open source? | Dual license CDDL and GPL | |
| Documentation | Documentation | |
| Download | Zipfile | Zipfile |
| Size of the downloaded file | 116 Mb | 66.6 Mb |
| Installation mode | Unzip | |
| Size once installed on drive | 142 Mb | 87.6 Mb |
| How to start it | $PAYARA_HOME/bin/asadmin start-domain | |
| Location of log files | $PAYARA_HOME/glassfish/domains/domain1/logs | |
| Best startup time | 3 001 ms | 1 953 ms |
| RAM at startup (later) | 144 Mb (43 Mb) | 118 Mb (33 Mb) |
| Welcome page | http://localhost:8080/ | |
| Admin console | http://localhost:4848 | |
JBoss 7 EAP
Unfortunatelly JBoss 7 EAP is still not final at the time of writing this blog, so no commercial support at RedHat. JBoss 7 EAP is based on WildFly, which is the community edition and evolves at a faster pace.
| Version | JBoss 7 EAP Beta 1 |
| What do you get ? | Java EE 7 Full Profile |
| Who is behind? | JBoss/RedHat |
| Open source? | LGPL |
| Documentation | Documentation |
| Download | Zipfile |
| Size of the downloaded file | 157.8 Mb |
| Installation mode | Unzip |
| Size once installed on drive | 184.2 Mb |
| How to start it | $JBOSS_HOME/bin/standalone |
| Location of log files | $JBOSS_HOME/standalone/log |
| Best startup time | 2 106ms |
| RAM at startup (later) | 337 Mb (41 Mb) |
| Welcome page | http://localhost:8080/ |
| Admin console | http://localhost:9990/console |
WildFly
WildFly is the community name for the JBoss application server. It has been supporting Java EE 7 since version 8.x. Both WildFly and EAP support the Full and the Web profile but there is no separate bundles, only one.
| Version | WildFly 8.2.1.Final | WildFly 9.0.2.Final | WildFly 10.0.0.Final |
| What do you get ? | Java EE 7 Full Profile | ||
| Who is behind? | RedHat | ||
| Open source? | LGPL | ||
| Documentation | WildFly 8.x | WildFly 9.x | WildFly 10.x |
| Download | Zipfile | Zipfile | Zipfile |
| Size of the downloaded file | 136.9 Mb | 136.7 Mb | 138.2 Mb |
| Installation mode | Unzip | ||
| Size once installed on drive | 140.4 Mb | 161.8 Mb | 163.4 Mb |
| How to start it | $WILDFLY_HOME/bin/standalone | ||
| Location of log files | $WILDFLY_HOME/standalone/log | ||
| Best startup time | 1 718 ms | 1 887 ms | 1 989 ms |
| RAM at startup (later) | 97 Mb (27 Mb) | 112 Mb (26 Mb) | 127 Mb (28 Mb) |
| Welcome page | http://localhost:8080/ | ||
| Admin console | http://localhost:4848 | ||
Weblogic 12.x
If Oracle still uses GlassFish as the Java EE reference implementation, its work and support goes into Weblogic. Weblogic 12 has recently been Java EE 7 certified.
| Version | Weblogic 12.2.1.0.0 |
| What do you get ? | Java EE 7 Full Profile |
| Who is behind? | Oracle |
| Open source? | No |
| Documentation | Documentation |
| Download | Zipfile + installer |
| Size of the downloaded file | 221.3 Mb |
| Installation mode | Unzip + install |
| Size once installed on drive | 749.9 Mb |
| How to start it | $WLS_HOME/user_projects/domains/base_domain/startWebLogic |
| Location of log files | %WLS_HOME/standalone/log |
| Best startup time | 8 411 ms* |
| RAM at startup (later) | 182 Mb (71 Mb) |
| Admin console | http://localhost:7001/console |
*Weblogic doesn’t show milliseconds in the logs so I had to change the starting script to add a few date +%s%3N.
Websphere and WebSphere Liberty 8.5.x
Today IBM has two versions of Websphere 8.5 : Classic and Liberty, both implementing Java EE 7. As for the Classic, you need to get lost in the IBM website maze and ask for help on Tweeter, to understand that you can’t install it on Mac OS X. So I just concentrated on WebSphere Liberty which is a completely different beast: download, unzip, run. There is no administration console but.
| Version | WebSphere Liberty 8.5.5.8 | WebSphere Liberty Web 8.5.5.8 |
| What do you get ? | Java EE 7 Full Profile | Java EE 7 Web Profile |
| Who is behind? | IBM | |
| Open source? | Commercial product | |
| Documentation | Documentation | |
| Download | Zipfile | Zipfile |
| Size of the downloaded file | 99.8 Mb | 67.5 Mb |
| Installation mode | Unzip | |
| Size once installed on drive | 122 Mb | 83.4 Mb |
| How to start it | $WLP_HOME/bin/server start | |
| Location of log files | $WLP_HOME%/usr/servers/defaultServer/logs | |
| Best startup time | 3 471 ms | 2 093 ms |
| RAM at startup (later) | 265 Mb (63 Mb) | 49 Mb (39 Mb) |
| Welcome page | http://localhost:9080/ | |
Java EE Web Profile 7 Application Servers
If GlassFish, Payara and Websphere have two different distributions (Full and Web profile), some application servers only implement the Web Profile. That’s the case of TomEE and Resin.
TomEE 7.x
Apache TomEE is the perfect success story for the Web Profile. TomEE is no more than Tomcat + OpenWebBeans + OpenEJB + OpenJPA + MyFaces + other bits. It really shows that Java EE is a jigsaw puzzle where you can take open standards, bundle them together and become a certified Web Profile application server. This TomEE version is bundled with Tomcat 8.0.29.
| Version | TomEE 7.0.0-M1 |
| What do you get ? | Java EE 7 Web Profile |
| Who is behind? | Apache |
| Open source? | Apache Software License |
| Documentation | Documentation |
| Download | Zip, tar.gz |
| Size of the downloaded file | 35.5 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 42.1 Mb |
| How to start it | $TOMEE_HOME/bin/startup.sh |
| Location of log files | $TOMEE_HOME/logs |
| Best startup time | 505 ms |
| RAM at startup (later) | 50 Mb (24 Mb) |
| Welcome page | http://localhost:8080/ |
| Admin console (*) | http://localhost:8080/manager/html |
(*) To be able to log on to the admin console, you need to change the $TOMCAT_HOME/conf/tomcat-users.xml configuration files and add a manager-gui role to a user.
Resin 5.x
Resin was one of the first servlet containers (such as Jetty or Tomcat) to move to the Web Profile. Again, it’s another success story that confirmed that profiles in Java EE were needed. Resin is very CDI centric and based on the Caucho implementation called CanDI.
| Version | Resin 5.0.2-alpha |
| What do you get ? | Java EE 7 Web Profile |
| Who is behind? | Caucho |
| Open source? | GPL License |
| Documentation | Documentation |
| Download | Zip, tgz, deb |
| Size of the downloaded file | 10.7 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 11.7 Mb |
| How to start it | $RESIN_HOME/bin/resin start |
| Location of log files | /tmp/resin/log/ |
| Best startup time | 1 303 ms |
| RAM at startup (later) | 2 processes : 61 Mb + 39 Mb |
| Welcome page | http://localhost:8080/ |
Servlet 3.1 Servers
You can see Java EE 7 application servers implementing everything (the Full profile), a bit (Web profile) or just the web server portion (Servlet 3.1). That’s the case of Tomcat, Jetty or Undertow. Unfortunately I do not benchmark Undertow because it doesn’t have a standalone installation: you need to start it up in your code.
Tomcat
Tomcat has been implementing Servlet 3.1 since version 8.
| Version | Tomcat 8.0.30 | Tomcat 9.0.0.M1 |
| What do you get ? | Servlet 3.1 | |
| Who is behind? | Apache | |
| Open source? | Apache Software License | |
| Documentation | Documentation | Documentation |
| Download | Zip, tar.gz | Zip, tar.gz |
| Size of the downloaded file | 9.7 Mb | 9.9 Mb |
| Installation mode | Unzip the file | |
| Size once installed on drive | 14.6 Mb | 14.5 Mb |
| How to start it | $TOMCAT_HOME/bin/startup.bat | |
| Location of log files | $TOMCAT_HOME/logs | |
| Best startup time | 401 ms | 404 ms |
| RAM at startup (later) | 45 Mb (17 Mb) | 40 Mb (15 Mb) |
| Welcome page | http://localhost:8080/ | |
| Admin console | http://localhost:8080/manager/html | |
Jetty
Jetty started to implement Servlet 3.1 from Jetty 9.
| Version | Jetty 9.3.6.v20151106 |
| What do you get ? | Servlet 3.1 |
| Who is behind? | Eclipse |
| Open source? | Apache License 2.0 and Eclipse Public License 1.0 |
| Documentation | Documentation |
| Download | Zip, tgz, deb |
| Size of the downloaded file | 11.7 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 13.4 Mb |
| How to start it | $JETTY_HOME/bin/jetty.sh start |
| Location of log files | $JETTY_HOME/logs |
| Best startup time | 479 ms |
| RAM at startup (later) | 52 Mb (8 Mb) |
| Welcome page | http://localhost:8080/ |
Undertow
JBoss Application Server used to use Tomcat as the web container. Since version 8 it uses Undertow that can be downloaded on its own.
| Version | WildFly Servlet 10.0.0.Final |
| What do you get ? | Servlet 3.1 |
| Who is behind? | RedHat |
| Open source? | LGPL |
| Documentation | Documentation |
| Download | Zip (unser Servlet-Only Distribution) |
| Size of the downloaded file | 29.2 Mb |
| Installation mode | Unzip the file |
| Size once installed on drive | 35.8 Mb |
| How to start it | $WILDFLY_HOME/bin/standalone.sh |
| Location of log files | $WILDFLY_HOME/standalone/log |
| Best startup time | 1 252 ms |
| RAM at startup (later) | 93 Mb (16 Mb) |
| Welcome page | http://localhost:8080/ |
Summary
As a summary I will show you two graphs with startup time and memory consumption so it’s easier to compare.
Startup time
When you are a developer, startup up time matters (even when you move your app server to the Cloud). Thanks to tools like JRebel, or hot deployment in our IDEs, we tend to restart our app servers less and less. But still, it’s important to have fast startup time. Something to stress out, is that most of the app servers bootstrap the minimum set of services and then, lazy load services when needed.
From left to right we have Full profile, Web profile and Servlet containers. Of course, Servlet containers start faster. But it’s interesting to notice that most app servers boot in less that 3 seconds, and half, in less than 2 seconds (remember that these measures are taken on a Mac OS X 16Gb RAMP SSD). Weblogic is by far the slowest (> 8 seconds) and WildFly the fastest of the Java EE Full profile app servers. TomEE 7 (Web profile) is slightly slower than Tomcat (Servlet).
Memory Footprint
The memory consumption is also important. In this era of Micro Services, we want fast startup time and low resource consumption. Without any JVM or server twists, most app servers use less than 150Mb of RAM. Here I take the measure at startup, perform a GC and wait a few seconds so it stabilize. So you get a min and a max memory usage.
Conclusion
First of all, as you can see there is no Websphere Classic. Despite being a big player, it is difficult to find, difficult to install, and has no Mac OS X support. I hope one day I’ll be able to add it to the benchmark.
What did I want to show in this blog ? Well, that application servers have changed. Most app servers take few resources, are modular and have nice tricks to start up fast (like lazy loading services).
And why did I want to show this ? Just to show that the “Tomcat is light, Java EE app servers are not” line is outdated. Use the app server that fits your need. You need Servlet ? Go for Jetty or Tomcat. You need some JAX-RS with CDI and JPA ? Choose TomEE, WildFly or Websphere Liberty. You need the full monty ? Use WildFly, Weblogic or GlassFish. You need production support ? Use JBoss EAP (you’ll have to wait a few extra month). You need to create Micro Services ? Wait for my next blog ;o)
References
Great article. Good to see that there are no “winners” in this test. Just the conclusion that you use what you like:)
Just a small correction. The link to documentation in WebSphere Liberty 8.5.x
seems to point to Wildfly.
Nice! BTW, I think would be more fair put the “Java EE Web Profile” servers in a separate startup time graph. Is’t it?
awesome man!!!
currently reading your java EE 7 book and waiting for the next vjug meetup,
WebSphere V8.5.5 is still only Java EE 6. WebSphere Liberty may contain further enhancements towards Java EE 7 but is not certified. We still wait for something like WebSphere V9 + Liberty, Java EE 7 compliant…
Roman, you are correct about WebSphere Classic – it currently does not support EE 7. WebSphere Liberty 8.5.5.6, however, was one of the first EE 7 compliant application servers on the market. The latest version is available here:
https://developer.ibm.com/wasdev/downloads/#asset/runtimes-8.5.5-wlp-javaee7
I suppose I can help by making some comments for EAP/WildFly:
– EAP comes with higher default memory settings (-Xmx, etc.) out of the box, since it targets primarily production environments. E.g. check the diff in /bin/standalone.conf between EAP & WildFly.
– Although you didn’t really do any application benchmark tests, EAP 7 Beta is based on a older fork of WildFly 10, after which we’ve done a large number of performance improvements. When the EAP7 GA release comes out, its overall performance numbers should be in line with WildFly 10, if not better.
– WildFly/EAP does *some* lazy loading but not too much, like other containers. With WildFly 10 about 50% of services are actually started when the boot sequence is completed, thus when real application are deployed you won’t get the hit of having to start those services.
– When comparing servlet container, you may just as well consider the “Servlet-Only Distribution” on the WildFly download page. This will give you essentially Undertow with deployment capability like Tomcat & Jetty so you can compare.
Thanks for sharing this article!
What do both colours mean at the memory footprint graph?
Thank you
When you start WildFly with “$WILDFLY_HOME/bin/standalone”, it’s not JavaEE Full Profile because it hasn’t JMS. add “-c standalone-full.xml” to be full profile compliant.
Regarding Dimitris’ comment on lazyness, a bench with a small app containing a servlet, an EJB, a small JPA entity,… would be more relevant.
WebLogic requires some level of entropy during startup (and domain creation), so it makes use of the entropy pool in Linux-based environments. By default, WebLogic uses whatever is defined in the JVM java.security file (which is /dev/random, a device that blocks the process until enough random bits are provided).
To speedup WebLogic on Linux-based development environments, developers should configure their environments to use /dev/urandom. This can be done in different ways:
– Modify java.security
– Modify setDomainEnv.sh
– set USER_MEM_ARGS temporarily
You can find more details in this blog: http://theheat.dk/blog/?p=1539
I strongly recommend, Antonio, that you do adjust your JVM java.security file to use /dev/urandom by default, and then rerun your benchmarks.
A second note on WebLogic:
– it has two distributions: one called “Generic”, and one called “Quick Installer”. The distribution used in this benchmark, based on the size of the download, is the Generic installer. The quick installer is smaller (~250MB) because it is compressed with pack200.
– both distributions are able to create a domain in Development Mode
– Development Mode deploys the Admin Console by default
– Production Mode deploys Admin Console on first access
So, the benchmark could also state whether the application server startup time includes the startup of the console or not.
Two items to also think about with WAS Liberty:
1) While you are testing with the Web and Full Profile, you do not need to load all of it. It is a fit for feature/function architecture. Just need Servlet and JDBC, that is all that you need to start.
2) Once you start it, there are only two reasons you need to restart it. 1) JVM changes. 2) you want to. All configuration and app changes are dynamically picked up. Because the command line arguments for a JVM are not dynamic, this is not possible for Liberty resolve.
For your microservices article, please see how the server package –include=minify feature works on size of server package and usage with microservices.
As most of features of WAS Liberty, you need to declare the need of admin console as shown in in this video https://www.youtube.com/watch?v=W8sMeTZmRkE .
PS:
WLP is strongly based on OSGi where every features is made of bundles that can be downloaded and installed as needed
The repository of feature is at https://developer.ibm.com/wasdev/downloads/#filter/assetTypeFilters=FEATURE
Resin 5 is still Java EE 6 based (not 7)
über useful post, congrat
For WebSphere Classic you can get WebSphere 9 (I don’t think 8.5 supports Java EE7) for development purposes Get the IBM installation manager and add the repositories as you need it
Application Server that you want to install:
• Version 9.0: https://www.ibm.com/software/repositorymanager/V9WASILAN
• Version 8.5: https://www.ibm.com/software/repositorymanager/V85WASDeveloperILAN
• Version 8.0: https://www.ibm.com/software/repositorymanager/V8WASDeveloperILAN
As for standards support, as much as I would like WildFly to work with a non-trivial resource setup I am unable to get it working at all.https://site.trajano.net/jee/
WildFly fails on deployment for JAX-WS Web Services and JPA with non-default data sources.
A little errata, for wildfly the admin port is located :9990. But to use it you need to add a management user.
Best regards
Tank for the article, very good !
Are you sure for time to start WebSphere Liberty Web?