O Java EE 7 Application Servers, Where Art Thou?

Updated

Thanks for your comments. I’ve added Undertow to the test ( “Servlet-Only Distribution” on the WildFly download page), corrected some mistakes, but more important, I’ve changed the memory benchmark: I now take a memory usage measure at startup, I then perform GC and wait a bit for memory to stabilize. This gives a nice min and max memory usage.

Damn, I’ve been waiting so long to publish this blog (which is the successor of the same post on Java EE 6 app servers). The idea is to do some “benchmarking”(basically, startup time, disk and memory usage) on application servers implementing Java EE 7. Java EE 7 came out in may 2013 and we have now many application servers that have passed the TCK. So let’s see where we are in terms of Java EE 7 implementations and production support.

State of Java EE Application Servers

Before diving into each implementation, let’s quickly see what happened in the application server land in these last 3 years.

Geronimo is no longer supported and will not run the race of Java EE 7. Not sure about JonAS but it hasn’t been updated since. GlassFish is still developped by Oracle and still is the Java EE reference implementation, but has no longer production support. You will have to look into Payara which is the supported version of GlassFish. Instead, Oracle is focusing on Weblogic which has Java EE 7 production support. JBoss 7 EAP is still in Beta but WildFly (the community edition of JBoss AS) has been supporting Java EE 7 since version 8. Websphere has now a Liberty and a Classic version. Jeus, from TmaxSoft and uCosminexus from Hitachi are still running the Java EE 7 race but are not as known as the other app servers.

The main editors also have a Java EE 7 Web Profile implementation (GlassFish, Payara, Websphere) and some only implement the Web Profile such as ResinSiwpas, or TomEE. Last but not least, the Servlet containers also run the Java EE 7 race as they implement Servlet 3.1, such as Jetty, Tomcat or Undertow.

The Benchmark

Disclamer: This is not a real benchmark !

The idea of this benchmark is to download a Java EE 7 application server, install it, start it, launch the admin console if any, and take some measures : size of download, ease of installation, size on disk once installed, startup time, memory usage... That’s all. I do not deploy any application, I don’t do fancy twists to gain performance… I’m just concerned about the usability of an application server for a developer in 2016. I’m doing all my tests on a Retina OS X (16 Gb or RAM, SSD), no Docker! I use JDK 1.8.0_66 (when it’s not bundled with the server). No optimization at all is made (I haven’t twisted the JVM, or any application server parameter… everything comes out of the box).

To calculate the startup time, I don’t do any fancy rocket science either. I just start the server a few times, check the logs to see how long it takes and use the best startup time. Also remember that some servers do not load any container at startup, making them very fast to start. That’s why I trigger the web admin console (when there’s one) so I’m sure at least one web application is deployed. To calculate the memory footprint, I use JConsole and take two measures: memory used at server startup, and memory used after few seconds (after the GC calms down a bit).

Java EE 7 Application Servers

Java EE 7 comes with a Web profile and a Full profile. Full profile means that all the Java EE 7 specifications are supported. Not all the application servers support the Full profile. Here I look at GlassFishPayara, JBoss EAPWildFlyWeblogic and Websphere Liberty.

GlassFish 4.x

GlassFish 4 is the open source reference implementation for Java EE 7. Being the reference implementation, GlassFish 4.x implements both Java EE 7 and the Web Profile 7, that’s why you have two different bundles you can download. Oracle has dropped commercial support for GlassFish and Payara is now the supported version.

Version GlassFish 4.1.1 GlassFish Web 4.1.1
What do you get ? Java EE 7 Full Profile Java EE 7 Web Profile
Who is behind? Oracle
Open source? Dual license CDDL and GPL
Documentation Documentation
Download Zipfile Zipfile
Size of the downloaded file 109.7 Mb 60.7 Mb
Installation mode Unzip
Size once installed on drive 136.9 Mb 82.4 Mb
How to start it $GLASSFISH_HOME/bin/asadmin start-domain
Location of log files $GLASSFISH_HOME/glassfish/domains/domain1/logs
Best startup time 2 821 ms 1 991 ms
RAM at startup (later) 140 Mb (44 Mb) 120 Mb (33 Mb)
Welcome page http://localhost:8080/
Admin console http://localhost:4848

Payara

Payara is the commercial support of GlassFish (so it also comes with a Full and Web profile). It is optimized for production environment and also has a micro-service distribution called Payara Micro.

Version Payara 4.1.1.154 Payara Web 4.1.1.154
What do you get ? Java EE 7 Full Profile Java EE 7 Web Profile
Who is behind? C2B2
Open source? Dual license CDDL and GPL
Documentation Documentation
Download Zipfile Zipfile
Size of the downloaded file 116 Mb 66.6 Mb
Installation mode Unzip
Size once installed on drive 142 Mb 87.6 Mb
How to start it $PAYARA_HOME/bin/asadmin start-domain
Location of log files $PAYARA_HOME/glassfish/domains/domain1/logs
Best startup time 3 001 ms 1 953 ms
RAM at startup (later) 144 Mb (43 Mb) 118 Mb (33 Mb)
Welcome page http://localhost:8080/
Admin console http://localhost:4848

JBoss 7 EAP

Unfortunatelly JBoss 7 EAP is still not final at the time of writing this blog, so no commercial support at RedHat. JBoss 7 EAP is based on WildFly, which is the community edition and evolves at a faster pace.

Version JBoss 7 EAP Beta 1
What do you get ? Java EE 7 Full Profile
Who is behind? JBoss/RedHat
Open source? LGPL
Documentation Documentation
Download Zipfile
Size of the downloaded file 157.8 Mb
Installation mode Unzip
Size once installed on drive 184.2 Mb
How to start it $JBOSS_HOME/bin/standalone
Location of log files $JBOSS_HOME/standalone/log
Best startup time 2 106ms
RAM at startup (later) 337 Mb (41 Mb)
Welcome page http://localhost:8080/
Admin console http://localhost:9990/console

WildFly

WildFly is the community name for the JBoss application server. It has been supporting Java EE 7 since version 8.x. Both WildFly and EAP support the Full and the Web profile but there is no separate bundles, only one.

Version WildFly 8.2.1.Final WildFly 9.0.2.Final WildFly 10.0.0.Final
What do you get ? Java EE 7 Full Profile
Who is behind? RedHat
Open source? LGPL
Documentation WildFly 8.x WildFly 9.x WildFly 10.x
Download Zipfile Zipfile Zipfile
Size of the downloaded file 136.9 Mb 136.7 Mb 138.2 Mb
Installation mode Unzip
Size once installed on drive 140.4 Mb 161.8 Mb 163.4 Mb
How to start it $WILDFLY_HOME/bin/standalone
Location of log files $WILDFLY_HOME/standalone/log
Best startup time 1 718 ms 1 887 ms 1 989 ms
RAM at startup (later) 97 Mb (27 Mb) 112 Mb (26 Mb) 127 Mb (28 Mb)
Welcome page http://localhost:8080/
Admin console http://localhost:4848

Weblogic 12.x

If Oracle still uses GlassFish as the Java EE reference implementation, its work and support goes into Weblogic. Weblogic 12 has recently been Java EE 7 certified.

Version Weblogic 12.2.1.0.0
What do you get ? Java EE 7 Full Profile
Who is behind? Oracle
Open source? No
Documentation Documentation
Download Zipfile + installer
Size of the downloaded file 221.3 Mb
Installation mode Unzip + install
Size once installed on drive 749.9 Mb
How to start it $WLS_HOME/user_projects/domains/base_domain/startWebLogic
Location of log files %WLS_HOME/standalone/log
Best startup time 8 411 ms*
RAM at startup (later) 182 Mb (71 Mb)
Admin console http://localhost:7001/console

*Weblogic doesn’t show milliseconds in the logs so I had to change the starting script to add a few date +%s%3N.

Websphere and WebSphere Liberty 8.5.x

Today IBM has two versions of Websphere 8.5 : Classic and Liberty, both implementing Java EE 7. As for the Classic, you need to get lost in the IBM website maze and ask for help on Tweeter, to understand that you can’t install it on Mac OS X. So I just concentrated on WebSphere Liberty which is a completely different beast: download, unzip, run. There is no administration console but.

Version WebSphere Liberty 8.5.5.8 WebSphere Liberty Web 8.5.5.8
What do you get ? Java EE 7 Full Profile Java EE 7 Web Profile
Who is behind? IBM
Open source? Commercial product
Documentation Documentation
Download Zipfile Zipfile
Size of the downloaded file 99.8 Mb 67.5 Mb
Installation mode Unzip
Size once installed on drive 122 Mb 83.4 Mb
How to start it $WLP_HOME/bin/server start
Location of log files $WLP_HOME%/usr/servers/defaultServer/logs
Best startup time 3 471 ms 2 093 ms
RAM at startup (later) 265 Mb (63 Mb) 49 Mb (39 Mb)
Welcome page http://localhost:9080/

Java EE Web Profile 7 Application Servers

If GlassFish, Payara and Websphere have two different distributions (Full and Web profile), some application servers only implement the Web Profile. That’s the case of TomEE and Resin.

TomEE 7.x

Apache TomEE is the perfect success story for the Web Profile. TomEE is no more than Tomcat + OpenWebBeans + OpenEJB + OpenJPA + MyFaces + other bits. It really shows that Java EE is a jigsaw puzzle where you can take open standards, bundle them together and become a certified Web Profile application server. This TomEE version is bundled with Tomcat 8.0.29.

Version TomEE 7.0.0-M1
What do you get ? Java EE 7 Web Profile
Who is behind? Apache
Open source? Apache Software License
Documentation Documentation
Download Zip, tar.gz
Size of the downloaded file 35.5 Mb
Installation mode Unzip the file
Size once installed on drive 42.1 Mb
How to start it $TOMEE_HOME/bin/startup.sh
Location of log files $TOMEE_HOME/logs
Best startup time 505 ms
RAM at startup (later) 50 Mb (24 Mb)
Welcome page http://localhost:8080/
Admin console (*) http://localhost:8080/manager/html

(*) To be able to log on to the admin console, you need to change the $TOMCAT_HOME/conf/tomcat-users.xml configuration files and add a manager-gui role to a user.

Resin 5.x

Resin was one of the first servlet containers (such as Jetty or Tomcat) to move to the Web Profile. Again, it’s another success story that confirmed that profiles in Java EE were needed. Resin is very CDI centric and based on the Caucho implementation called CanDI.

Version Resin 5.0.2-alpha
What do you get ? Java EE 7 Web Profile
Who is behind? Caucho
Open source? GPL License
Documentation Documentation
Download Zip, tgz, deb
Size of the downloaded file 10.7 Mb
Installation mode Unzip the file
Size once installed on drive 11.7 Mb
How to start it $RESIN_HOME/bin/resin start
Location of log files /tmp/resin/log/
Best startup time 1 303 ms
RAM at startup (later) 2 processes : 61 Mb + 39 Mb
Welcome page http://localhost:8080/

Servlet 3.1 Servers

You can see Java EE 7 application servers implementing everything (the Full profile), a bit (Web profile) or just the web server portion (Servlet 3.1). That’s the case of Tomcat, Jetty or Undertow. Unfortunately I do not benchmark Undertow because it doesn’t have a standalone installation: you need to start it up in your code.

Tomcat

Tomcat has been implementing Servlet 3.1 since version 8.

Version Tomcat 8.0.30 Tomcat 9.0.0.M1
What do you get ? Servlet 3.1
Who is behind? Apache
Open source? Apache Software License
Documentation Documentation Documentation
Download Zip, tar.gz Zip, tar.gz
Size of the downloaded file 9.7 Mb 9.9 Mb
Installation mode Unzip the file
Size once installed on drive 14.6 Mb 14.5 Mb
How to start it $TOMCAT_HOME/bin/startup.bat
Location of log files $TOMCAT_HOME/logs
Best startup time 401 ms 404 ms
RAM at startup (later) 45 Mb (17 Mb) 40 Mb (15 Mb)
Welcome page http://localhost:8080/
Admin console http://localhost:8080/manager/html

Jetty

Jetty started to implement Servlet 3.1 from Jetty 9.

Version Jetty 9.3.6.v20151106
What do you get ? Servlet 3.1
Who is behind? Eclipse
Open source? Apache License 2.0 and Eclipse Public License 1.0
Documentation Documentation
Download Zip, tgz, deb
Size of the downloaded file 11.7 Mb
Installation mode Unzip the file
Size once installed on drive 13.4 Mb
How to start it $JETTY_HOME/bin/jetty.sh start
Location of log files $JETTY_HOME/logs
Best startup time 479 ms
RAM at startup (later) 52 Mb (8 Mb)
Welcome page http://localhost:8080/

Undertow

JBoss Application Server used to use Tomcat as the web container. Since version 8 it uses Undertow that can be downloaded on its own.

Version WildFly Servlet 10.0.0.Final
What do you get ? Servlet 3.1
Who is behind? RedHat
Open source? LGPL
Documentation Documentation
Download Zip (unser Servlet-Only Distribution)
Size of the downloaded file 29.2 Mb
Installation mode Unzip the file
Size once installed on drive 35.8 Mb
How to start it $WILDFLY_HOME/bin/standalone.sh
Location of log files $WILDFLY_HOME/standalone/log
Best startup time 1 252 ms
RAM at startup (later) 93 Mb (16 Mb)
Welcome page http://localhost:8080/

Summary

As a summary I will show you two graphs with startup time and memory consumption so it’s easier to compare.

Startup time

When you are a developer, startup up time matters (even when you move your app server to the Cloud). Thanks to tools like JRebel, or hot deployment in our IDEs, we tend to restart our app servers less and less. But still, it’s important to have fast startup time. Something to stress out, is that most of the app servers bootstrap the minimum set of services and then, lazy load services when needed.

From left to right we have Full profile, Web profile and Servlet containers. Of course, Servlet containers start faster. But it’s interesting to notice that most app servers boot in less that 3 seconds, and half, in less than 2 seconds (remember that these measures are taken on a Mac OS X 16Gb RAMP SSD). Weblogic is by far the slowest (> 8 seconds) and WildFly the fastest of the Java EE Full profile app servers. TomEE 7 (Web profile) is slightly slower than Tomcat (Servlet).

Memory Footprint

The memory consumption is also important. In this era of Micro Services, we want fast startup time and low resource consumption. Without any JVM or server twists, most app servers use less than 150Mb of RAM. Here I take the measure at startup, perform a GC and wait a few seconds so it stabilize. So you get a min and a max memory usage.

Conclusion

First of all, as you can see there is no Websphere Classic. Despite being a big player, it is difficult to find, difficult to install, and has no Mac OS X support. I hope one day I’ll be able to add it to the benchmark.

What did I want to show in this blog ? Well, that application servers have changed. Most app servers take few resources, are modular and have nice tricks to start up fast (like lazy loading services).

And why did I want to show this ? Just to show that the “Tomcat is light, Java EE app servers are not” line is outdated. Use the app server that fits your need. You need Servlet ? Go for Jetty or Tomcat. You need some JAX-RS with CDI and JPA ? Choose TomEE, WildFly or Websphere Liberty. You need the full monty ? Use WildFly, Weblogic or GlassFish. You need production support ? Use JBoss EAP (you’ll have to wait a few extra month). You need to create Micro Services ? Wait for my next blog ;o)

References

 

 

 

 

Categories: Java

13 Comments »

  1. Great article. Good to see that there are no “winners” in this test. Just the conclusion that you use what you like:)

    Just a small correction. The link to documentation in WebSphere Liberty 8.5.x
    seems to point to Wildfly.

  2. Nice! BTW, I think would be more fair put the “Java EE Web Profile” servers in a separate startup time graph. Is’t it?

  3. WebSphere V8.5.5 is still only Java EE 6. WebSphere Liberty may contain further enhancements towards Java EE 7 but is not certified. We still wait for something like WebSphere V9 + Liberty, Java EE 7 compliant…

  4. I suppose I can help by making some comments for EAP/WildFly:

    – EAP comes with higher default memory settings (-Xmx, etc.) out of the box, since it targets primarily production environments. E.g. check the diff in /bin/standalone.conf between EAP & WildFly.

    – Although you didn’t really do any application benchmark tests, EAP 7 Beta is based on a older fork of WildFly 10, after which we’ve done a large number of performance improvements. When the EAP7 GA release comes out, its overall performance numbers should be in line with WildFly 10, if not better.

    – WildFly/EAP does *some* lazy loading but not too much, like other containers. With WildFly 10 about 50% of services are actually started when the boot sequence is completed, thus when real application are deployed you won’t get the hit of having to start those services.

    – When comparing servlet container, you may just as well consider the “Servlet-Only Distribution” on the WildFly download page. This will give you essentially Undertow with deployment capability like Tomcat & Jetty so you can compare.

    Thanks for sharing this article!

  5. When you start WildFly with “$WILDFLY_HOME/bin/standalone”, it’s not JavaEE Full Profile because it hasn’t JMS. add “-c standalone-full.xml” to be full profile compliant.

    Regarding Dimitris’ comment on lazyness, a bench with a small app containing a servlet, an EJB, a small JPA entity,… would be more relevant.

  6. WebLogic requires some level of entropy during startup (and domain creation), so it makes use of the entropy pool in Linux-based environments. By default, WebLogic uses whatever is defined in the JVM java.security file (which is /dev/random, a device that blocks the process until enough random bits are provided).

    To speedup WebLogic on Linux-based development environments, developers should configure their environments to use /dev/urandom. This can be done in different ways:

    – Modify java.security
    – Modify setDomainEnv.sh
    – set USER_MEM_ARGS temporarily

    You can find more details in this blog: http://theheat.dk/blog/?p=1539

    I strongly recommend, Antonio, that you do adjust your JVM java.security file to use /dev/urandom by default, and then rerun your benchmarks.

  7. A second note on WebLogic:

    – it has two distributions: one called “Generic”, and one called “Quick Installer”. The distribution used in this benchmark, based on the size of the download, is the Generic installer. The quick installer is smaller (~250MB) because it is compressed with pack200.
    – both distributions are able to create a domain in Development Mode
    – Development Mode deploys the Admin Console by default
    – Production Mode deploys Admin Console on first access

    So, the benchmark could also state whether the application server startup time includes the startup of the console or not.

  8. Two items to also think about with WAS Liberty:
    1) While you are testing with the Web and Full Profile, you do not need to load all of it. It is a fit for feature/function architecture. Just need Servlet and JDBC, that is all that you need to start.
    2) Once you start it, there are only two reasons you need to restart it. 1) JVM changes. 2) you want to. All configuration and app changes are dynamically picked up. Because the command line arguments for a JVM are not dynamic, this is not possible for Liberty resolve.

    For your microservices article, please see how the server package –include=minify feature works on size of server package and usage with microservices.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s