TomEE on CloudFoundry

Deploy TomEE on CloudFoundry is not (yet?) proposed out of the box but it is not so complicated to make it real!

Preparing your TomEE Instance

CloudFoundry proposes a framework called standalone which can be used with the java runtime. This mainly means you can do what you want in java with such a configuration.

We’ll use it to create a configured TomEE locally then deploy the whole TomEE.

Create the TomEE

To ease all phases i’ll use Maven. So to create the TomEE i’ll use the tomee-maven-plugin:

<plugin>
  <groupId>org.apache.openejb.maven</groupId>
  <artifactId>tomee-maven-plugin</artifactId>
  <version>1.0.1-SNAPSHOT</version>
  <executions>
    <execution>
      <id>package-tomee</id>
      <phase>package</phase>
      <goals>
        <goal>build</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <zip>false</zip>
    <keepServerXmlAsthis>true</keepServerXmlAsthis>
    <tomeeClassifier>jaxrs</tomeeClassifier>
    <catalinaBase>${cf.builddir}</catalinaBase>
    <removeTomeeWebapp>true</removeTomeeWebapp>
    <warFile>${project.build.directory}/ROOT</warFile>
    <libs>
      <lib>org.cloudfoundry:cloudfoundry-runtime:0.8.2</lib>
      <lib>org.codehaus.jackson:jackson-mapper-asl:1.4.1</lib>
      <lib>org.codehaus.jackson:jackson-core-asl:1.4.1</lib>
    </libs>
  </configuration>
</plugin>

So first i say to the plugin to not zip the tomee (we want it to be exploded) – this option is available on the snapshot but not on the previous version which wasn’t doing any zip. Then i ask to keep the server.xml file i provide (we’ll see soon why). Finally i use the <finalName>ROOT</finalName> and i point on it in <warFile> tag to use the exploded archive instead of the war (not very important for the deployment). Finally i add to the container jackson and cloudfoundry-runtime to be able to get some information from cloudfoundry when deploying (Note: this could have been done without these libs but it avoids to rewrite some code so for a quickstart it is perfect).

Configuration files

We have 3 files to provides:

1. server.xml: cloudfoundry kills the JVM to stop it so the shutdown port should not be used (should be set to -1 – otherwise it can prevent scaling since fixed port will be used)
2. we want to block the JVM when starting so we have to update startup.sh to use run instead of start
3. we want to use the dynamic port allocated by cloudfoundry (to be well load-balanced) so we need to set it as system property when launching the TomEE JVM then re-use it in server.xml

Here are the files. First the server.xml (src/main/tomee/conf):

<?xml version='1.0' encoding='UTF-8'?>
<Server port="-1">
  <Listener className="org.apache.tomee.catalina.ServerListener" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <Service name="Catalina">
    <Connector port="${tomee.cf.http}" protocol="HTTP/1.1"
               connectionTimeout="20000"  />
    <Engine name="Catalina" defaultHost="localhost">
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true" />
    </Engine>
  </Service>
</Server>

As you can see the http port is dynamically set to a value and the shutdown port is -1.

Then the startup.sh (src/main/tomee/bin):

#!/bin/sh

# resolve links - $0 may be a softlink
PRG="$0"

while [ -h "$PRG" ] ; do
  ls=`ls -ld "$PRG"`
  link=`expr "$ls" : '.*-> \(.*\)$'`
  if expr "$link" : '/.*' > /dev/null; then
    PRG="$link"
  else
    PRG=`dirname "$PRG"`/"$link"
  fi
done

PRGDIR=`dirname "$PRG"`
EXECUTABLE=catalina.sh

# use run and not startup to let cloudfoundry kill it at stop command
exec "$PRGDIR"/"$EXECUTABLE" run "$@"

It is the TomEE one excepted for the last line (and remove some environment dependent lines to be more readable but the real change is just this run at the end).

And finally the setenv.sh to get the dynamic port (src/main/tomee/bin):

#! /bin/bash

if [ -z ${VCAP_APP_PORT} ]; then
    export VCAP_APP_PORT=8080
fi

export JAVA_OPTS="-Dtomee.cf.http=$VCAP_APP_PORT $JAVA_OPTS"

The “if” is here only to be able to run the server locally (such a trick could be used for the shutdown port too ;)).

Now running the following command:

mvn clean package tomee:run

We’ll get a TomEE with our war (if the current project is a war ;)).

Now we just need to deploy it to cloudfoundry.

Deploying to CloudFoundry

Cloudfoundry propose a maven plugin so we can deploy our TomEE pretty quickly:

<plugin>
  <groupId>org.cloudfoundry</groupId>
  <artifactId>cf-maven-plugin</artifactId>
  <version>1.0.0.M2</version>
  <configuration>
    <command>bin/startup.sh</command> <!-- we'll use our custom startup.sh command to start TomEE -->
    <runtime>java</runtime>
    <framework>standalone</framework>
    <memory>256</memory>
    <path>${cf.builddir}</path>
    <url>${cf.url}</url>
    <target>${cf.api}</target>
    <server>cloudfoundry</server> <!-- matches credentials in settings.xml, see below -->
    <services>${cf.services}</services>
    <appname>demo-tomee</appname>
    <!-- server needs
     <server>
      <id>cloudfoundry</id>
      <username>xxxx</username>
      <password>yyyyy</password>
    </server>
    -->
  </configuration>
</plugin>

Now to push your application simply run:

mvn cf:push

Or to update it:

mvn cf:update

Services?

Ok but now i added a MySQL service how do i manage it? First you can define your DataSource in your application in resources.xml but if you prefer add mysql to TomEE libs and define it in tomee.xml (src/main/tomee/conf) it will work too, it is up to you.

But the question is: how to get the password, username, url…?

That’s why we needed to add jackson and cloudfoundry-runtime librariries: we’ll use a hook which will be executed before the application is deployed (why the libs are in the container and not the application is because the application classloader doesn’t exist at this moment). We’ll simply get the environment variable cloudfoundry set with all the services configuations then set the configurations as system properties:

package bar;

import org.apache.openejb.observer.Observes;
import org.apache.openejb.observer.event.ObserverAdded;
import org.cloudfoundry.runtime.env.CloudEnvironment;
import org.cloudfoundry.runtime.env.RdbmsServiceInfo;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Properties;

public class CloudInitializer {
    private static final boolean DEBUG = Boolean.getBoolean("tomee.cloudfoundry.debug");

    private static boolean init = false;

    /**
     * replace cloudfoundry runtime (see CloudEnvironment and PropertySetter)
     */
    public void onStartup(@Observes final ObserverAdded observer) {
        if (init) {
            return;
        }

        if (isCloudFoundry()) {
            try {
                final CloudEnvironment cloudEnvironment = new CloudEnvironment();

                // light version of PropertySetter
                final Properties cloudProperties = cloudEnvironment.getCloudProperties();
                final List<String> keys = new ArrayList<String>(cloudProperties.stringPropertyNames());
                Collections.sort(keys);
                for (String key : keys) {
                    setProperty(key, cloudProperties.getProperty(key));
                }

                // set urls to be able to use it in resources.xml
                final List<RdbmsServiceInfo> dbservices = cloudEnvironment.getServiceInfos(RdbmsServiceInfo.class);
                for (RdbmsServiceInfo service : dbservices) {
                    setProperty("cloud.services." + service.getServiceName() + ".connection.url", service.getUrl());
                }
            } catch (Exception e) {
                e.printStackTrace(System.err);
            }
        } else { // mock for local environment
            mock();
        }

        init = true;
    }

    private static boolean isCloudFoundry() {
        return System.getenv("VCAP_APPLICATION") != null;
    }

    private static void mock() {
        final Properties properties = new Properties();
        InputStream is = null;
        try {
            is = CloudInitializer.class.getResourceAsStream("/cloud-local.properties");
            final File file = new File("cloud-local.properties");
            if (is == null && file.exists()) {
                is = new FileInputStream(file);
            }
            properties.load(is);
            System.getProperties().putAll(properties);
        } catch (IOException e) {
            throw new RuntimeException(e);
        } finally {
            if (is != null) {
                try {
                    is.close();
                } catch (IOException e) {
                    // no-op
                }
            }
        }
    }

    private static void setProperty(final String key, final String value) {
        System.setProperty(key, value);
        if (DEBUG) {
            System.out.println("[CONFIG] " + key + " = " + value);
        }
    }
}

Once this is done simply use these system properties in your resources.xml. If your service is called tomee-mysql:

<?xml version='1.0' encoding='utf-8'?>
<resources>
  <Resource id="jdbc/cf" type="DataSource">
    JdbcUrl = ${cloud.services.tomee-mysql.connection.url}
    UserName = ${cloud.services.tomee-mysql.connection.user}
    Password = ${cloud.services.tomee-mysql.connection.password}
    JdbcDriver = com.mysql.jdbc.Driver
    JtaManaged = true
  </Resource>
</resources>

Finally you can provide in your war a META-INF/context.xml file to stop the JVM with application undeployment:

<?xml version='1.0' encoding='utf-8'?>
<Context>
  <Listener className="com.vmware.appcloud.tomcat.AppCloudLifecycleListener" />
</Context>

Conclusion

Here we are: we are able to deploy a TomEE on CloudFoundry with a single command and get CloudFoundry services very easily.

Once you deployed your application you get CloudFoundry strength. For instance to scale simply run vmc instance <your app> 10 and you’ll get 10 instances well load-balanced :).