Tag Archives: openejb

BatchEE + ApplicationComposer = Batch.main(args)


BatchEE + ApplicationComposer = Batch.main(args)

Batches are quite particular as application cause you have generally the choice with keeping a JVM running and using a scheduler to trigger them or to just use an external scheduler to trigger the JVM and once the batch is done just exit the process.

The choice can depend on several criteria but most of the time in medium/big companies this choice is in place when you need to write a batch you just need to respect it.

Where having a long time running container is not different than any application, writing a main can be quite challenging.

Let’s see how OpenEJB (TomEE) and BatchEE makes it easy.

BatchEE is a JBatch implementation but provides a lot of tooling on top of JBatch like a management GUI, some maven plugins, a lot of extensions/components (csv, flr, json readers, chained processors, typed components etc…) and much more. One of the proposed tooling is to setup BatchEE “as” a container and run a web application (war) or a batch application (bar, format created by BatchEE dedicated to batches). This solution is nice since it is integrated with some well known containers like OpenEJB, Spring or any CDI embedded container (OpenWebBeans, Weld).

However the pitfalls of such a solution is to suppose you can deploy a “batch container” – BatchEE standalone – and your application. This is not always possible and the “standard” for your company can be to just deploy a jar.

The simplest solution is to write a (J)Batch relying only on SE features and do a shade but then you loose all the EE programming model which is very beneficial to write profesionnal and maintainable batches.

To be able to mix both you need to be able to wrap your batch execution once a container is launched.

Using the whole BatchEE ecosystem there are several solutions but a simple one I used several times with success is to use OpenEJB/TomEE ApplicationComposer to handle the container integration and just BatchEE Batches helper to wait for the end of the execution. One nice bonus with ApplicationComposer is that you fully control the classes part of your application which often allows to make your batch deployment lighter.

For our example we will suppose we have a batch with a single step which is a chunk. It will have the following chain:

fileReader -> entryProcessor -> httpWriter

Of course all these components are CDI beans.

Step 1: define your application

The first step is to define an ApplicationComposer model. This post is not about ApplicationComposer itself so let’s make it simple and define it like that:

public class SampleApplication {
    @Module
    @Classes(cdi = true, value = {
        FileReader.class, EntryProcessor.class, HttpWriter.class, HttpClient.class
    })
    public WebApp application() {
        return new WebApp();
    }
}

Of course in previous sample HttpWriter get injected HttpClient so if you need more CDI classes you need to register them (or use @Jars/@Default).

Step 2: execute your batch

Executing a batch is as simple as obtaining a JobOperator, starting your batch and waiting for its end.

If you use BatchEE, the main module provides Batches class which has this feature. The implementation either uses batchee internal to wait for the end or use a simple polling on the batch status.

In practise it means starting and waiting for the end of the batch looks like:

protected JobExecution runBatch(final String batchName, final Properties config) {
    final JobOperator operator = BatchRuntime.getJobOperator();
    final long id = operator.start(batchName, config);
    Batches.waitForEnd(operator, id);
    return operator.getJobExecution(id);
}

Step 3: ensure BatchEE is initialized properly

I already spoke about how to enforce BatchEE BeanManager initialization properly on this blog but in fully embedded mode and with OpenEJB 4.x where BatchEE is not integrated (as opposed to OpenEJB/TomEE 7) we need another step: enforce the initialization of the BeanManager and the TransactionSynchronizationRegistry.

Note: of course if you don’t care about JTA you only need the BeanManager integration.

The easiest is to use a @Singleton to do so:

@Startup
@Singleton // batchee spawns its own thread so ensure EE is init in an EJB with all EE instances bound
public class BatchEEBeanManagerInitializer {
    @PostConstruct
    private void forceInit() {
        try {
            // enfore init of jta tx registry in this ejb context
            Class.forName(Synchronizations.class.getName(), true, Thread.currentThread().getContextClassLoader());

            // enfore cdi bean manager capture
            ServicesManager.find().service(BatchArtifactFactory.class).load("init");
        } catch (final Exception e) {
            throw new IllegalStateException(e);
        }
    }

    @Named
    protected static class Init {
    }
}

Of course we need to add this class in our SampleApplication.

Step 4: handle the container lifecycle

So we have BatchEE properly setup, our application defined and our batch ready to be executed so we just miss a way to start the container. This is where ApplicationComposer does its job:

final SampleApplication app = new SampleApplication();
new ApplicationComposers(app).evaluate(app, () -> {
    // run your batch here
});

And here we are, we have all the steps to do our shade!

Step 5: wait we need a main!

When you have a single batch you can put it all together in a main(String[]) but it happens quite regularly to put several batches in the same “application” because they are close in term of code or just linked in term of lifecycle.

In this case using a plain main means handling the batch selection yourself. It also makes the wiring of the batch parameters harder – as you can guess since my chunk starts reading a file the file path will likely be a parameter you can pass to the command line.

Said more concretely it means the need of a “CLI” library is quickly obvious. Of course and without surprise I will use Tomitribe Crest but Args4j, Airline for instance are very good alternatives if you are used to them.

The nice thing putting all your launchers in a “command” class is you can share the reporting of the batches (it is important to report the execution time, batch status etc…). Also if your scheduling tool uses exit code to determine if the execution was successful or not it is a good place to share the exit code logic.

Here is what it looks like with Crest for our ApplicationComposer BatchEE batch:

public class Batch {
    @Command
    public static void sample(@Option("input") final String input,
                              @Out final PrintStream printStream) {
        of(new SampleApplication()).ifPresent(a -> run(printStream, a, () -> a.execute(input)));
    }

    private static void run(final PrintStream printStream, final Object app, final Supplier<JobExecution> report) {
        try {
            new ApplicationComposers(app).evaluate(app, () -> { // in container
                final JobExecution execution = report.get(); // run the batch

                // log the execution
                final String executionString = "Job name: " + execution.getJobName() + lineSeparator() +
                    "Execution Id: " + execution.getExecutionId() + lineSeparator() +
                    "Create Time: " + execution.getCreateTime() + lineSeparator() +
                    "Execution Time: " + TimeUnit.MILLISECONDS.toSeconds(
                    of(execution).map(JobExecution::getEndTime).map(Date::getTime).orElse(0L) - execution.getCreateTime().getTime()) + "s" + lineSeparator() +
                    "Exit Status: " + execution.getExitStatus() + lineSeparator() +
                    "Batch Status: " + execution.getBatchStatus();
                printStream.println(executionString);

                // exit status will be -100 if status is failing, -1 if an unexpected error occured and 0 if it succeed
                if (execution.getBatchStatus() != BatchStatus.COMPLETED) {
                    throw new FailedExecution(executionString, execution);
                }

                return null;
            });
        } catch (final RuntimeException re) {
            throw re;
        } catch (final Exception e) {
            throw new IllegalStateException(e);
        }
    }

    @Exit(-100)
    @Getter
    public static class FailedExecution extends CommandFailedException {
        private final JobExecution execution;

        public FailedExecution(final String executionString, final JobExecution execution) {
            super(new IllegalStateException(lineSeparator() + executionString), execution.getJobName());
            this.execution = execution;
        }
    }
}

Things to note there:

  • Exit code handling uses @Exit handling of crest
  • we added an execute method to our application which is responsible to wire command parameter to the batch
  • run method is generic for any batch command and handles the reporting of the batch execution and makes it returning -100 exit code if the batch failed

Side note on exit status: on some UNIx system using negative exit codes make it moving to the positive side cause it uses unsigned int (-100 becomes 156 for instance).

For completeness here is the SampleApplication with the full code:

public class SampleApplication {
    @Module
    @Classes(cdi = true, value = {
        BatchEEBeanManagerInitializer.class,
        FileReader.class, EntryProcessor.class, HttpWriter.class, HttpClient.class
    })
    public WebApp application() {
        return new WebApp();
    }

    public JobExecution execute(final String input) {
        return runBatch("sample", new Properties() {{
            setProperty("input", input);
        }});
    }

    protected JobExecution runBatch(final String batchName, final Properties config) {
        final JobOperator operator = BatchRuntime.getJobOperator();
        final long id = operator.start(batchName, config);
        Batches.waitForEnd(operator, id);
        return operator.getJobExecution(id);
    }
}

Note: of course since we know we execute batches we can move runBatch to be abstracted and use @Batch(name = “sample”) or something like that.

But wait, the part was about to get a main and we just got a command? Yes cause now our main can be Crest one: org.tomitribe.crest.Main.

Step 6: the big final, create the shade

Now we just need to shade it all together. Nothing very special here excepted we need to take care of java SPI (META-INF/services/*), of openwebbeans.properties files which need to be merged, to set the right main (Crest one) and if you use several BacthEE extensions with the BatchEE shortnames to merge batchee.xml:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>2.4.2</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <createDependencyReducedPom>false</createDependencyReducedPom>
        <shadedClassifierName>bundle</shadedClassifierName>
        <shadedArtifactAttached>true</shadedArtifactAttached>
        <transformers>
          <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
          <transformer implementation="org.apache.maven.plugins.shade.resource.XmlAppendingTransformer">
            <resource>META-INF/batchee.xml</resource>
          </transformer>
          <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
            <resource>META-INF/openwebbeans/openwebbeans.properties</resource>
          </transformer>
          <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
            <mainClass>org.tomitribe.crest.Main</mainClass>
          </transformer>
        </transformers>
      </configuration>
    </execution>
  </executions>
</plugin>

Then once built you will get a artifact-version-bundle.jar and you can execute your batch using:

java -jar artifact-version-bundle.jar sample --input=/opt/applications/sample/input/data.csv

Looks a lot of steps maybe but if you step back it is the main trick is to wait for the batch before the end of the execution. A nice bonus is to wire BatchEE persistence to a database to be able to deploy somewhere else BatchEE administration interface to be able to follow batch executions graphically.

Last thing before closing this post, since all the model is based on ApplicationComposer, testing these batches is as easy as using ApplicationComposerRule or runner :).

Happy (thanks)batching!

Advertisements

ApplicationComposer Maven Plugin


Even if it comes from test side of OpenEJB, ApplicationComposer is now a solution to develop microservices. Because of this evolution it now gets a maven plugin to be able to go further in the development pipeline.

Continue reading

OpenEJB 5 + Groovy = service done fast!


If you read the previous post I wrote about ApplicationComposers run method you surely wonder what a service could look like using groovy and @Grab to resolve maven dependencies?

Here is a quick post showing it:

Continue reading

Did you ever realize OpenEJB ApplicationComposers is the EE Spring boot?


I spoke a lot of ApplicationComposers on this blog. If you don’t know it yet in two words it is originally a way to fully define its EE application programmatically and then run test against this described application.

However with a trivial main you can make it run your application!

Continue reading

Rule Them All meets PhantomJS to test JAX-RS javascript client generation


HTMLUnit is great to test web applications but sadly it doesn’t support recent js libraries (AngularJs/JQuery to not say their names).

PhantomJs is a great alternative but sometimes setup can be boring or too complicated for what you need.

Continue reading

CDI and ApplicationComposer: finally easy to test deltaspike with AppComposer


OpenEJB/TomEE ApplicationComposer design is to describe a test application with an in memory model close to the xml EE one (Beans for beans.xml, EjbJar for ejb-jar.xml…).

But when it comes to CDI and since it was mainly designed for small apps it is hard to say “I have a CDI dependency/library”.

The more obvious use case is DeltaSpike!

To make it easier we added @Jars recently.

Continue reading

TomEE and more advanced resource configuration


TomEE 2 got two new interesting features (in particular in enterprises) to configure resources: default placeholder values and properties-provider.

Continue reading