TECHNOLOGY

Enterprise Integration Framework:

Communication between different external systems are increasing a lot to serve the specific business need. The number of applications that must be integrated is increasing, too. There has to be some way by which the integration of these applications can be modeled in a standardized way, realized efficiently and supported by automatic tests. To address this need, Integration Framework were developed. It provides set of standards that supports the interoperability between applications build on different technologies, protocols and data formats.

Approaches for Integrating Systems:

There are three approaches that can be used for integrating applications. EIPs can be used in each approach.

Approach 1: Own custom Solution

Implement an individual solution that works for your problem without separating problems into little pieces. This works and is probably the fastest alternative for small use cases. You have to code all by yourself.

Approach 2: Integration Framework

Use a framework which helps to integrate applications in a standardized way using several integration patterns. It reduces efforts a lot. Every developer will easily understand what you did (if he knows the used framework).

Approach 3: Enterprise Service Bus (ESB)

Use an enterprise service bus to integrate your applications. Under the hood, the ESB also uses an integration framework. But there is much more functionality, such as business process management, a registry or business activity monitoring. You can usually configure routing and such stuff within a graphical user interface – you have to decide at your own if that reduces complexity and efforts. Usually, an ESB is a complex product. The learning curve is much higher. But therefore you get a very powerful tool which should offer all your needs.

Comparison Criteria

Following criteria can be taken into account while choosing the right framework for your business need:

Open source
Basic concepts / architecture
Testability
Deployment
Popularity
Commercial support
IDE-Support
Error handling
Monitoring
Enterprise readiness
Domain specific language (DSL)
Number of components for interfaces, technologies and protocols
Expandability

There are a lot of open source and commercial integration frameworks available in market today. Some of the leading popular open source frameworks include Mule ESB Fuse ESB, Apache Camel and Spring Integration. They all implement well-known Enterprise Integration Patterns and therefore offer a standardized, domain-specific language to integrate applications.

When to use Spring Integration or Custom Solution?

For simple integration needs, either your own custom solution can be or spring integration can be a good choice.

When to use Apache Camel?

Apache Camel is the best integration framework if you want to integrate several applications with different protocols and technologies. Main advantage of using Apache Camel is that Every integration uses the same concepts! No matter which protocol, which technology, which domain specific language (DSL) you use - it can be Java, Scala, Groovy or Spring XML. You do it the same way. Always!

Two other very important features are its support for error-handling and automatic testing. You can test EVERYTHING very easily using a Camel-extension of JUnit! And again, you always use the same concepts, no matter which technology you have to support. Apache Camel is mature and production ready. It offers scalability, transaction support, concurrency and monitoring. Commercial support is available by FuseSource: http://fusesource.com/products/enterprise-camel

ESB:

An ESB is the right tool for very large integration project. It offers many additional features such as BPM or BAM. Several production-ready ESBs are also available. Usually, open source solutions are more lightweight than commercial products such as WebSphere Message Broker (you probably need a day or two just to install the evaluation version of this product)! Well-known open source ESBs are Apache ServiceMix, Mule ESB and WSO2 ESB. There are some ESB based on the Apache Camel framework (e.g. Apache Service Mix and the Talend ESB). Thus, if you like Apache Camel, you could also use Apache ServiceMix or the commercial Fuse ESB which is based on ServiceMix.

Features of Camel:

Routing and Mediation Engine:

The core feature of Camel is its routing and mediation engine. It allows you to decide from which sources to accept messages, and determine how to process and send those messages to other destinations based on the routing rules you have defined.

Enterprise Integration Patterns (EIPS):

EIPs are helpful not only because they provide a proven solution for a given problem, but also because they help us to define and communicate the problem itself. Patterns have known semantics, which makes communicating problems much easier. Camel is heavily based on EIPs. Although EIPs describe integration problems and solutions and also provide a common vocabulary, the vocabulary isn’t formalized. Camel tries to close this gap by providing a language to describe the integration solutions. There’s almost a one-to-one relationship between the patterns described in EIP and the Camel DSL.

Domain-Specific Language (DSL):

Camel uses a Java Domain Specific Language or DSL for creating Enterprise Integration Patterns or Routes in a variety of domain-specific languages (DSL) as listed below.

Java DSL - A Java based DSL using the fluent builder style.
Spring XML - A XML based DSL in Spring XML files
Blueprint XML - A XML based DSL in OSGi Blueprint XML files
Groovy DSL - A Groovy based DSL using Groovy programming language
Scala DSL - A Scala based DSL using Scala programming language
Annotation DSL - Use annotations in Java beans.

Examples of DSL:

Java DSL

from("file:data/inbox").to("jms:queue:order");

Spring DSL

<route>

</route>

Scala DSL

from "file:data/inbox" -> "jms:queue:order"

Camel also allow us to use combination of any of these DSL’s depending on the use case.

Extensive Component Library:

Camel provides an extensive library of around 120 components. These components allow you to bridge to many different APIs, protocols, data formats, and so on. Camel saves you from having to code these integrations yourself, thus it achieves its primary goal of making integration easier.

Payload-Agnostic Router:

Camel can route any kind of payload. You aren’t restricted to carrying XML payloads. This means that you don’t have to transform your payload into a canonical format to facilitate routing.

Modular and Pluggable Architecture:

Camel has a modular architecture, which allows any component to be loaded into Camel, regardless of whether the component ships with Camel, is from a third party, or is your own custom created component.

POJO Model:

Camel understands the POJO programming model. Camel allows you to use bean anywhere anytime in your integration project. By using beans, you get an advantage of reduce coupling. Camel not only offers reduced coupling with beans, but you get the same loose coupling with Camel routes.

Easy Configuration:

The convention over configuration paradigm is followed whenever possible, which minimizes configuration requirements. In order to configure endpoints directly in routes, Camel uses an easy and intuitive URI configuration.

For example, Configuration of a file consumer to scan recursively in a subfolder and include only a .txt file, as follows:

from("file:data/inbox?recursive=true&include=*.txt")

Automatic Type Converters:

Camel has support for around 150 built-in type-converter that automatically converts between well-known types. This system allows Camel components to easily work together without having any type mismatches.

Example:

String custom = exchange.getIn().getBody(String.class);

The getBody method is passed the type you want to have returned. Under the hood, type-converter system converts the returned type to a String if needed.

Camel also allows you to create your own custom type convertor.

Lightweight Core:

Camel’s core library is pretty lightweight, with about 1.6 MB and only have a dependency on Apache Commons Logging and Fuse-Source Commons Management. This makes Camel easy to embed or deploy anywhere you like, such as in a standalone application, web application, Spring application, Java EE application, JBI container, OSGi bundle, Java Web Start, or on the Google App engine. Camel was designed not to be a server or ESB but instead to be embedded in whatever platform you choose.

Test Kit:

Camel provides rich set of testing API easier for you to test your own Camel applications. Test Kit contains test-specific components that, for example, can help you mock real endpoints. It also contains setup expectations that Camel can use to determine whether an application satisfied the requirements or failed.

Camel’s architecture:

Let’s now take a look at high level architecture of camel:

CamelContext:

The CamelContext is the runtime system of Apache Camel and connects its different concepts such as routes, components or endpoints. The CamelContext provides access to many useful services, the most notable being components, type converters, a registry, endpoints, routes, data formats, and languages. CamelContext is started when loading the application and stopped at its shutdown.

Routing Engine:

Camel’s routing engine is what actually moves messages under the hood. This engine isn’t exposed to the developer, but you should be aware that it’s there and that it does all the heavy lifting, ensuring that messages are routed properly.

Routes:

Routes are a crucial part of Apache Camel. The flow and logic of an integration is specified here. Each route in Camel has a unique identifier that’s used for logging, debugging, monitoring, and starting and stopping routes. Routes also have exactly one input source for messages, so they’re effectively tied to an input endpoint. Routes are defined using one of Camel’s domain-specific languages (DSLs).

Domain-Specific Language (DSL):

To wire processors and endpoints together to form routes, Camel defines a DSL. Camel provides support for multiple DSL languages.

Processor:

Processors are used to transform and manipulate messages during routing and also to implement all the EIP patterns, which have corresponding keywords in the DSL languages. Processor is a simple Java interface with one single method: process. Inside this method, you can do whatever you need to solve your integration problem such as transform the incoming message, call other services, and so on.

Example:

public class LoggingProcessor implements Processor {

@Override

public void process(Exchange exchange) throws Exception {

System.out.println("Received Order: " +exchange.getIn().getBody(String.class));

}

The exchange parameter contains the Messsage Exchange with the incoming message, the outgoing message, and other information. Due to implementing the Processor interface, you have got a dependency to the Camel API. This might be a problem sometimes. Maybe you already have got existing integration code which cannot be changed (i.e. you cannot implement the Processor interface) In this case, you can use Beans, also called POJOs (Plain Old Java Object). You get the incoming message (which is the parameter of the method) and return an outgoing message, as shown in the following snipped:

public class TransformationBean {

public String makeUpperCase(String body) {

String transformedBody = body.toUpperCase();

return transformedBody;

Component:

Components are the main extension point in Camel. There are over 80+ components in the Camel ecosystem that range in function from data transports, to DSLs, data formats, and so on. You can even create your own components for Camel. The most amazing feature of Apache Camel is its uniformity. All components use the same syntax and concepts. Every integration and even its automatic unit tests look the same. Thus, complexity is reduced a lot.

For Example:

File component in DSL route looks like:

<to uri=" file:target/outbox"/>

Now I want to use JMS component:

<to uri=" jms:queue:orders"/>

Content-based router:

Content-Based Router (CBR) is a message router that routes a message to a destination based on its content. The content could be a message header, the payload data type, part of the payload itself. Camel also has support for conditional routing. Keywords choice, when, to can be used in the DSL.

The choice method creates a CBR processor, and conditions are added by following choice with a combination of a when method and a predicate.

from("jms:incomingOrders")

.choice()

.when(predicate)

.to("jms:xmlOrders")

.when(predicate)

.to("jms:csvOrders");

You may have noticed that we didn’t fill in the predicates required for each when method. A predicate in Camel is a simple interface that only has a matches method:

public interface Predicate {

boolean matches(Exchange exchange);

}

For example, you can think of a predicate as a Boolean condition in a Java if statement.

Message Filters:

A Message Filter allows you to filter out uninteresting messages based on some condition. Incoming messages only pass through the filter if a certain condition is met. Messages failing the condition will be dropped.

Example of JAVA DSL:

from("jms:xmlOrders").filter(xpath("/order[not(@test)]"))

.process(new Processor() {

public void process(Exchange exchange) throws Exception {

System.out.println("Received XML order: "

+ exchange.getIn().getHeader("CamelFileName"));

}

});

In the above example, suppose the orders coming from client are in xml and is placed in jms queue named “xmlOrders”. We have usedv XPath expressions for creating conditions based on XML payloads. The orders with extra test attribute set will be filtered out. The xpath expression will evaluate true for orders that don’t have the test attribute. Thus only orders which does not contain test attribute will be send to processing in processor.

Spring DSL for the above message filter example will look like:

<route>

<xpath>/order[not(@test)]</xpath>

</filter>

</route>

Here we have reference to the processor as orderLogger, which is defined as a bean entry in the Spring XML file.

Calling web services using Apache Camel:

Now I will walk you through an example, in which I have used apache camel to invoke remote services hosted on server.

For accessing and publishing web services, Camel uses Apache CXF. CXF is a popular web services framework that supports many web services standards.

In this example, I am using a maven build project. You need to add following maven dependencies in pom.xml to use apache camel libraries in your project.

<groupId>org.apache.camel</groupId>

<artifactId>camel-core</artifactId>

</dependency>

<groupId>org.apache.camel</groupId>

<artifactId>camel-spring</artifactId>

</dependency>

<groupId>org.apache.camel</groupId>

<artifactId>camel-cxf</artifactId>

</dependency>

In order to use the CXF component, we have some camel-cxf dependency

Configuring CXF Using a Cxf Endpoint Bean:

<cxf:cxfEndpoint id="routeOrderServiceEntryPoint"

address="/services/routeOrderServiceEntryPoint"

serviceClass="com.impetus.sei.OrderServiceEntryPoint" >

</cxf:cxfEndpoint>

id: bean id

address: URL where the endpoint is exposed

serviceClass: SEI

Creating routes with Spring:

Before creating a route, we need a camel context.

<beans xmlns="http://www.springframework.org/schema/beans"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="

http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans-3.0.xsd

http://camel.apache.org/schema/spring

http://camel.apache.org/schema/spring/camel-spring.xsd">

...

</beans>

Above code snippet will automatically start the camel context. When you start Camel, it creates a CamelContext object that holds many information on how to run it, including the definition of the Route we created. Now we need to define routes in camel context.

<route>

<camel:from uri="cxf:bean:routeOrderServiceEntryPoint" />

<camel:to uri="bean:orderCXFServiceClient?method=processOrder"/>

<camel:to uri="bean:orderProcessResponse"/>

</route>

Route is define in spring dsl using <route /> tag. We can have multiple routes in the camel context.

I have referenced the cxf endpoint in route as:

I have to define the cxf client, which will be used to invoke the remote cxf services:

</bean>

orderProcessResponse bean used in camel route refer to below OrderProcessResponse Processor. It process the web service response coming from to uri="bean:orderCXFServiceClient?method=processOrder". We can also have processing logic inside OrderProcessResponse processor class.

public class OrderProcessResponse implements Processor {

@Override

public void process(Exchange exchange) throws Exception {

String orderId = (String) exchange.getIn().getBody();

exchange.getOut().setBody(orderId);

}

This example can be deployed in Apache Tomcat, so you will have to package the .war file and copy it to the webapp folder of Tomcat. You can then use SoapUI or another web service client and send a request to the URL: http://localhost:8181/camel-example-cxf tomcat/webservices/services/routeOrderServiceEntryPoint?wsdl
The wsdl is located at: http://localhost:8181/camel-example-cxf-tomcat/webservices/services/routeOrderServiceEntryPoint?wsdl
And CXF outputs which web services it has from this url http://localhost:8181/orderapp/OrderProcess?wsdl

Camel Expression language:

Simple expression language in camel can be used to evaluate an expression on the current instance of Exchange that is under processing. It is used for both expressions and predicates, thus making it a perfect match to be used in Camel routes.

If else implementation in camel expression:

<route>

<when>

<simple>${body} contains 'Camel'</simple>

</when>

</otherwise>

</choice>

</route>

for loop implementation in camel

<loop> tag is used in camel for implemention for loop. <header> tag determines the loop count.

<when>

<simple>${property.groupId} == 0</simple>

<loop>

<header>loop</header>

<process ref="createGroupRequestProcesor"/>

<to uri="bean:groupClientInQuote?method=createGroup"/>

<process ref="createGroupResponseProcesor"/>

</loop>

</when>

</otherwise>

</choice>

Operator support:

Operator	Description
==	equals
>	greater than
>=	greater than or equals
<	less than
<=	less than or equals
!=	not equals
contains	For testing if contains in a string based value
not contains	For testing if not contains in a string based value
regex	For matching against a given regular expression pattern defined as a String value
not regex	For not matching against a given regular expression pattern defined as a String value
in	For matching if in a set of values, each element must be separated by comma.
not in	For matching if not in a set of values, each element must be separated by comma.
is	For matching if the left hand side type is an instanceof the value.
not is	For matching if the left hand side type is not an instanceof the value.
range	For matching if the left hand side is within a range of values defined as numbers: from..to. From Camel 2.9 onwards the range values must be enclosed in single quotes.
not range	For matching if the left hand side is not within a range of values defined as numbers: from..to. From Camel 2.9 onwards the range values must be enclosed in single quotes.

Syntax:

${leftValue} <OP> rightValue

The value on the left side must be enclosed in a ${ } placeholder. The operator must be separated with a single space on the left and right. The right value can either be a fixed value or another dynamic value enclosed using ${ }.

Example:

simple("${in.header.foo} == Camel")

The OGNL feature

Both the Simple language and Bean component support an Object Graph Navigation Language (OGNL) feature when specifying the method name to invoke. OGNL allows you to specify a chain of methods in the expression.

Suppose the message body contains a Customer object that has a getAddress() method. To get the ZIP code of the address, you would simply use the following:

simple("${body.getAddress().getZip()}")

You can use a shorter notation, omitting the get prefix and the parentheses.

simple("${body.address.zip}")

null-safe operator

In this example, the ZIP code will be returned. But if the getAddress method returns null, the example would cause a NoSuchMethodException to be thrown by Camel. If you want to avoid this, you can use the null-safe operator ?. as follows:

simple("${body?.address.zip}")

The methods in the OGNL expression can be any method name. For example, to invoke a sayHello method, you would do this:

simple("${body.sayHello}")

Access Map or List:

We can access Map or List objects directly using their key name (with or without dots):

simple("${body[foo]}")

Suppose there was no value with the key foo then we can use the null safe operator to avoid the NPE as shown:

simple("${body[foo]?.name}")

Set property of exchange:

We can use <setProperty /> tag in Spring DSL to set property on exchange object.

</setProperty>

Example to use the property set in exchange:

${property. customerObj.customerId}

customerObj is my property name which contains customer object, I want value of customerId out of it, so I have used property. customerObj.contactId.

Conclusion:

Apache Camel is a light weight integration framework. It enable you to integrate several different technologies by always using the same syntax and concepts – including very good testing support. Again still choosing the right framework depends on your business need. Remember: Often, a fat ESB has too much functionality, and therefore too much, unnecessary complexity and efforts. Use the right tool for the right job!

TECHNOLOGY

Tuesday, 8 April 2014

Ride on Apache Camel

When to use Spring Integration or Custom Solution?

Operator support: