Polling a JSON REST API with an Apache Camel component

If you have been playing around with integration frameworks, you are probably aware of Apache Camel, as it is one of the two big open source players together with the Spring Integration framework. While benchmarking Talend ESB and Redhat Fuse ESB products, which both include Apache Camel in their stack, I had the opportunity to dive a bit more into the concepts of the framework and figured there was some interesting material for a blog post !

The purpose of this post is to provide some guidelines to build a custom Camel component for your enterprise integration needs. Our example will show how to poll an external JSON-based REST API – in this case, the Feedly API – to periodically retrieve and process information – in this case, to log retrieved RSS feed titles.

camel-custom-feedly-compone

This example requires the following dependencies:

Step 1: Maven configuration

Add the following dependencies to your pom.xml file.

<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>2.15.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.4</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>1.7.1</version>
</dependency>

Step 2: Discovering the Feedly API and setting the properties file

For this example we will be using the Feedly Search API, as it is simple to use and does not require user authorization. It provides suggestions of feeds related to a topic; hereafter an example of a GET call to the API:

GET http://cloud.feedly.com/v3/search/feeds?query=iot

There are two parameters related to this API call, which are the “query” parameter and the request frequency (“delay”). Isolate these two parameters in a dedicated /src/main/resources/feedly.properties file:

#Feedly API call properties
delay=10000
query=iot

Step 3: Setting up the custom component, endpoint and consumer

Camel custom components are responsible for creating Camel endpoints, which in turn create consumers and producers. The RedHat Fuse ESB documentation explains those concepts quite clearly. For our example, we will be therefore creating 5 classes:

  • FeedlyComponent: our custom Camel component
  • FeedlyEndpoint: our custom Camel endpoint
  • FeedlyConfiguration: a configuration POJO, which makes it easier to configure our custom endpoint
  • FeedlyConsumer: our custom Camel (polling) consumer, which periodically makes the API calls and converts JSON responses to Camel Exchange objects
  • Feed: simple POJO to hold the RSS feed information parsed from the JSON responses

Note that we do not create a Producer implementation, as we only do read-only GET calls to the API.

FeedlyComponent class

The class inherits UriEndpointComponent in order to allow better documentation generation, as we shall see later. The createEndpoint method is responsible for instanciating new endpoints and populating the endpoint configuration from the parameters map. Note that the parameters map MUST be emptied afterwards, which can be achieved automatically by using the inherited setProperties method; otherwise Camel shall throw an ResolveEndpointFailedException.

public class FeedlyComponent extends UriEndpointComponent {

    public FeedlyComponent() {
super(FeedlyEndpoint.class);
}

@Override
protected Endpoint createEndpoint(String uri, String remaining, Map parameters) throws Exception {

FeedlyEndpoint endpoint = new FeedlyEndpoint(uri, remaining, this);
FeedlyConfiguration configuration = new FeedlyConfiguration();

// use the built-in setProperties method to clean the camel parameters map
setProperties(configuration, parameters);

endpoint.setConfiguration(configuration);
return endpoint;
}
}

FeedlyEndpoint class

The class uses the @UriEndpoint and @UriParam annotations for documentation purposes. The createConsumer method inherited from DefaultPollingEndpoint is responsible for instanciating the consumer.

@UriEndpoint(scheme=”feedly”, title=”Feedly”, syntax=”feedly://operationPath”, consumerOnly=true, consumerClass=FeedlyConsumer.class, label=”feeds”)
public class FeedlyEndpoint extends DefaultPollingEndpoint {

    public FeedlyEndpoint(String uri, String operationPath, FeedlyComponent component) {
super(uri, component);
this.operationPath = operationPath;
}

private String operationPath;

@UriParam
private FeedlyConfiguration configuration;

public Producer createProducer() throws Exception {
throw new UnsupportedOperationException(“FeedlyProducer is not implemented”);
}

@Override
public Consumer createConsumer(Processor processor) throws Exception {
FeedlyConsumer consumer = new FeedlyConsumer(this, processor);
return consumer;
}

public boolean isSingleton() {
return true;
}

public String getOperationPath() {
return operationPath;
}

public void setOperationPath(String operationPath) {
this.operationPath = operationPath;
}

public FeedlyConfiguration getConfiguration() {
return configuration;
}

public void setConfiguration(FeedlyConfiguration configuration) {
this.configuration = configuration;
}
}

FeedlyConfiguration class

Again, @UriParams and @UriParam annotations shall be useful later for generating the documentation of our component.

@UriParams
public class FeedlyConfiguration {

    @UriParam
private String query;

@UriParam(defaultValue = “60000”)
private int delay = 60000;

public int getDelay() {
return delay;
}

public void setDelay(int delay) {
this.delay = delay;
}

public String getQuery() {
return query;
}

public void setQuery(String query) {
this.query = query;
}
}

FeedlyConsumer class

Our custom consumer extends ScheduledPollConsumer in order to automatically gain polling capabilities. The poll method is responsible for choosing the Feedly API method to call, based on the operationPath stored by the endpoint. The HTTP client does the call while GSON converts the JSON response into a list of Feed items. Finally the list is stored in an Exchange object to be passed to the next Camel processor.

public class FeedlyConsumer extends ScheduledPollConsumer {

    private FeedlyEndpoint endpoint;

public FeedlyConsumer(FeedlyEndpoint endpoint, Processor processor) {
super(endpoint, processor);
this.endpoint = endpoint;
this.setDelay(endpoint.getConfiguration().getDelay());
}

@Override
protected int poll() throws Exception {

String operationPath = endpoint.getOperationPath();

if (operationPath.equals(“search/feeds”)) return processSearchFeeds();

// only one operation implemented for now !
throw new IllegalArgumentException(“Incorrect operation: ” + operationPath);
}

private JsonObject performGetRequest(String uri) throws ClientProtocolException, IOException {

HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(“http://cloud.feedly.com/v3/” + uri);
HttpResponse response = client.execute(request);
if (response.getStatusLine().getStatusCode() != 200)
throw new RuntimeException(“Feedly API returned wrong code”);

JsonParser parser = new JsonParser();
InputStreamReader sr = new InputStreamReader(response.getEntity().getContent(), “UTF-8”);
BufferedReader br = new BufferedReader(sr);
JsonObject json = (JsonObject) parser.parse(br);
br.close();
sr.close();

return json;
}

private int processSearchFeeds() throws Exception {

String query = endpoint.getConfiguration().getQuery();
String uri = String.format(“search/feeds?query=%s”, query);
JsonObject json = performGetRequest(uri);

JsonArray feeds = (JsonArray) json.get(“results”);
List feedList = new ArrayList();
Gson gson = new GsonBuilder().setPrettyPrinting().disableHtmlEscaping().create();
for (JsonElement f : feeds) {
Feed feed = gson.fromJson(f, Feed.class);
feedList.add(feed);
}

Exchange exchange = getEndpoint().createExchange();
exchange.getIn().setBody(feedList, ArrayList.class);
getProcessor().process(exchange);

return 1;
}
}

Feed class

Simple POJO to be populated by GSON. Note the toString() implementation, as it will be used for logging purposes.

public class Feed {

    private String feedId;
private String title;
private String website;
private long subscribers;

public String getFeedId() {
return feedId;
}

public void setFeedId(String feedId) {
this.feedId = feedId;
}

public String getWebsite() {
return website;
}

public void setWebsite(String website) {
this.website = website;
}

public long getSubscribers() {
return subscribers;
}

public void setSubscribers(long subscribers) {
this.subscribers = subscribers;
}

public String getTitle() {
return title;
}

public void setTitle(String title) {
this.title = title;
}

@Override
public String toString() {
return feedId;
}
}

Step 4: Enable auto-discovery of the custom component

Create the /src/main/resources/META-INF/services/org/apache/camel/component/feedly file and indicate the fully qualified name of your custom component class. In our example it is com.paloit.examples.camel.custom.feedly.FeedlyComponent.

class=com.paloit.examples.camel.custom.feedly.FeedlyComponent

By doing so we allow Camel to automatically register our custom component so that we can start building routes using it!

Step 5: Run the example

Finally create the Main class below to build a route using our custom component. We use the Camel PropertiesComponent to load our previous feedly.properties file and get the “delay” and “query” parameter values. Also note that, since our custom consumer produces a list of Feed items, we need to use the Camel split EIP to process each Feed individually. Our example just logs the Feed, but it could be stored or forwarded to another endpoint, depending on your needs.

public class Main {

public static void main(String[] args) throws Exception {

System.out.println(“Starting example, press CTRL+C to terminate”);

org.apache.camel.main.Main main = new org.apache.camel.main.Main();
main.enableHangupSupport();

main.addRouteBuilder(new RouteBuilder() {

@Override
public void configure() throws Exception {

PropertiesComponent properties = new PropertiesComponent();
properties.setLocation(“classpath:feedly.properties”);
getContext().addComponent(“properties”, properties);

from(“feedly://search/feeds?delay={{delay}}&query={{query}}”)

.split(body())

.log(body().toString());
}
});

main.run();
}
}

Next steps

You may find the complete sources for this example on our GitHub: https://github.com/Palo-IT/Examples/tree/master/camel-custom-feedly-component

In a next post I will describe how to generate documentation for our custom component using the Camel annotations.

Share
Alexandre ESTELA

9309

Comments

  1. can i run this as a standalone program?

Leave a Reply

Your email address will not be published. Required fields are marked *