The yearly Youredi R&D roadmap process I spoke of in my previous blog is nearing its end. Yet still, it is constantly providing good possibilities to look back – not only think about what we have achieved over the last few years, but also how that has been achieved.
What we have established so far is a cloud-based integration platform, or iPaaS (Integration Platform-as-a-Service), as often stated. It is a combination of software, best practices, and internal processes, making sure that this combination evolves over time and fulfills the customer's needs now as well as in the future.
But looking back, what does it actually take to create an integration platform? If you just look at the integration and what it is all about, it should actually be pretty simple. It’s simply about transferring information from one place to another – making sure that all the information is available at the right place at the right time so that your customers’ business processes can work in the way they are expected to.
So, looking back in time – and planning for the future – there are really just a few steps that you need to follow when creating such an integration platform. These steps are not necessarily carried out in a specific order, but you need to consider all of these things simultaneously in all phases of the platform lifecycle.
In this series of blogs, I will provide a five-step guide on how to create an integration platform, if you are interested and ready for that journey. For us, this road has been very fascinating, and we learn something new every day – as is always the case when dealing on a day-by-day basis with new customers, partners, new applications and technologies.
The first part of my five-step guide is about the very basics – making sure that your integration platform really acts as an integration solution. Hence in this part, we will cover the following:
When you have all these elements addressed, you can move forward.
Take care of the basic functionality
This is likely the easiest part when you are about to create an integration platform, but it has to be taken very seriously – after all, it is responsible for the core part of everything that your integration platform is designed to accomplish.
An integration solution is almost always pretty simple, at least if you look at it from a distance. Integration is always made of a limited set of functionalities, and these functionalities are well-known. After all, there has been some sort of integration solution since the time there were more than two computers that needed to work around the same business process.
So, what does it take to manage a simple task of transferring information from one application to another? Four things: interfaces, translation, control, and monitoring.
Interfaces
To begin, you need to have interfaces to the integrated applications. You need to create software that is capable of communicating with different applications, built by different vendors, running on different servers, possibly managed by different organizations, and may be distributed geographically. For this, you will likely create some adapters or connectors that make the communication possible. In addition to this, you will create a set of APIs that the integrated applications can use when they are asking for your integration platform to do something on their behalf.
In the past, there were integration platforms that had a substantial number of different interfaces. There was an adapter for application A, another adapter for application B, application C, and Infinitum. But today, the truth is that there are only a handful of adapters (or API interfaces) required. This is partly due to the standardization of APIs, but more likely, because of the thought software development process many vendors have come through over the past years. Instead of believing that everything can be done, with one single software suite, the “best-of-breed” approach has gained acceptance in the modern business. Instead of having a big single ERP, enterprises are relying on specialized software suites for different operations – whether it is accounting, order management, warehouse management, project management… you name it. And because all these applications are designed to accomplish only a limited task within a business process, they need to be able to communicate with the applications taking care of other parts.
This has lead into a situation where applications are providing more and more documented, well-defined APIs using a limited set of standard communication protocols such as REST or Web Services. Add some other protocols on top of that – say, FTP, SFTP, AS2, database connectivity and maybe some others, based on your customers’ requirements – and you are there. You have taken care of the first part of the basic functionality.
Translation
Even though applications provide APIs, they still do not speak the same language. For us humans, we are able to switch to a common language when we meet (quite often that is English), but applications still have their own internal “language” which they utilize, and there is no similar lingua franca as English available. Because of this, there is a need for translation services available within your integration platform. The message that application A sends may contain a business object (e.g. an order) that both parties understand, but because application A communicates with different “language”, application B does not understand what A means. It’s similar to a situation in which you are bargaining with a merchant that doesn’t share any common language with you – you both know that you are negotiating a price, but none of the parties understand how the bargaining proceeds.
Because of this, your integration platform must be able to translate messages and documents between different participants. Again, this rarely becomes a problem when you are developing your integration platform, as there are multiple ways to achieve such translation services. In fact, all the modern programming languages provide more than enough functionality for this; add some XSLT transformations, graphical mappers and such tools into the equation, and you are done with this requirement.
So, you have interfaces to communicate, and you can translate the information exchanged between different parties. You are almost there, but there are two things you still need to take care of with your base functionality – controlling and monitoring. These two components create an “umbrella” under which everything interesting – transferring and transforming information – happens.
Control
The first of the two missing parts that are critical for creating an integration platform is the control. It is not enough to just receive information, translate/transform it, and send it to another application. The truth is almost always a bit more complex.
First off, there may be several source systems you need to fetch information from, and there may be several destination systems you send your data to. You have to take care of one-to-one, one-to-many, many-to-one, and many-to-many scenarios with your platform. Add things like content-based routing, batching, de-batching, etc., and you will find yourself thinking about how to manage and control these more complex situations.
In addition to this, you have to remember that you are dealing with applications that are not always under your control. When creating an integration solution built on top of your platform, you have to consider all the possible error scenarios. Some of the most common scenarios include (but are surely not limited to, believe me!) things like:
- Integrated application is not available: maybe there is a network error, maybe the application is down for maintenance, or something else has happened
- Integrated application suddenly starts to speak a completely unknown language or dialect: maybe because of a software upgrade or a change in the configuration you were unaware of
- The contents of the message you have received are simply wrong: you need to be able to analyze and validate the input before transmitting the failure to other parts of the business process
- You have asked an integrated application to process information and are waiting for a response but never get one: maybe your integration application has provided some erroneous information for the destination application and it is choking…
And so on. When dealing with multiple integrated applications, there are a million things that can go wrong. Fortunately, you can always narrow the possibilities down to a limited number of most probable possibilities, but you still need to be able to handle these errors. When you continue the use of your platform, you will find other ways for your integration solution to fail, and will address those possibilities as well. Because of this, you need to develop some kind of a manageable, configurable, and re-configurable platform control layer on top of your integration solution.
In almost all modern integration solutions, this control layer is created in the form of process/workflow/orchestration control. (We at Youredi are talking about processes). These processes or workflows manage the overall communications routine. A process typically starts from an incoming message that triggers a number of actions within an integration platform. The process is a “programming language” that has control structures familiar to different programming languages (if-then-else, switch, on-error-goto, throw-catch…) and is able to call atomic processing steps as a part of the process (transform/translate, validate, parse…)
In its essence, a process is at the very core of what you want to achieve with an integration solution. A developer building an integration solution on top of your platform needs to be able to define this process just the way he wants – or his customer wants. The developer will end up spending the majority of his workday defining these processes, defining how the information exchange should be controlled, and finding out all the possible ways the process can fail. So make sure that your process/workflow/orchestration engine is user-friendly and complete. Work together with those who are using these tools in order to make this very core process of developing integration solutions as efficiently as possible.
Monitoring
Now, this may be one of the most crucial elements in your platform. You will always find your developers and business users in a situation where they want to know what is happening and what has happened. When someone is developing an integration solution, he needs a “debugging” view to see how his solution works; when the solution is deployed, your customer wants to see if his information is exchanged correctly; and finally, when something goes wrong, someone (maybe your customer, or maybe your own support personnel) needs to be able to see what went wrong, and even more importantly, why.
Monitoring can be divided into three things you probably want to be providing from your integration platform:
- Platform-level monitoring: You have to be able to understand how your platform is working. Are all the servers up? Is the network up? What is the throughput of your platform, etc. Depending on how you are planning to deploy your platform decides who is the end user – for on-premise integration platforms, the end user is most probably your customer’s IT organization, and for cloud-based integration platforms, it is most probably your own operations.
- Solution-level monitoring: Your developer and maybe your support people want to understand what is happening and what has happened in the solution that has been deployed. As discussed, there are a million things that can go wrong, and once information does not transfer correctly, you need to be able to act fast.
- Business monitoring: As a solution built on top of your integration platform will be delivering information between applications, and as it is controlling information exchange that might be crucial for your customer’s business processes, they most probably want to have a high-level view of what is going on. Your business users are not interested if one of the technical integration processes has failed or how it is working internally, but more at the top-level metrics. These metrics often have properties like “volumes”, “value,” or “throughput” – anything that could be used for monitoring on how the business works. This level of monitoring – often referred to as “Business Activity Monitoring” in integration solutions – is the place where you would most probably want to use some graphs, traffic lights, and pie graphs.
You have to be prepared to serve all these different monitoring needs. They are all crucial for both business and operations and will most probably have different end users with different interests and skill sets. Take this into account when creating your monitoring solution(s) – bear in mind that different people want to see different parts of the same underlying integration solution.
With our own platform, we started the development several years ago from these basic functionalities. The basic process definition and execution were the parts we started our development from; we continued adding new capabilities for the processes and simultaneously worked on different monitoring interfaces. We made some careful choices in the beginning in order to make it possible to extend these functionalities in the future, and that has paid off; when you are creating your own integration platform, make sure that you won’t hang yourself in something that cannot be extended in the future. For instance, our Analytics offering – the business-oriented part of the overall monitoring – was introduced much later than the other monitoring functionalities.
So, now you should be up to speed with the development of your integration platform. It provides all the basic capabilities that an integration platform needs to accomplish. There are still a few steps to get it deployed to your customers, make sure that you can develop it further, and find out where your niche market is. Next time we’ll discuss how to make it reachable – how to make sure that someone will start using your platform.
