Have you ever wondered what might be the most popular data integration tools in the world? Initially, if you work in IT, you may be thinking of some household IT vendor names and their solutions, trying to figure out which one might be the most popular. However, that would be boring. The correct answer is actually a pretty interesting one (and no, it’s not the Youredi iPaaS platform), so we decided to write a blog about it.
The Most Popular Data Integration Tool in the World
Gartner analyst Massimo Pezzini had a great opening line for his presentation at this year’s Symposium in Barcelona. He asked the audience: “Do you know what the most popular integration tool in the world is?”. No one dared to raise their hand, and albeit I was tempted to shout out the Youredi name, I had to admit I had no real clue. After a relatively long silence in the room, he gave us the rather surprising answer: “It’s you”.
Apparently, the human being is by far the most common and popular integration tool used in today’s world to transfer data between different applications or systems. For better or for worse, we still key in information from one system to another numerous times every single day. Why? Because sometimes it really is the most feasible way to get things done. With small amounts of data that must be transferred at random intervals (maybe even just once), the “human integration” is typically sufficient to “get the job done”. It works even when the data is not in a digital format, to begin with.
We all know that humans tend to make errors on a regular basis. While there are different methodologies that enable to reduce the number of errors (as well as detect errors when they happen), it’s safe to say that “human integration” is not a particularly good integration method when the amount of data increases. And the amount of data does not have to increase by much before we humans start to struggle. As an example, if you had to copy a piece of text (e.g., a password) from one system to another, it’s pretty easy to do it flawlessly if it’s a normal 8-character string of numbers and letters. Now, try doing that with an 80-character string of text, and your chances of getting it right the first time reduce significantly. Furthermore, getting the job done correctly will probably take you more than ten times longer.
The Second Most Popular Data Integration Tool in the World
This brings us to the “second most popular integration tool in the world”. You may have guessed it already. The marvelous “Copy&Paste”. We all use it with our computers, smartphones, and other digital gadgets, as it reduces the effort and improves the quality of the “data integration” significantly. There’s no need to worry about “typos” or upper/lower cases etc. Also, it’s relatively quick and easy to “Copy&Paste” even slightly larger amounts of information between different systems. What’s best when you are copying text from one system/application to another? Usually, you can transfer the data even when each system uses different formats.
Obviously, the “Copy&Paste” method does have its limitations as well. First of all, it’s pretty boring work and becomes very difficult when the amount of data increases. While it’s ok to copy a few data fields occasionally from one system to another, no one really wants to do this day in, day out. Furthermore, even though the data set that is being copied would not change during the “Copy&Paste” process if you have to copy tens or even hundreds of data sets separately between systems, the likelihood of making a mistake increases significantly. In the end, the “Copy&Paste” integration is really just an enhanced version of the basic key-in “human integration”. You can learn more about the b2b data quality issues from our blog - Data Quality: “No Garbage In, No Garbage Out”
The Third Most Popular Data Integration Tool in the World
As the amount of data increases, we typically start using some file transfer methodology. The simplest example is copying files from one folder to another or sending files by e-mail from one place to another. Again, this is still one form of “human integration”, but it allows for a large number of data to be transferred in a very short time.
Transferring messages by digital means is extremely popular in business. More often than enough, we are asked to fill in spreadsheets and send them over to another unit of our company. Or our trading partners provide us with data in a CSV file or a text file, which we then hoover into our own system to be further processed. File transfer is great when the data at both ends is structured the same way.
A more sophisticated version of this type of data integration is an automated “Managed File Transfer” process, where the transfer of data files is done automatically by the IT systems, and humans do not have to actively be involved in the process after it has been set up. Systems will automatically handle the file transfer, and humans may only oversee that the process is working as it should. If something goes wrong (e.g., the process is stopped), then the error is fixed by a person whose job it is to look after the process.
Where the simple file transfer process runs into trouble is when the data structure in the originating system does not match the data structure of the receiving system. In that case, we are forced to manipulate the data somehow before we can use it in the receiving system. Typically, this manipulation is done manually through “human integration”, which means increased efforts and workload. Sometimes is unbelievable how complex “human integration” processes companies are forced to use just because they haven’t figured out a better way to transfer data between different systems.
Other Honourable Mentions - RPA
One of the most interesting developments in IT over the last few years has been the emergence of Robotic Process Automation (RPA). While the idea of RPA itself is nothing particularly new, the hype around RPA has gone through the roof during the past few years. The RPA market growth has been exponential, and the major technology providers such as Automation Anywhere, BluePrism, and UiPath have grown their revenues at an unusually high pace for a business to business software.
Basically, RPA emerged from the need to replace the boring and laborious “human integration” work with the work done by “software robots”. Software robots are most commonly used as a human to do the key-in or “Copy&Paste” integration work between different applications or systems. The processes that the software robots work on are basically the same that humans have done in the past. The process is “taught” to the RPA solution (or it learns the process by observing the humans running it), whereafter, the robot keeps repeating the tasks in the same order. Access to the systems is typically done through the same Graphical User Interfaces (GUIs) that humans use.
With the cost of RPA technology going down, the increase of solutions in the market (including some open source solutions), and the relatively easy way of setting up an RPA process, it is likely that the RPA integration solution will continue to grow rapidly in the future. Albeit RPA solutions can handle much more data than any human ever could, the RPA solutions are still limited to relatively simple, typically point-to-point integrations and, moreover, typically only used within an organization’s own operations. Also, RPA solutions are not able to automatically adapt to changes in the GUIs, so they still need humans to do maintenance work on them on a regular basis (depending on how often the integrated applications change their GUIs).
Systems Integration
Systems integration is a process that links together an organization’s various IT systems, services, and/or software to enable all of them to work functionally together. Through Systems integration, the various IT systems can “talk to each other” by exchanging data to speed up information flows and reduce operational costs for the organization. System integration is also used to connect an organization’s IT systems with third parties that the organization trades with.
In addition to just data exchange, system integration can also include several processes where the data is enhanced, enriched, harmonized, and/or transformed from one format to another. Systems integration is typically used when the amount of data or the frequency of the data exchange is high or the data cannot be exchanged easily by other methods. Systems integration also allows for many-to-many integration scenarios. The downside of system integration has traditionally been the relatively high costs and maintenance requirements that it has.
However, modern cloud-based system integration solutions like iPaaS (integration platform as a service) have decreased the total cost of ownership for system integration, making it more accessible for organizations of all sizes. (Read the "B2B Integration - Why Does the Total Cost of Ownership (TCO) Matter?" article to learn more regarding the total cost of ownership when building integrations in-house)Today there are several large iPaaS solution providers as well as numerous smaller players in the market for organizations to choose from. So, is it clear that the demand for these types of integration solutions is also growing all the time.
Choosing the right data integration tool for your business
So, what’s the lesson to be learned from all this? Well, as in life in general, you should always choose the right solution for each case. Each of the integration solutions mentioned above can and should be used when the occasion is fitting. What is important, though, is that each organization has a clear understanding and a plan regarding its data integration needs and has the right solutions available to it for each case. For the more complex the case, the more sophisticated tools are needed.
Furthermore, integration is a constantly changing and evolving process. What might work “ok” today may not be an ideal solution anymore in the future. So, as an organization, you should also prepare for the changes that may be lurking just around the corner. With the rapid pace of development in technology (and in society in general), don’t allow yourself to be caught off guard by any new requirements that you have to comply with.
Our team has prepared a list of 15 data integration vendors that, in our opinion, provide the best solution for the global logistics and supply chain domain.
