EDI translation and EDI validation are essential components of EDI integration. While EDI as a technology has been present for over five decades, the communication between business-to-business trading partners did not become less uncomplicated.
Exchanging EDI transactions with several stakeholders is still a hassle today. While creating connectivity between different systems and applications is one part of the challenge, another is that different companies use different data standards and data formats for messaging.
Thus, it is essential that the EDI integration process would be more than just a channel for delivering a message from one endpoint to another. During the event transmission, the process should also take care of the translation of the data formats, as well as the validation of all data fields that are defined by the customer.
In this blog, we will shortly look at EDI integration from a critical point of view. We will explain how different data standards and data formats make data sharing more complicated. Last, we will elaborate on the necessity of EDI translation and validation and how these solutions can help to ensure that one would also has the right data in the correct format at their disposal. By the end of the article, you will understand how modern cloud integration solutions have made the EDI translation and validation processes a lot more simple and more ingrained into EDI integrations.
Electronically exchanging business-to-business transactions has many positive impacts. EDI integration has many benefits: using EDI integration to communicate with all business stakeholders will result in more efficient business processes, less manual interventions, elimination of the data entry function, thus less data-entry errors, and straightforwardly in decreased overhead costs. Nevertheless, Benjamin, De Long, and Morton (1990) questioned the advantages of EDI. In their article, “Electronic Data Interchange: How Much Competitive Advantage?” they wrote as:
“Although the odds are better than the lottery, the prospects of hitting the jackpot with EDI application are still slim.”
Much has changed since 1990 when you consider the available technologies for EDI. But still, creating EDI connectivity across multiple endpoints has remained an enormous challenge. The more endpoints one needs to connect, the larger the number of the EDI transactions are, the more data standards and formats they need to harmonize, the chances that the data quality will be poor will increase. No wonder that most studies conducted about EDI consider data quality as the biggest pitfall of the technology.
Very often, the first challenge is to connect legacy EDI systems with cloud-based applications through application programming interfaces (APIs). Once it’s resolved, organizations still need to deal with the EDI translation and validation process.
To avoid data quality issues, it is essential to work with EDI vendors that do not only provide software for connecting endpoints to automate data flow, but they can also work around challenging connectivity cases (e.g., connecting legacy EDI software with REST APIs). They need to be able to deal with any data standard and data format and handle the EDI mapping, and translation process, as well as they must help with creating the business logic for validating all EDI transaction messages.
Some may use EDI as a synonym for EDIFACT. EDI is simply the technology – it’s like a pipe that transfers data between applications.
EDIFACT, on the other hand, was developed to be the data standard that is used for EDI transactions.
The only problem that EDIFACT has quite a few different versions. While EDIFACT is popular in Europe, ANSI X12 is the most commonly used data standard. Nevertheless, these data standards have sub-versions, such as EANCOM, HIPAA, ODETTE, RosettaNet, SWIFT, TRADACOMS, VDA, or VICS. Different industries may use different EDI data standards. There could also be significant differences in how they use it and what data fields they include.
If it wouldn’t be already tricky enough, many enterprises have developed their own proprietary data formats. To work with these formats, integration architects need to have proper documentation available.
At the same time, newer data formats are becoming popular, such as XML or JSON.
While creating new standards may sound worthwhile for many (just think about all the data standardization initiatives regarding blockchain), in the end, it won’t reduce the number of already available data standards and formats.
Luckily, the technology available for translating and harmonizing data formats is mature, thus it’s easy to ensure that all stakeholders receive their data in a format that their applications can comprehend.
When two or more trading partners are exchanging information, it is very likely the data formats and standards will be different in each case. To ensure that they can still communicate with each other, all endpoints need to receive data in a format that they understand. Harmonizing the data format is not enough. The structure of the message needs to be mapped to in a specific way.
Before System B would process the message sent from System A to System B, there needs to be a step in between that will take care of the transformation of the data format. There must be a set of transformation rules to extract values from the event that System A sent and to insert these events into the message format of System B.
As messages can carry a lot of variants of data fields, as well as depending on the EDI data standards and formats, working with them could be complicated. Complex transformations will require concatenating values and aggregation of values.
Let’s look at a simple example. In the message of System A, the address is in one long string, while System B’s message format requires the address field to separate for the street name, street number, or city name. The transformation rule must be exact to ensure that the right information from the source string is extracted to the correct fields in the target event.
The B2B integration technology vendor most typically executes this process as they most usually have the experience and the capabilities to handle different syntaxes.
In general, the EDI translation has to happen for each of the endpoints, as each endpoint can use a different syntax in its messages. Most EDI integration scenarios will deal with a legacy application. This means that the vendor needs to have experience in translating, for example, EDIFACT to XML. Some vendors have also developed EDIFACT translators to accelerate the transformation process.
Data mapping is one of the terms that we hear the most around our office. It's not a surprise. Data mapping is perhaps the most vital part of business-to-business data integration, and it plays a crucial role in the EDI translation process.
EDI mapping happens through creating a file that will be used to translate one data format into another (e.g., converting EDIFACT to a flat file). Applications that facilitate this process generate this file or so-called 'map'. EDI mapping is essential when EDI connections are built, for example, between legacy systems that use, e.g., EDIFACT or proprietary in-house formats.
Poor quality data negatively affects the satisfaction of companies with EDI. While, in theory, you would expect that EDI will provide good-quality data, it is not always the case. All trading partners expect to receive good quality, reliable and relevant information. This is why EDI service providers need to emphasize a higher degree of alignment between the sender and receiver of EDI transactions for improved processing and efficiency.
This is where EDI validation plays a vital role in the B2B integration process. Defining EDI validation guidelines will help to improve the accuracy of the data and improve its quality. EDI validation is an automated process during the data transfer process – preferably before the translation part – that detects errors in the EDI files and rejects them. The validation process will reject a file again and again until all the information is flawless and matches the expectations of the guidelines defined by the customer.
The EDI validation process can consist of the following based on one’s specific requirements:
Cloud integration solutions have been replacing traditional EDI tools. While in the past, one could use an EDI VAN that did not necessarily have the capabilities to handle any other sub-processes, today, it’s essential to work with a provider that helps with all aspects of EDI integration.
The importance and necessity of EDI translation services and EDI validation tools cannot be emphasized enough. Working with a vendor that can support you with all of these processes will not only accelerate the speed of communication with your trading partners, but it will allow you to rapidly connect with new stakeholders and scale the solution as your transaction volumes are growing.
A vendor using cloud integration solutions, such as an iPaaS, will be able to help you with the most challenging connectivity cases and also ensure that you and your partners always receive the information in your own specific data format when you need it while maintaining the data integrity based on your business rules.