As a shipper sending cargo worldwide or any other stakeholder of the global supply chain interested in receiving vital information timely, you likely rely on APIs—the most convenient way to exchange data with your business partners. If so, you must be familiar with the pain point, such as the lack of webhook functionality in APIs you must work with. Often, such an issue makes you allocate unplanned human and IT resources to ensure you receive data the moment you need it. Plus, in some specific instances, you can even encounter system overloads caused by your system's inability to cope with the amount of data received as a result of repetitive queries to partners' APIs. This article explains how to avoid these issues and ensure a flow of business data in near to real-time even if your business partner's API does not support webhooks functionality.
API in the logistics and global supply chain domain
API (Application programming interface) is a type of software interface that allows a company to expose some in-house data to an external consumer in a controlled and secure manner. Being based on standardized protocols (XML or JSON), API makes it relatively simple to start exchanging data with your business partners regardless of the size or location of your business. Publishing an API makes it possible for your trading partners to quickly connect to your data, given they have the technical capability and tools to integrate with an API.
The versatility of APIs makes them applicable in logistics and supply chains. In fact, the universal nature of API enables countless scenarios of how it can be leveraged within the industry. However, the most common use case for it is integrations with various systems (e.g., visibility platforms, transportation management systems, port community system, ocean booking platforms, etc.) and automation of different processes and transactions (e.g., requests of rates, invoicing, booking confirmations, dispatching or tracking shipments). Undoubtedly, having API as a central element in data integrations strategy enables various benefits and opportunities to grow your business faster. This results in a constantly increasing number of companies adopting the API-first approach.
Does API have any drawbacks?
Although the interface provides an opportunity to connect anything to everything and helps to automate certain business processes, basic APIs are reactive by nature, so an API-based communication between software applications is always driven by requests.
Each time you need to get information from your partner, your system must send a query. As a result of such a query, you will get a payload (a specific set of data) that needs to be processed, making it digestible for your IT system.
Looking at this process, we can distinguish two elements - Polling logic (a piece of code responsible for sending queries based on specific needs) and data processing (e.g., applying data formatting or data harmonization to a payload and filtering out irrelevant data).
Many APIs based purely on polling require the client to maintain the status of data that has already been processed. Depending on the API, the logic for handling duplicates and filtering meaningful events might get complicated. The complexity heavily depends on the needs of your own business processes and the IT systems receiving the data. For instance, if you need to receive data once or twice a day, it is not as difficult and resource-intensive as if you need constantly sending requests to a partner.
A good example of the latest is when you use API to track whether your ocean shipments are moving (e.g., DCSA T&T status received from a carrier). Since such data is often needed for planning actions or process optimization, it is essential to get it rapidly. So, you need to query a carrier's API with specific intervals. For instance, daily, hourly, or even on a minute basis, depending on the case.
Given that a status update for ocean shipments happens unpredictably, your intervals cannot be too long because, most likely, you will fail to spot right moment. Choosing shorter intervals increases the amount of data to be received. So, you need to be sure your IT system can digest and process payloads received from partners. Otherwise, you put yourself at risk of overloading your system with the same type of events coming repeatedly. This is especially critical when you work with multiple parties simultaneously.
To give you a more familiar example, think of this process as if you were expecting a package delivery from an e-commerce platform like eBay or Amazon. Imagine you purchased something simple like a t-shirt for the upcoming summer. Since it is still wintertime, most likely, you will not be that interested in getting a status update on the package's current location. So, you occasionally click a tracking link, knowing that the arrival date will not make a difference unless it is within a promised interval. But when you expect something special, perhaps a birthday present for a loved one, and its delivery affects a sequence of actions in preparing a surprise party for the same, you will constantly be checking the status until it arrives.
In the second scenario, when you have more at stake, you need to put more time and effort into ensuring that all will be according to the plan. The same logic applies to the business world. The timely received data guarantees more efficient actions and planning, plus lower operations costs.
If you track your delivery through a courier website, you probably know that nowadays, to improve the delivery experience for customers, most postal services provide a "subscribe for status update" feature as a part of their tracking service. This feature automatically notifies you via SMS or an email each time your package changes its status. The same "notifications" are possible for the world of logistics and supply chains. However, unlike the example above, in the b2b communications, instead of SMS, companies often prefer to receive a status update as data is delivered via API to their IT systems the moment the change accrues. And this is possible thanks to webhooks.
What are webhooks, and why do they matter?
A webhook (also known as an HTTP push API) is an API mechanism for enabling a one-way automated data push triggered by a specific event (a pre-defined action or occurrence of your interest). Whatever event you specify, the information automatically flows to your system when it happens. This means no request is necessary to receive updated data. Thus, you can receive data in real-time and initiate automated actions.
Simply put, it is a subscription (user-defined HTTP callbacks) for an event critical for your operations. A good example is a subscription to Track and Trace events happening on the carrier side. You no longer need to send scheduled queries anymore to get vital data. Instead, you configure a callback, so the server automatically sends data to your system when an event happens. Some examples include a container that has been loaded to a ship, a container that has been discharged or delivered to a customer, or anything else.
One extra thing that webhooks require is the capability of the data consumer to put up a receiving HTTP endpoint and the authentication mechanisms for receiving the webhook calls.
What if your business partner's API does not support Webhooks?
Today, despite the growing popularity of APIs within the logistics and supply chain domain, webhooks are a rather rare thing.
One example is the API of ocean carriers. Although most of them rely on standards that imply subscription capabilities/webhooks (e.g., DCSA Track and Trace), their APIs still do not provide this functionality for customers. That often happens because implementing webhooks requires an R&D team to maintain, modify, and manage all subscriptions, potentially being a significant burden for many.
So, if a carrier you work with is one of those, you have several options:
1. Build a basic polling logic and continue working with reactive APIs.
Although this one seems to be the easiest way, you assume all risks and limitations described above if you go with this option.
2. Built a data integration solution with sophisticated logic capable of replacing the webhook functionality missing from a carrier side.
Here, you need a team of experts to design, deploy, and maintain the solution—every single day. All the callbacks/events/logic will be maintained, modified, and managed by your IT team and will require no actions from a carrier end. This way, you can get data in real-time and improve your business processes and operations. As a drawback, you will get an additional financial burden, unpredictable results, and most likely HR-related issues since it involves R&D and talent management elements.
3. Partner with a data integration vendor who provides integration solutions developed specifically for your domain.
Leveraging a third party to do the work is the fastest route to solve the problem. However, you need to ensure that the vendor has expertise in your domain and a team that will fully manage the offered solutions. Otherwise, you still have to deal with almost the same drawbacks as if you do it in-house. Plus, teaching a generic integration vendor the ins and outs of your business domain is also time-consuming and costly.
What if you can't publish HTTP interfaces for receiving webhooks?
Webhooks are a handy mechanism for receiving notifications for events as they happen. Still, they do require some IT capabilities from your side, namely the ability to publish receiving HTTP endpoints for the different APIs you connect to. You might have IT systems that are able to receive the data but communicate with older transport protocols and data formats.
Leveraging Youredi's integration solutions to Grow Your Business
Youredi is the leading provider of fully managed data integration services and solutions for logistics and the global supply chain. Our solution is perfect if your company operates in this domain and needs to consume data via APIs that lack webhooks—or if the webhooks are available in the API but your own company lacks the technical capability to receive webhooks, our solution is what you need.
The solutions we provide are fully managed and work as a cloud-based layer between you and your business partners. That means you need not change anything in your existing IT systems to get it fully operational. Overcoming the webhook issue is one of our team's typical integration scenarios daily.
The process in brief: we connect to an API on behalf of the data consumer (our client). We build a logic similar to webhooks and implement the polling logic enhanced with filtering, as well as the data harmonization feature, if necessary. The data consumer receives data vital for its business timely and in the format digestible for its IT system.
As you integrate our solution into your business process, you have no more need for internal resources to send queries, process the response, and maintain connections with all your business partners' APIs. This way, you lower operational costs and ensure a seamless flow of reliable data vital for your operations.
Benefits you can expect by working with us:
- No need for additional resources to keep an eye on new events
- Instant, real-time notifications regarding specific custom-build events
- Process automation - no need to send requests to your partner's API to get the events
- Data processing (including conversion and harmonization) to match your needs
- A team of experts that will design, implement and maintain all your API connections
- High scalability and flexibility to keep up with your business growth
Youredi API management service for data providers
In our portfolio, we also have an offering for data providers – Youredi API management.
Youredi API management is a service helping global logistics and supply chain actors accelerate interconnectivity across systems and applications through their ecosystems.
Youredi API management is an excellent choice for those who want to expose their in-house data as a modern API but don't know how to or don't want to invest in in-house resources.
Our team will build, host and manage an API on your behalf. We provide complete lifecycle API management services from design to implementation and further maintenance and monitoring of your APIs. As your business requirements change, so do the specs of your APIs, we help you to implement the changes through versioning your APIs.
Relying on us, you will get a layer of modern APIs that will satisfy all the needs of your business partners, even if your backend systems are not capable of supporting modern data formats and transport protocols.
Do you have questions about the API management solution or want to learn more about Youredi? Please schedule an online meeting with our team of integration experts or contact us via this form.
Recommended reading:
"Field of Dreams: The Trouble With API Integration"
