Using an Enterprise Service Bus (ESB) model is recommended when you have multiple business applications, data repositories and data consumers that need to talk to each other through a service layer that contains the rules and a common framework on how to communicate between those entities. It provides a seamless process of utilizing connections from the services layer to get the data from any application whenever and wherever it is needed to facilitate business functions. Applications talk to the services layer and not directly to the underlying systems.
For example, a service bus that provides a patient’s insurance coverage can be called by any application such as Claims Management System or a Policy Underwriting System or an Agency Management System to view or update an insured’s policy profile as part of a business function performed by an end-user. Thus all these applications would use one common service to get the patient data. Similarly, there would be additional common services in your ESB layer that provide unique business data by connecting to the underlying systems. Having a library of services that your business can tap into to get the data from any application whether it’s local or on Cloud removes the need to implement point-to-point connections.
Some of the best practices in implementing SOA based ESB revolve around accurately defining the various touch-points to different systems, sending accurate information to the business users (consumers of data) that execute a business function based on that data, updating all the dependent systems that require status updates as and when the transaction completes.
A typical service has these elements:
Now each one of these steps or services above can be further analyzed as:
By addressing these questions a proper SOA implementation strategy can be utilized that adheres to the best practices and guidelines.
Now let’s look at some of the common hazards of implementing an ESB.
Is your bus service easy to access from where your data lives and does it smartly take into account the road and traffic conditions to reduce delays as much as possible? In other words, providing ability for applications to discover what is available in your services layer and what data is provided when called is an important aspect of a good ESB solution.
Foundational architecture of a solution should be simple and should also be designed in a way that can support timely changes without redesigning and going through long test cycles. Creating any solution that is “business aware” is difficult but understanding the business environment of your company and discussing your business road-map in terms of what changes have occurred in the past and what changes can occur in the future should provide your solution team sufficient information to think-through how to add the necessary placeholders in the current ESB which would make it easy to incorporate those future changes if and when they arise.
Too often the IT is well… too “IT focused” and not “business focused”. By that I mean there is hardly any contextual awareness between architectural designs that your IT team develops versus the disruptions your business may face in the future. Let’s take a scenario where your business acquires a new company and now wants to merge the customer data between their systems and yours. How would you now achieve a mashup of disparate systems to get customer data with your existing architecture that is basically hard-coded and point-to-point?
Your ESB does not exist in a vacuum, in other words yesterday your integrations were SOAP based, today they are REST and JSON API based, and tomorrow you can have more abstract formats that are beyond what your current architecture can handle.
Having a solution that provides flexibility in adding or obsoleting older services and allowing you to make these changes quickly should be the basic foundation of your ESB implementation.
Quite often in an ESB implementation I have seen developers building specific services that are meant for internal use and others for external use even though they are providing same data points. Developing services based on which end-point is going to consume the data leads to duplication where you have multiple instances of the same data service but now with different rules on which application can call that service. Developers get into the habit of “let’s build a new service for everything” rather than reviewing what they have currently in their library and figuring out what can be reused. Building a “services spaghetti” is often a symptom of “I can do anything” syndrome that is sadly too prevalent in today’s IT.
For example, there would be a service to process Purchase Orders in EDI or in JSON or in SOAP or in IDoc each with different end-points and request parameters. What if there’s one service that can provide the same data in different formats so that you have one service to manage going forward rather than several.
There’s often talk about how SOA has not lived up to its expectations when it comes to ESB. Reasons thrown around are that it’s too complicated, or developers don’t understand it, or it’s just easy to solve the current pain points through traditional roll-up-your-sleeve coding. I think the reason why the traditional ESB applications have failed in enabling SOA is because they are inherently code based glorified Eclipse platforms that don’t have the features or the underlying architecture to support SOA.
At the end of the day when your ESB generates code then it’s not an ESB!
It’s just another code generation platform that does not provide the SOA features of reusability, governance, meta-driven framework of building rich services.
ESB must support long running transactions that require waiting for application data to arrive before going to the next application “bus stop”. Supporting not only transactional APIs but also human workflows in a business process is an important part of the service bus implementation. Waiting for a certain event to occur before taking next steps is critical in establishing a true ESB. Too often an ESB is purely transactional which does not take into account the need to stop at a particular step in a business process before proceeding to the next stop. It assumes that the data is always correct, it assumes that the response to the request is always synchronous, it assumes that the data is tactical and is not “business aware” and does not wait for more meaningful data to arrive, it ignores context, it is indifferent to additional inputs as it is purely key-value centric. ESB if the data is not there, too bad, it simply moves on.
Example, an underwriter writing a policy for a commercial property may require additional supporting documents to complete the policy and that may take a few days. In other words, combining ESB with BPM (Business Process Management) helps organizations have better control of their entire business processes from the perspective of what applications and databases are being connected under what business use cases.
Another aspect of the long-running transaction support is showing rich dashboards to the end-users to track what step the flow is waiting and the current status. A major flaw in the ESB products is that they don’t have a rich dashboard facility for business users.
Typical ESB implementations are local, they are connecting on-premise applications and do not connect internal systems with cloud applications. They are primarily static connections with systems that are internal to an organization, in other words within a company’s firewall.
Thus Hybrid Integration should be considered an important factor when it comes to designing an ESB for your company. Today companies are outsourcing their business processes to cloud applications such as using Quickbooks Online for accounting to Workday for HR to Salesforce for managing customer relationships. So the key question is does your ESB handle integration with cloud applications, does it allow internal applications to talk to cloud applications seamlessly? How about connecting cloud applications to other cloud applications?
For example, there may be a need to build a common service that provisions a new employee into Workday with an internal HR approval workflow.
Similar to how a boat changes speed and direction when its sail catches a wind, similarly your service bus should offer dynamic services that can behave differently based on the type of request sent by the client. Current ESBs have a plethora of services that are static. For example, a service that provides a payload in one format will always deliver that data in that format. An invoice sent by Service A is in XML, what if tomorrow you want that invoice in EDI or in JSON. What if you don’t want to build an intermediary step in your application to transform the data into your format, you want the service to provide you the data in the format that you expect. Having an ability to format your request in a way that allows the target application to understand and convert the data in accordance to your format is key to a dynamic service. In this scenario, the client defines what that service does and responds back with in terms of data and its structure.
There are several examples of how Adeptia ESB is used as a dynamic service library. One of the examples is a Mapping service where the client sends a request consisting of what type of mapping transformation is required, schemas of the source and the target format and where it wants to deliver the response payload. When called the Mapping Service would send the result exactly how the client wants into the location where it wants it.
A bad engine may have multiple issues ranging from putting diesel instead of gasoline, low oil or bad transmission. To resolve this problem a key SOA based integration strategy for ESB would be to address the following points carefully:
i. Involves the ability to connect and receive data
ii. Involves architectural knowledge needed to make the connections to that system (mainframes, databases, custom programs, etc.)
iii. Systems may change over time and the ability to learn from the interactions and be able to work with those changes
iv. Forms and Web-based interaction portals that would allow end-users such as business users, customers to interact with the information
Having a clear understanding of these questions would allow you to come up with an ESB Software that meets your business goals.
The Adeptia Enterprise Service Bus (ESB) model to circumvent ESB challenges: