The IT infrastructure of most organizations is a result of years of adding and subtracting components to meet changing needs. Organic growth resulted in a jungle of disparate species that talk to each other on a one-to-one basis — but everyone in the jungle needs to communicate with each other in order to thrive and grow. This kind of disparate infrastructure makes it difficult to be nimble and agile in response to changing needs.
Enter the age of Service Oriented Architecture (SOA) — and the enterprise service bus (ESB) as the knight in shining armor. Gartner VP Roy Schulte coined the term ESB in 2002. An ESB is an architectural pattern that provides loose coupled integration between disparate applications. It is unique and encompassing in its ability to incorporate any organizations’ legacy systems, point-to-point messaging systems, as well as newer commercial off-the-shelf (COTS)-based solutions. It links people, processes and information to provide better alignment between IT and business goals.
The advent of modern software development patterns of microservices and the API economy, coupled with cloud and mobility, led to a decline in monolithic systems and a move away for any large centralized patterns that could create a bottleneck. The death of the ESB began as the middleware market declined and organizations moved away from unified hub-and-spoke IT environments. However, let us look at some of the emerging architectural patterns as a result of the innovations:
- Enterprise integration today has become simpler on the surface, but more complex as we dig deeper.
- There is still need for encapsulation of legacy components, mediations and routing.
- Interoperability of data through scalable and efficient transformations remains an issue for all large organizations.
- Organizations face an additional challenge to now integrate their on-premises solutions (e.g. Oracle talking to SAP) along with the need to integrate Software-as-a-Service (SaaS) delivered solutions in the cloud, especially in cases where built-in adaptors are not available. They need an integration Platform-as-a-Service (iPaaS), which is essentially an ESB.
A number of additional aspects need to be examined when building integration architecture solutions: messaging format support, transport protocol support, routing and transformations, performance and quality of service, scalability, cloud capability, API management and integration, security etc.
Thus, the concept of an ESB as an architectural pattern is certainly not dead. Instead, it has been resurrected with new names and counterparts. In fact, it is more relevant than ever before and part of the future hybrid integration architectures. The ESB is here to stay although it has started assuming new roles in the fast-growing enterprise of API and cloud spaces. ESBs are now being used to bring together multiple cloud arrangements (one provider for infrastructure, another for platform, others for applications) that are integrated and talking to one another via an ESB, data centers running ERP, and the public cloud. Some of the more common evolution patterns of the ESB include:
Related Article: Have Microservices Rendered SOA Obsolete?
1. The Federated ESB
The problem today is organizations often have multiple departments, sometimes even each sector or department location, selecting its own ESB or separate messaging implementation. This leads us to the concept of a federated ESB or architecture solution of heterogeneous ESBs that will become the central control point for the underlying messaging strategy of any organization. One of the key benefits of the ESB is the ability to decouple integration logic from application and process logic. So, by definition, an ESB is by its nature a distributed architecture.
While it is ideal to have a unified ESB, the reality is requirements for departmental, business or enterprise deployments differ. While we all recognize the ESB as a core enabling technology, multiple configurations and topologies are possible. Some may consider having specialized ESBs serving different functions. And quite often, within the organization, you might find ESBs from different vendors. Thus, the integration among different ESBs should be a key consideration in selection criteria. Some of the reasons an organization would pick a federated ESB solution over a single solution are:
- Different “domains” in the enterprise.
- Distributed business and funding models.
- Distributed geographical locations.
- Distributed governance.
- Differing ESB requirements best met by different products.
- Acquisitions have existing ESB infrastructure in place.
- Decoupling to allow asynchronous development and deployment.
Many ESBs from different vendors now proliferate within organizations. Management of heterogeneous ESBs will become the central control point for dictating which underlying ESBs get deployed.
Related Article: Google's Deal With Red Hat Enables a 'Real' ESB
2. ESB in the Cloud
The emergence of cloud-computing models is steadily changing the way in which we build distributed systems. Cloud computing is enhancing and influencing our ability to host enterprise services in private or public clouds.
A cloud-based ESB, better known as iPaaS, can be used to enable capabilities, such as API management, message routing, publish‑subscribe transformations and service orchestration. In addition, we can also blend cloud-based and general business-process integration via a hybrid integration infrastructure, combining the benefits of centralized management with the efficiency of distributed architectures. Cloud-based ESBs come with few challenges in terms of security and architecture. Infrastructures running in cloud environments usually have a variety of existing connections to external as well as internal environments. When operating in a cloud-based environment, it is important to keep the following points in mind:
- When an ESB is opened up to an external cloud, you need to consider issues of scalability, security, latency and network connectivity.
- Making middleware, such as an ESB, available in the cloud is an important step towards achieving deployment flexibility. However, make sure that available cloud services provide a governance structure for the ESB.
- Message encryption and security is less of an issue in cloud architectures that live behind the firewall because the enterprise controls the network, but are major issues for public clouds.
- Latency can become an issue when moving an ESB from private to public clouds because an enterprise may need to work with transport protocols for which the ESB was not originally configured.
- One of the big things in the cloud is being able to deploy without really having to know what the exact IP address is. However, while loose coupling can be a goal for any ESB, those running in cloud environments can have an even wider variety of connections to make between internal and external services.
Related Article: 7 Considerations for IT's Never-Ending Integration Challenges
The Future of ESBs
With the advent of cloud and virtualized technologies, the ESB market appeared to be declining, viewed as a centralized mechanism that only added to enterprise complexity. However, the ESB has evolved into a hybrid pattern in the fast‑growing enterprise of APIs, microservices and cloud infrastructures. ESBs are now being used to integrate legacy components, bring together on-premises data and cloud SaaS models, as well as provide iPaaS ease of use and pay-per-use models for smaller enterprises that lack the internal expertise or money for legacy ESBs.