Traditional integration approaches, like that of Enterprise Service Buses (ESBs) and Integration Platform-as-a-Service (iPaaS), rely primarily on proprietary connectors. This architecture served its purpose in the late 1990s when there were few participating applications in an enterprise, and most of them had either no interfaces or at best interfaces based on proprietary protocols. However, this reliance on connectors has continued to persist, primarily because the practice is an extremely profitable business model for most integration vendors, who price and sell connectors separately from their core integration engines. Vested commercial interests have led to muted innovation in the integration industry, which still tries to vend these add-on proprietary connectors.
This decade-old approach to integration, bogged down by proprietary connectors, needs to be revisited. The enterprise has evolved and the technology alternatives available are far more flexible and cost effective. In this post, I bring out the limitations of this connector-based integration architecture and provide some initial thoughts on what an alternative integration approach should look like.
The Fragmented Digital Enterprise
The modern enterprise is drastically different. With the advent of cloud and SaaS, the average number of applications and services used by an enterprise typically ranges from 600 to 800 and is growing at an exponential rate. While these applications need to be integrated, businesses often do not have the appetite nor patience to wait an additional 3-6 months for their integration scenarios to be implemented by IT. If a connector does not exist, it takes far longer to develop a custom adapter. Developing freshly-coded connectors invariably leads to more bugs and errors – which implies either taking on higher risks or else buffering in more time for extensive testing.
Moreover, with “mobile first” initiatives taking precedence within digital enterprises, it is not only necessary to integrate these applications, but also provide integrated data sets and services available as APIs for internal and external developers. The mobile dev teams need to consume these APIs to create mobile Apps. Also, to be part of an emerging digital eco-system, enterprises need to make available these integrated datasets to their partners so that they can offer value-added connected services to their customers. Traditional ESBs and most iPaaS do not provide an infrastructure to publish and secure APIs.
The increasing number of applications, business needs for quick turnaround, the reliance on proprietary and expensive connectors, and the need to serve digital channels like mobile and IoT are all factors that ESBs or the ESBs in the cloud (iPaaS) cannot provide. It requires us to investigate and adopt a new approach to integration. This is what leads me into the discussion about Connector-less Integration.
Applications from the late nineties often shipped without programmatic interfaces or if they did, it was often in a proprietary format, e.g. BAPI for SAP etc. This led to the connector-based approaches in ESBs. These connectors usually tunneled directly into the associated databases and exposed the related datasets for integration. The connectors had to be coded and were deployed as part of the ESB runtime.
However, most SaaS and modern applications have standard based REST APIs, SOAP services or some other variant of commonly accepted standard interfaces. These standard interfaces can be directly used from within a lightweight agile integration gateway that provides data mapping, orchestration, transformation and policy management. This API-based integration architecture is becoming a reality and a much-needed welcome alternative to proprietary connectors. For ease of use and drag-and-drop user experience, the APIs can be configured (not coded) and made available as out-of-the-box library of hooks available within the integration eco-system. At SOA Software we call these APIHooks.
As new applications are provisioned, APIHooks can be configured on the fly. An APIHook is a combination of interface definition, the associated bindings and policies. The policies ensure security and compliance to enterprise IT mandates and industry and regulatory requirements. APIHooks can be configured, saved, exported and imported, with the need of writing any additional line of code. The Cloud Integration Gateway can be configured to leverage these APIHooks and build the integration logic focusing on the business logic and orchestrating the integration scenario at runtime.
This concept of APIHooks, though not novel in itself, but with the ubiquity of APIs and the demands of a fragment digital enterprise, has become practical, real and achievable for most integration scenarios. There are still some cases where a connector-less integration alternative might be a stretch, but similar limitations exist with ESB and iPaaS, wherein no vendor provides connectors for all the possible applications and services. On the contrary, the APIHooks based approach is more open, agile, faster, cheaper, and applicable for most modern SaaS and cloud applications that have standards based interfaces.