Its not business-as-usual for us refer to biblical stories to understand the challenges being faced for technology integration, but then its not often that such tectonic shifts happen in the data and its integration space to be called as the next Industrial Revolution.
Of course, by now everyone & their aunts have heard about big data and how it is changing everything around us. For those of us who work with organizations in different industries, sizes and use cases (aka the Integration Architects), the challenge is how do we help organizations handle this flood of information and still, make sense by integrating their customers, ecosystems & operations using an ever growing set of 3-letter acronyms that technology providers throw at this challenge.
For a business leader working through the exciting realm of Digital Customer Experience using people facing technologies, the challenge at hand is pretty different from someone trying to streamline manufacturing operations using IoT or “Things” which in turn is significantly at variance with a procurement specialist looking to enable on-demand procurement by integrating cloud based procurement exchange applications with on-premise financial system. Obviously as the integration goals are varied, so are the integration patterns and the challenges. Hence the analogy of organizational leaders today feeling about the data deluge and their own “arks” of technology solutions with which they are trying to solve their respective challenges.
To put the issue in context, probably the most useful framework is provided by Gartner and the underlying 4-V’s of big-data ie, Velocity, Variety, Volume and Veracity. Simply put, these define the different dimensions of the data & its integration challenges for modern technology landscapes.
- Volume dimension describes the evolution from MB / GB sizes to current TB / PB scale of data.
- Data Variety has just exploded in recent years, growing mostly from table / DB based data to social / mobile / web based photo / video / audio based unstructured data .
- Velocity is being driven by need for real time / near real time data & analytics from yesteryears periodic / batch based integrations.
- Probably the one which is most underappreciated but is definitely growing in importance is the Veracity dimension, which describes the data reliability in terms of accuracy, probability, efficiency and exposure.
We used this framework with typical enterprise integration patterns of Application Integration, People Integration and now exploding pattern of IoT / Things integration. Based on conversations with executives, architecture & implementation experiences and good old “crystal ball gazing”, we came up with our own scores for each of the 4-V’s mapped on the integration patterns.
Sharing at a macro level the results (which should not be surprising). What is clearly evident is:
- Applications integration (A2A / B2B / Data / API / Structured etc) have their biggest driver as the growing volumes being sought, with velocity being the other key driver
- People integration (B2C / B2E / B2B / Social communities / GUI etc) is being dominated by the variety of data (structured / unstructured) and also, by the ability to validate the reliability of the inputs
- IoT / Things integration (M2M / Bots like Alexa / personal devices like Fitbit and other increasing items) is driven by sheer velocity of the data and again, need for validating the veracity of it.
While these are very macro level scores which should be refined to be usable, probably most practitioners of integration will agree that broadly the Patterns -Drivers mapping looks rational.
Next logical questions from an executive responsible for such an integration might be: This is good, but how do I use it with my current / near future technology portfolio?
- Will my EAI hub work well with the API’s model I am looking to monetize for real time management of fleet logistics?
- What do I need to change so that my B2B portal works for the new B2C / mobile app we are launching for Digital Customer Experience?
- How will I secure my on-premise financial applications when they start integrating with cloud based procurement exchanges for on-demand buying?
All of these queries and probably many more fall in the major integration need which we see evolving already, ie, need for increasingly Fluid-Integrations.
This need for increasingly Fluid-Integrations is driven by organization imperative integrate across organization boundaries, amorphous formats, different technology protocols, disparate consumption patterns and ever newer data elements.
In the next blog in this series, we will leverage the Fluid-Integrations model to identify specific components which are suitable for each of these challenges and also, refine the integration patterns more granularly for architects to connect the dots better. Keep reading