Why is integration so important in today’s digital world?

Jon Boxall (JB): I’m a scientist by training; I did a PhD in Microbiology and spent the first few years of my career as a research scientist, before realising I had a natural affinity to computers and moved across into IT. I initially worked as a C++ developer for a commerce firm in the early .com days, worked with several big-name retail clients such as Tesco and M&S.

I moved around a bit over the next few years, mainly in end-client roles, before joining a consultancy called Conchango (now EMC Consulting). I transitioned to architecture because I was more interested in how systems are put together to support a business –  the “big picture” stuff – than coding. I started out as a solutions architect but transferred pretty quickly into the enterprise architect role. My move into integration was somewhat by chance – I took on an interim role running an Integration Centre of Excellence and just stayed with it. Then, a few years ago, I co-founded Wheeve.

JB: I saw a need for a consultancy that could help organisations deliver systems integration well. Most organisations were essentially building point-to-point interfaces through their middleware platforms rather than building interfaces for reuse.  I could see that the root cause of this was that they were treating integration as a purely technical discipline, and as a result they were failing to understand that, to do integration properly, they needed to consider the entire integration operating model – from architecture and design through to delivery and support.

As a result, most organisations would end up with a complex tangle of integrations in their middleware platforms, and those platforms were making integration more complex and harder than before, rather than adding any benefit. Wheeve was founded to help people build effective integration competencies so they could release the true value of integration rather than ending up in a costly mess!

JB: I think there are a few reasons. Firstly, software is becoming more commoditised and software solutions are becoming more specialised. No longer are organisations choosing one-size-fits-all ERP applications; now they are assembling solutions comprising complementary services from hundreds of different cloud options that map more closely to their particular functional requirements. As a result, organisations are building architectures that have many more smaller units of functionality and data that they need to integrate.

Second, the boundary of the enterprise is becoming more porous.  It is now a given that data exchange needs to occur between organisations rather than just within them.  Good integration solutions are at the heart of the IT solutions that meet these needs.

Third, data is king. There is a distinct competitive advantage that comes from having access to high quality data; integration provides this access and makes it available to business users.

Finally, integration is becoming important in today’s digital world because organisations are now expected to deliver seamless, connected, omni-channel user experiences and this simply cannot be done without those systems sharing their data.  Furthermore, modern businesses expect to be able to deliver new experiences much faster than before, so effective, rapid integration capabilities are needed.

JB: In the past few years, almost without question, it’s the industry’s move to cloud; there was a natural evolution from monolithic applications being broken into smaller functional units and then the internet came along, and we started pushing stuff out to that.

With that move to cloud comes a raft of architectural considerations around, for example, data security, network resilience and ownership.  Cloud architectures also offer a whole host of new opportunities such as elastic scalability – something that was unthinkable (or prohibitively expensive) in the past.

There’s also been a big change in architectural terms in how we purchase software – whereas it used to be more common to purchase it, and for those purchasing decisions to be controlled or at least managed by IT, now anyone with purchasing power across the organisation can sign up to lease a SaaS enterprise system. Together these have transformed how we have to think about architecting systems.

JB: In integration, I think an important trend is the move of APIs to mainstream – becoming something that not only senior IT leaders but also the wider organisation is starting to talk about.

An increasing number of digitally savvy professionals now understand the importance of exposing their systems capabilities to clients, customers and business units through APIs. This acknowledgement, of the importance of unlocking data held in silos in order to compose and recompose systems architectures in line with business and customer needs, is a fundamental and positive shift.

I think another important trend has been the rise of the microservices architectural style which promotes decomposing systems into smaller self-contained units of functionality.  This approach enables organisations to recompose their systems landscapes quickly to meet new challenges, business models and customer journeys; something that was not possible with course-grained or monolithic applications landscapes.

However, a microservices architecture isn’t for the faint-hearted as it brings with it challenges around complexity, dependency management and version management (to name a few).  I think there is still a long way to go before microservices are mainstream and I wouldn’t recommend this approach without thought, care and a lot of preparation and training; you can get yourself into an awful mess with it.

JB: For me Digital Transformation is the evolution of an organisation to enable it to exploit new technologies to create new opportunities. Technological change is so rapid now and digital transformation is all about changing the way a business operates so it can adopt and exploit technological advances.

New technology enables organisations to innovate new business models and find exciting new ways to service customers. However, the challenge is that these technologies are often available to all and organisations must move fast to capitalise on them, or risk being left behind.

Digital Transformation is far from just a technology or an IT concern; it’s about an entire organisation being agile, responsive and able to change, and that’s the challenge. I’ve seen a number of digital projects fail to meet their goals because they were just run by IT and were effectively side-lined by the wider business; for digital adoption to succeed, the whole business has to embrace it and embrace the change.

JB: I think enterprise architecture is a critical ingredient of digital transformation; I don’t think it’s ever been more important. A digital organisation cannot survive without relying on technology, and the technological landscape has to be crafted to support the shape of an organisation. Strong architecture is predictive of competitive advantage because it allows businesses to run as they want to, not as the software prescribes.

You see this in digitally native businesses like Amazon who have disrupted entire industries because their systems landscape is architected to work with how the business is meant to run. To be this disruptive, traditional businesses have to build enterprise architectures that won’t stop them from developing in the future.

JB: Loads! For one, the speed at which organisations need to move; there is now a need for businesses to be transform their operating models very quickly, and to have the ability to immediately recompose what they do, and work in an agile way in order to react to changing requirements.

I also think we will see the death of the traditional IT roadmap; in so many industries and businesses, people are not planning more than a couple of years ahead because so much is changing. One of the core tools of the enterprise architect is the roadmap and as this disappears the ability to set direction and offer guidance and governance becomes challenging.

A huge challenge is also the safety and security of data; as we open it up to everyone through, for example, APIs, it becomes easier to for it to “leak out” of the organisation, so security must improve in order to protect it sufficiently.

JB: Broadly speaking, yes.  There are not enough people with the skills required to move old-fashioned businesses to being digital. These skills are not about primarily technical; the skills that are missing are change management, organisation design and architecture skills. Organisations need to completely turn themselves inside out and change how they operate and there are not a lot of people who can do that well.

JB: I think the biggest gap in integration is Integration Solution Architecture. Integration is more of an art than a science; the real value comes with designing the integration solution and getting the right pattern for the problem to enable the reuse of assets.

Once you’ve done that you have to actually implement it, but usually that’s not the difficult bit. If you haven’t got a set standards and policies then those implementations can become complex, difficult to manage and won’t support reuse.

JB: I think finding people with the right mind-set is key; people that get integration patterns, architecture and the technology stack; probably in that order. It can be difficult to find people with this combination of capabilities, but if you get people to learn a set of patterns for integration then that’s a good start. But, you will still need to find at least one integration specialist who can help you grow a strong team.