Integration in the cloud can be completed effectively and without breaking your budget, but not if you try to force the “tried and true” on-premises integration methods into a cloud model.
Integration in the cloud requires a new way of looking at connectivity, an adjustment of IT’s thinking of integration as a one-off task within a larger project. Cloud integration calls for the adoption of a concept called integration lifecycle management, which includes four distinct phases: design, develop, deploy and run.
Integration Lifecycle Management: Design
The design phase requires a radical shift in mindset. This is where you plan for connectivity between cloud applications that can last years, but also change daily. In the design phase, plan for short-term flexibility and long-term agility from the get-go.
Start by defining users and plain language use cases
Before even beginning to design, identify the participants for the entire project and describe the data integration requirements for participants at each user level. Create plain language descriptions of behaviors at the application level, and thoroughly vet all descriptions and definitions.
Design for a rapid pace of change
Cloud applications change quickly, so build in the agility needed to modify integration workflows, data mappings and business rules, and to add new endpoints as needs evolve. This flexibility can be built in with a modular design, which separates the endpoints from integration logic to make it easier to deal with modifications to the API of each application later.
Address performance as a critical feature
Design for performance and scalability needs up front — the development phase is too late. Analyze your requirements for throughput versus latency, performance and function before beginning to develop your integration.
Which applications can rely on batch-based integration, and which require real-time integration based on triggers? Go through this analysis for both migration and integration needs, and utilize change data capture and bulk methods where possible.
Obsess over the data
If you don’t obsess over the quality of the data, chances are you’re missing something. Create a plan to address data quality and duplicates, and look at opportunities to enrich the data — this helps build enthusiasm from users during the deploy/training phase.
The design phase is also where you want to obsess over exactly what information your business users need. To avoid cluttering the application with unnecessary data, start with the key business drivers, and only integrate what is absolutely necessary to achieve the desired outcome.
Integration Lifecycle Management: Develop
So now you’re thinking you can cruise through the rest of the project after all of that time spent designing the integration, right? Wrong. Stay focused during the development phase to build efficiency into your integration process — remember you want this connectivity to last well beyond the initial deployment.
Treat integration as a team sport
The development phase is your chance to create a collaborative work environment. Bring in various application experts to make sure one person isn’t in charge of the whole project.
Even the best developer will make mistakes and incorrect assumptions when working solo, which will cost you in end-user satisfaction or project length/cost. Bringing in multiple experts keeps you on track and saves time in the long run by allowing you to reduce errors and better meet business needs.
Re-use, re-use, re-use
Remember that in the cloud, business users are deploying apps faster than ever, so you’re going to be integrating far more frequently. Be aware of repetitive actions, and look for ways to re-use, cache, build a class, etc. to reduce time and failure points, make modification easier and optimize performance. Document as you go, so the next time a business user requests an integration, you’ll know exactly how to leverage the reusable assets you’ve built.
Integration Lifecycle Management: Deploy
You’re ready to deploy, but don’t jump the gun! This is the stage where everything comes together, so take it one step at a time.
Iterate, test and validate the integration
Take the time to build out environments for build, test and production with a documented and structured change management/version control process. Then use these environment to optimize your deployment with a phased approach, starting with the initial sync and core objects, and then adding additional workloads one at a time.
A phased approach is simpler to validate, and each additional phase should go faster based on previous learnings if you’re documenting properly. An in-line debugger toolset also helps with this process by shortening your cycles, increasing responsiveness and providing more thorough testing capabilities.
Show your work
Continue documenting as you go to keep the process from becoming unwieldy. Strong documentation is critical for long-term staff continuity, and often becomes a big failure point if not done properly. When someone else takes over the integration, the documentation will be their most valuable asset — unfortunately most teams hate writing documentation, so choose a self-documenting approach if possible.
Integration Lifecycle Management: Run
You’ve made it to the finish line! Only with cloud applications, remember that the finish line is constantly moving, meaning your integration project is never truly “done.” With the right prep, you can keep everything humming with minimal investment, starting by applying preventative care methods to maintain healthy connectivity.
Never think ‘set it and forget it’
After a successful deployment, keep tabs on the health of the integration by setting up proactive alerting and automated health checks on the data and environment. A multi-tenant management environment is helpful for maintaining your integration health, as one person may miss something, and with the frequency of changes in cloud apps, the question isn’t whether something will break, but when.
Informative error messaging is critical here — when something goes wrong, do you simply need to update the connections to meet changes in the target-side API, or did something more serious happen?
Did I say obsess over the data?
I said it before, and I’ll say it again — obsess over your data. Build data quality, duplicate avoidance and data enrichment into your run state plan to head issues off at as many points of entry as possible. The more time you spend now, the less you’ll spend on preventable issues down the road.
The Long-Term Integration Lifecycle Management Approach
Following the integration lifecycle management steps I laid out, you can lower the total cost of your integration while turning IT into a true value driver, enabling staffing flexibility and creating an integration that can change alongside applications throughout their entire lifecycles.
How has the cloud affected your IT department? Have you implemented any of your own best practices to integrating data in the cloud? Let me know in the comments or on Twitter at @peterrchase.