SOA Governance in a mashed-up world

Or, how to deliver innovation with control through Implicit Governance, by Kelly Emo, HP SOA Product Marketing Manager

Why SOA, why Web 2.0, why now?

Two key waves are coming to shore for IT. One has had a longer incubation although it is early in its broad scale adoption and that is Service Oriented Architecture. According to IDC Research, the percentage of SOA-based applications will double within two years [1]. The other wave has just become to materialize but is crashing upon the beach of IT quickly and with great force and that is Web 2.0.

An interesting dynamic is occurring. IT is busy and focused at modernizing and rationalizing its infrastructure, dealing with leveraging legacy infrastructure, changing point-to-point integration to more flexible loose-coupling, and breaking up application silos to become more modular and ultimately responsive – reflecting both the benefit and challenge of SOA. SOA promises IT agility and responsiveness to the business but also can be disruptive to existing IT organizations and processes, require new skill sets and the adoption of new processes and best practices. As a result, SOA realization can be slow. IT has started embracing and using proven SOA disciplines such as identifying key re-useable capabilities and exposing them as standards-based services. However, end-users don’t have time for planned and meticulous processes and they are taking many of their needs into their own hands using “wild west” approaches such as mash-ups and Web collaboration – Web 2.0 approaches.

Web 2.0 is being adopted rapidly by end-users because it offers lightweight, flexible and essentially frictionless technology that empowers non-technical end-users with the ability to personalize their IT environment and experience. However, enterprise adoption of Web 2.0 is being hampered because it has no obvious control mechanisms. Leveraging a SOA approach with governance can offer a new level of control. In many cases, these emerging mash-ups are resulting in big productivity gains, grabbing the attention of CEOs, and becoming business critical applications in their own right – regardless of buy-in from IT. It is inevitable that the business will put more pressure on IT to embrace and support these highly productive solutions.

In IT’s hands: embrace or stifle innovation:

IT is at an inflection point of “hero or zero”. Because of the productivity gains and innovative tooling and programming models being delivered from the Web 2.0 community, users will be inextricably drawn to innovate this way. In addition, rogue applications will proliferate with or without the influence, support and best practice impact of IT. IT has an opportunity, right now, to plan for and embrace the new “wild west of web 2.0” and combine it with both the productivity and architectural best practice of SOA by effectively combining innovation and discipline. IT can manage Web 2.0 with the implicit planning and control of SOA so that they establish governance without the Web 2.0 consuming participants being aware of the influence of IT. Users can still feel free to innovate and mash-up, but IT can manage the consumption of resources and the resulting operational integrity.

How to grow innovation without undue risk:

How is this possible? One of the key enablers for successful, managed yet still “free” web 2.0 is implicit SOA governance.

The development of Rich Internet Applications, mash-ups and other composite application styles being driven through Web 2.0 is driving the need for increased automation and participation in “implicit” governance processes. Today, many enterprises are investing in and establishing appropriate “explicit” SOA governance to manage and expose SOA-based re-useable services with their known set of consumers. Services are planned and built by service owners while going through a governance lifecycle of establishing checkpoints and policy conformance. When a consumer wants to re-use a governed service, they discover the service and collaborate with the service provider – negotiating and establishing a provider-consumer contract with explicit governance terms and expectations. However, with the rise of Web 2.0, the range of potential new consumers is now expanding significantly, with the potential to become a huge wave – with a big splash and potentially big business impact.

Furthermore, the Web 2.0 community isn’t particularly interested in direct participation with what could be considered a heavy weight governance process which might slow adoption and restrict innovation and flexibility. They simply want to find services and mash them up to support their objectives. Web 2.0 with implicit governance allows for broad scale consumption activities worldwide by all types of consumers. By connecting these new consumption patterns with implicit governance capabilities, classic IT is able to maintain enough control over services, including service usage tracking, capacity planning, automation of service catalog and web site population, and notification of new releases, to keep the integrity of their infrastructure and service delivery in place.

The Web 2.0 dynamic parallels the early days of the Web. When large organizations first put up Web sites, they didn’t have the controls in place to ensure quality of service and an external event outside of their control could cause a spike in traffic to bring down the whole site. IT realized that they would need to put in place web site traffic and access governance and controls to manage access and increase the throttle when needed and without the users knowledge or interruption of service.

Web 2.0 and SOA governance is a similar scenario, although it embraces a much larger set of processes and roles.

Today, when services are created and an organization provides a service catalog for potential consumers, these consumers are a known set and are part of what is essentially an "explicit" SOA governance process. The service consumers are project teams in organizations or partners and are part of a planning process. But, in the “open space” world of Web 2.0, the way in which potential consumers typically interact with a service catalog is much looser. They could come from inside an organization, from a partner or from the consumer web at large. Essentially these consumers interact in a WYSIWYG (what you see is what you get) kind of consumption pattern. The companies that advertise some of these kinds of services today, such as Amazon, eBay, and Google (http://code.google.com/), seem to have developed complete Web sites to support their "catalogs." Furthermore, each of these innovative companies has WYSIWYG terms of service for service consumption and can support a huge number of unknown, unplanned consumers.

SOA governance in a mashed-up world

With this in mind, enterprise IT can embrace a similar approach to support its own brand of “mash up” projects and partnerships. For formal, back-office inter-domain projects and integration projects, IT can establish formal and mature explicit governance processes, while at the same time, exposing key services as “mash-able” for front-office innovation and situational applications. These mash-able applications can still be governed but in a much more implicit way. Using SOA governance, testing and management technology tools for assistance, IT and take the following key steps:

1. Estimate and determine expected capacity
2. Author and create a finite number of WSYWIG contracts and
3. Establish and automate a simple, well-defined process for all consumers to dynamically accept contracts and interact with services using Web 2.0 technologies while being contained in an acceptable boundary of response time, performance and other attributes defined as part of the WSYWIG agreement
4. Proactively capture and track the consumption load on the service

In addition, with SOA governance in place and SOA governance automation software, IT can determine when they are getting close to full capacity, proactively allocating more resources for additional contracts before they hit the wall.

What does implicit governance look like?

Implicit governance requires several key processes to enable IT to have the visibility, trust and control they need over its dynamic services environment to enable composite and situational applications made up of services that deliver business flexibility with operational integrity. It starts by embracing governance across the service lifecycle while accommodating for the fact that certain services will become mash-able and consumption relationships less predictable and more dynamic. With this in mind, here are some key considerations and steps:

1. In the planning process, the business owners and enterprise architects collaborate to determine the key business services that will support the business strategy. Assuming that the business is planning to foster innovation through new in-house business processes and end-user applications but also by allowing consumers both inside and outside the organization to participate in the business collaboratively, the business owners will need to work with the enterprise architecture team to identify which services will need to be candidates for “mash-ups.” This attribute then becomes a key piece of metadata in a SOA governance infrastructure that supports service discovery and consumption.

By identifying a service as mash-able, this will alert the enterprise architecture team that a unique contract type that needs to be associated with that service. This will indicate that a certain percentage of consumers will be unpredictable and possibly from outside corporate boundaries and will expect the WSYWIG contract type. Thus, the service will need to have consumer-provider relationships of both types, internal more structured contracts for back-office or planned consumption relationships and light-weight WSYWIG contracts for mash-up consumption.

2. As the frequency and instantiation of services that can be mashed-up proliferates, it is also important to equip the enterprise architecture team to auto-discover services that have the potential to be mash-able and there are several technologies available today to help with service auto discovery including service discovery inside HP’s Business Availability center for SOA.

3. Next, before any service that has the ability to be offered as “mash-able” is delivered into production, it needs to be carefully load-tested so that the IT organization can determine its carrying capacity. This then allows IT to be able to track what load the service can handle and track the number of new consumption relationships being established with their estimated load impact. This gives IT the ability to proactively determine when consumption load will outstrip the service’s ability to handle it and then make a business determination whether additional resources should be provisioned to scale the service or whether consumption should be throttled.

4. This quality and performance testing is part of an automated service lifecycle process so that no service is provisioned as mash-able until IT has a clear idea of its capacity and can establish confidence around its ability to handle dynamic consumption. Once the load testing is complete, that metadata is associated with the service for review before sign-off to the next stage.

5. At this point, the identity of the service is mash-able, and contracts have been created to support both consumption models – planned and dynamic/mashed, and have determined a clear understanding of capacity and how many consumption relationships the service is able to handle. Now, the service can be deployed either to a staging or run-time environment. And, Web 2.0 composite application or mash-up environments can be made aware of this service by discovering it and its associated taxonomy as a mash-able type service. Consumers are now able to use the service in the creation of situational applications.

HP is enabling this process to be more automated through attaining seamless interoperability between the governance platform and the key tooling environments that enable end-users to create new dynamic applications and mash-ups. HP and a variety of partners are collaborating on the Governance Interoperability Foundation that enables interoperability between a SOA governance platform with its system of record and Web 2.0 tooling. As an example, HP has integrated with leading Web 2.0 tools from vendors such as JackBe and Nexaweb so that services can be auto-discovered and made available to the creative Web 2.0 end-user development environments that support creation of situational applications and mash-ups.

6. Meanwhile, IT is now tracking a service usage pattern and performance service levels against its predicted capacity. When the number of consumption relationships, service message and transaction traffic and performance load gets closer to it’s top capacity that was determined during the load testing phase, operations can collaborate with the enterprise architecture teams to determine if additional run-time resources need to be applied to grow capacity. If that need is determined, a new change governance process can be instantiated for testing the additional load carrying and perhaps updating service endpoint resolution with new higher capacity end systems or load balancing techniques. All of this r is invisible to the mash-up consumer who is completely unaware that they are participating in a mature governance process.

By establishing this level of service lifecycle governance, management and control, IT has the confidence that it will maintain operational rigor while extending flexibility and agility to internal and external constituencies.

The alternative: risk-adverse-based “lock down”

So what happens if IT does not plan for the impact of Web 2.0 in their SOA governance processes? On one end, IT can plan for Web 2.0, embrace its needs and make it part of governance and control processes, ensuring for operational integrity while maintaining consumption flexibility. On the other end, IT can go into a state of fear-based lock down. In this state, IT locks down their services and does not allow for Web 2.0 consumption. However, necessity is the mother of invention and users will find a way. Users will tunnel through fire walls, find back doors and openings and all of these techniques will have unexpected and usually adverse effects on performance, reliability and security. IT will be forced to, in knee-jerk and frantic response, scramble to accommodate these newly productive situational applications and instead of being the champion of business innovation, IT will be the roadblock. However, eventually, Web 2.0 will happen and the best business outcome will be to embrace Web 2.0 while enabling control and governance for IT to meet their service level agreement expectations while delivering business agility. The combination of Web 2.0 and SOA is best suited for this.

Plan for the best of both worlds – innovation with discipline

What’s the call to action? IT needs to understand the new Web 2.0 trends and begin an open dialog with business users to understand the types of situational applications that may be needed for business innovation. In addition, when IT is putting the organization and processes in place for effective explicit SOA governance, it should be planning for both models: explicit governance where the contractual agreements are known in advance and both consumer and provider participate in service lifecycle governance activities, and implicit governance where IT has service provider controls in place but from the consumer’s perspective so they can easily and intuitively interact and mash-up the service without perceiving they are part of a governance process at all.

By planning for implicit governance, IT gets the best of both worlds: the ability to effectively deliver reliable, resilient core business services and open up key functionality for the innovative web 2.0 and mash-ups solutions that can only come from the minds of creative business users both inside and outside the organization. Web 2.0 and SOA are intrinsically synergistic and will give IT the ability to truly become the champion of business innovation.

Summary for the Business:

• Realize that Web 2.0 will happen and it will enhance digital business innovation
• Web 2.0 can be supported with control and operational rigor through investing in and establishing the people and processes for implicit governance at the beginning
• Users will see flexibility and access to new mash-able services while IT will deliver consistency, performance and availability
• IT can become the champion of digital business innovation, not the obstacle

Summary for IT:

• Implicit governance must be planned at the beginning of service design – put the processes and systems in place now
• Ensure that Web 2.0 enabled services can be discovered and identified as “mash-able” by consumers
• Put the processes and enabling technologies in place to test and understand capacity and performance limits on consumption for services and put in the capability to manage usage metrics in real-time with controls so that IT can proactively determine when to increase capacity with no user disruption

Footnote:
[1] IDC Developer Network Survey, 2007, n=490 IT professionals