Two Dimensions of Enterprise Integration

In a complex enterprise such as a bank, two topologies (bus and hub-and-spoke) often need to be simultaneously deployed in order to achieve Enterprise Integration.

If applications are integrated in a point-to-point manner, the number of interfaces and the associated complexity grows rapidly with growth in applications (approaching n-squared, where n is the number of applications). Another implication of point-to-point connectivity is the tight coupling between applications. When a new application is introduced, other applications have to be aware of the precise data format expected, the integration style, the protocol used, etc. Integration styles can range from service calls (which may be RESTful web services, SOAP web services, CORBA or RMI calls etc.), to message-based integration (e.g. MQ), file-based integration or even using a shared-database for integration. Such tight coupling strangles flexibility, or the ability of the enterprise to iterate.

Bus topology, in the form of an Enterprise Service Bus is typically used solve these problems and to decouple interfacing applications. 

Using a bus, interfacing applications are shielded from the protocols, message formats,  physical endpoints etc. used by their peers. Additionally, the bus can be used to publish messages to multiple subscribers, and even to implement an Event-Driven Architecture if so desired.

However, this level of decoupling is not sufficient. If an application needs to provide data (say transactions or portfolio positions or customer data etc.) to 40 other applications, generally it still needs to extract the data independently for all 40 applications and feed it over the bus. This is because of variation in the needs of the 40 applications (such as expecting incremental data vs. a full set, end-of-day vs. near-real-time, reconciled/clean data or raw, etc.).

To solve this problem, Hub-and-Spoke topology in the form of a Data Hub is typically created. 

This might be called an Operational Data Store (ODS) and contains commonly used data which is replicated from source systems to the Data Hub. Applications form the spokes and obtain common data from the hub. With a Data Hub, the next level of decoupling occurs. Data consumers do not need data producers to be available at the precise moment they need the data, so availability of applications goes up. Applications are shielded from the volume of data requests, and can focus on executing business functions without impact to performance. Apart from scalability, another advantage of the Hub is that data consolidation and data enrichment/correction can occur in the hub.

Apart from a Data Hub, a Service Hub is often created to house common services that access data in the Data Hub and/or invoke other applications to fulfill requests (including Service Orchestration). The services in the Service Hub can be used to implement common semantics across the enterprise.

The Data/Service Hub has be overlaid on the bus i.e., the Data/Service Hub itself has to be interfaced to the bus.

About Simple Architectures for Complex Enterprises

I read with great interest Roger Session's new book "Simple Architectures for Complex Enterprises".

The area that is most relevant to me regards the reuse of ABCs (i.e. units of process and technology components, that Roger calls Autonomous Business Components).
Let us assume that there are differences in the business processes in various regions. The way that loans are processed are different for New York, Asia and Frankfurt, leading to three ABCs, each specialized for regional business conditions (legal, administrative, cultural, environmental).

The alternative is to have one Global Loan Processing system

The question is - which avoids complexity better?
If I start with the Global system, and see high complexity due to it having to handle different regions, (i.e., regional variables add high complexity), I am tempted to partition the system into regional systems. The complexity falls from N*N*N to 3N as the systems can be tested and changed independently.

On the other hand, if I start with regional systems, I am tempted to combine the systems in a bid to reduce costs (and complexity) - as mentioned in the book.

I asked Roger to comment on this apparent contradiction.

Roger emailed me back -

I don't know if there is a simple answer to your question. It comes doesn to how similar the three systems are. And it is also important to keep in mind that just because architecturally we are considering systems to be different, that there aren't opportunities to reuse work at the implementation level. And in some cases, we can break out that part that makes them different into other ABCs.

My current rule of thumb is, "When in doubt, make them different". I think in general more harm is done by trying to force things to be the same than allowing them to be different. But this is very dependent on individual circumstances, as the case study in my book shows.

I find this perspective highly pragmatic and useful.

Roger has a blog related to the book as well.

The Toyota Way and Software companies

Arun recently pointed me to a news report that he found particularly amusing. It is about an Indian software company trying to adopt the Toyota way. Unfortunately they seem to be getting it wrong.

Since August 2007, employees of LogicaCMG India have been maintaining hourly logs of their activities at work. This includes the nature of work, the status of the work, tasks completed, meetings attended; indeed, just about everything, including the time spent at the cafeteria.
It then shows how many hours were spent on each production activity including support , testing, development, project management, review and rework ; on preventive activities like process improvements and self-development ; on organization support activities like interviews , training, meetings and customer visits; and finally on idle time like waiting-for-work or personal time.
The report adds up all that to give the productive hours utilized. It then subtracts that from the previously calculated "available hours", and Verma gets the unutilized time. In the one instance we were shown, the unutilized time stood at 1.5 hours out of a total of 656 available staff hours, or 0.23%.
"I can check such things project-wise, location-wise , competency-wise . I can check the changes over time. I can exactly identify who was responsible for a delay in a project. In fact, we have broken things down in a way that everything can be quantified," says Verma.
Verma of LogicaCMG admits there was resistance when employees were asked to log in hourly details: "There were questions like: I'm doing intellectual work, why should I do such things." But over time, everybody appears to have got into the groove.

To me this resembles Taylorism and Scientific Management . I thought they went out of fashion in the 1950s.

Instead, a focus on Lean Operations might be called for -

The Principles of Lean

Improve Continuously - Get your employees to suggest ways to improve and implement the good ideas on a continuous basis.

Involve Everyone - Software programmers dont like to enter detailed logs of how they spent every minute. Especially if that data is being used to find out "who delayed the project".

Eliminate Waste -

There are 7 types of waste ("S W I M T O O") which can be recognised in most organisations:

• Scrap (or Defects) - Do you detect defects early? Do you practice prototyping? Can your employees "stop the production line" if they see problems?
• Waiting - Is your staff kept waiting for signoffs, approvals etc.?
• (Excess) Inventory - Are you understaffed/overstaffed in a particular area - E.g., is your Business Analyst swamped with volumes of paperwork awaiting processing? Are your servers underutilized?
• Motion - Do you have unnecessary movement and travelling?
• Transportation - Are your teams flat enough? Does information make unnecessary hops? Does your staff have to keep switching between unrelated projects?
• Overproduction - What steps in your process don't add real value?
• Overprocessing - Do you require six sign-offs when three will do?

Reusable processes and SOA

Joe McKendrick has an interesting post on SOA rituals that made me smile.

I agree that hatered against the 'poor beleagured' Silo may be misplaced. Some boundaries are necessary for you to be able to act and implement. If everything in the enterprise was to be implemented and controlled globally, very little could get done.

Joe then describes what he thinks SOA really is - "it is a scope of architecture, like Building Design vs. Urban Planning". I disagree. I think the "Urban Planning" space belongs to Enterprise Architecture. With SOA, I don't worry about what lies behind the service interface. When doing EA, I consider all the technologies and applications and systems in the Enterprise. I look at the gaps in the Business Architecture and look for ways to fill the gaps. I do this irrespective of whether they provide loosely coupled re-usable services or not (which is what I see SOA as doing).

On Reusable processes and SOA

James McGovern has a post where he casts a big shadow on the prospect of reusable services. He thinks reusable business rules might actually work but not reusabe services. In his experience, applications get rebuilt or rewritten if they have to be reused.

The way SOA is generally positioned, services should encapsulate reusable business processes. I don't find too many potentially reusable services when I look around.

• They require access to data. Enterprise data is often local, not global. Regulatory requirements might prevent you from having global data stores (especially in the banking domain.) In any case, data for UK is likely stored in UK and data for Malaysia is likely stored in Malaysia.
• The processes differ. The business process in one location may well differ from another location. There are differences in the business environment (cultural, political, administrative, regulatory, legal, geographical, economical etc.) that require that the process be different.
• They involve people and roles. Large enterprises usually have global employee databases. However, the employees are organised differently and have different roles - again due to different business environments.

These factors greatly diminish the set of services that can be simply reused without customization.

Moving on to other topics

A lot of write offs have occurred since my last post

Some lessons that could be learnt from the subprime crisis -

• Ways to generate wealth must be sustainable. It is not sustainable to for an industry to "sell lemons to each other indefinitely" as some have described it.
• By slicing and mincing risk and passing it all around, the overall risk does not go down.
• Models for risk must seriously take liquidity into account. What is a bond worth if no one wants to buy it?
• Risk (i.e. loans) should not be created by parties that have no stake in what happens subsequently.

Residential Mortgage Backed Securities (RMBS) - A brief tutorial

When one takes a loan to buy a house, chances are high that the loan will be bundled with a large number of other similar loans and sold to investors as bonds called MBS or Mortgage-backed-securities.

Why MBS?

MBS is about transferring risk. Risk is transferred from the bank (or mortgage company) that made the loan to investors who have an appetite for risk and wish to profit from it.

The profit comes from the interest payments on the loan. The risk comes from the fact that the loan may not be repaid. Or, the loan may be repaid early (!) which is likely if it can be refinanced at a lower rate of interest. So, changes in interest rates also add to the risk for investors.

Also, banks love getting these housing loans off their balance sheet. Banks are required to hold a percentage of their loans as reserve and using MBS, they can comply with the reserve ratio requirements while freeing up the capital.

In addition, the investment banks issuing the mortgage-backed bonds profit from the difference in interest rates paid by individuals who take a housing loan and interest rates expected by investors who invest in mortgage-backed bonds.

Slicing and dicing

The pool of mortgage loans is split into slices (also called tranches) of varying levels of risk.
As shown is this diagram, the highest rated tranches offer the lowest risk and the lowest returns. On the other end of the spectrum the most risky tranche (which is usually unrated) offers the highest returns.

Investors can choose the tranche based upon their risk appetite. A pension fund might buy the “senior” conservative bond while a hedge fund might buy the “junior” speculative bond.

Risk protection

It is important to understand that the lowest tranches offer protection to the senior tranches.

Any losses will be absorbed by the investors of the most junior tranche until the value of their investment reaches zero. If this occurs, investors in the next junior most tranche absorb further losses, and so on until the senior most tranche is reached.

The lowest tranche contains loans made to subprime customers (i.e., who have a blemished credit history and have a high chance of defaulting on their mortgage payments).

The crucial lowest 10%

The lower tranches make up around 10% of the entire MBS. The catch is, the lower tranches have to be sold in the first place. If the lower tranches cannot be sold, the higher tranches cannot be sold either, and the entire deal is off!

Who buys the risky stuff?

CDOs (Collateralized Debt Obligation) buy the riskier parts of an MBS. The next post will discuss CDOs.