Basic Concepts
In the first part of this article “Introduction to Complex Event Processing I” we saw some of the most important principles of CEP. This section will provide a detailed analysis of the core concepts upon which complex event processing is based.
Events and Causality
An event is an object that represents or records an activity that is happening, or is considered to be happening. |
The result of the weather forecast obtained by running a weather simulator, the confirmation of a purchase recorded in Carrefour’s website, data recorded using RFID sensors to make sure that the suitcases are being correctly routed through the airport’s baggage handling system, a missing signal indicating the possible loss of a suitcase.
This definition refers to event objects that describe real or virtual events and are processed in information systems. These objects are any kind of data structure from strings to complex data types.
Sometimes we talk about events and messages indistinctly, but the truth is that an event contains more information than a mere message. Namely, the activity that happened, when it happened, how long it lasted and how it relates to other activities.
The table below shows a small subset of interrelated events:
Happening | Start time | End time | Triggered by other events |
Event 1: Tokyo announces that for various socioeconomic reasons it cannot take on the 2020 Olympic Games. | 1 August 2014, at 20:35 | 1 August 2014, at 20:45 | Event 0: Sudden 200-point increase in the Japanese risk premium. |
Event 2: The IOC holds an emergency meeting to choose a new venue from among the finalists. | 2 August 2014, at 09:00 | 2 August 2014, at 20:45 | Event 1 |
Event 3: An exponential growth in the demand for café con leche (coffee with milk) in the bars of Plaza Mayor in Madrid is detected. | 3 August 2014, at 07:00 | 12 August 2014, at 22:45 | Event 2 |
It is not always necessary to manage timestamps for the start and end of an event; it will depend on the type of processing to be done.
Let’s look at another example showing the controversial concepts of event correlation and causality
Happening | Start time | End time | Triggered by other events |
Event 1: Well-known culture representatives protest in the centre of Madrid against cuts | 19 July 2012, at 17h 00’ 00’’ | 19 July 2012, at 20h 01’ 00’’ | Event 0 |
Event 2: The Spanish risk premium shoots up to 579 points | 19 July 2012, at 18h 00’ 00’’ | 19 July 2012, at 18h 01’ 00’’ | Event 1? |
Event 3: A newspaper that shall remain nameless overlaps the two images on its front page establishing a relation of causality. | 20 July 2012, at 06h 00’ 00’’ | 20 July 2012, at 06h 00’ 00’’ | Event 2 and Event 1? |
Causality between events is not always objective. This is a clear example of what happens when a system confuses event correlation and causality just because they both share the same time window.
It is very important to conduct a thorough analysis of causality objectives between events because a complex event processing system can generate incoherent conclusions if causal relations are established that do not reflect reality.
Event Cloud
The diagram above shows an event cloud related to customer actions at a given banking website. Each linear sequence of events represents customer actions when they access their accounts and conduct various bank transactions in any order.
The term Event Cloud usually refers to the set of all events that can affect a business, and includes time measurements, causality relations and any other type of significant relations. |
Event Clouds reach a company through its information systems and its communication layer. The number of input data formats can be arbitrarily high. Again, it is more helpful to use diagrams.
It is precisely this jungle of data, information systems, and communication protocols that gives power to the events. However, they can also become troubled waters where only some fishermen will gain.
Think for example what would happen if we started detecting the following transaction sequences: “A user enters the website, changes his password and immediately runs an automatic payment order to an external account”, all in less than 5 minutes. Each one of these events by itself is normal, but their sequence, including the time variable, makes the set, the pattern suspicious. Why change the password just before withdrawing money?
But it could perfectly be that someone returning from summer holidays decides to pay for the gym after changing their password because they have been thinking for some time now that it was not secure enough.
Detecting event patterns is an enormous challenge, and in future articles we will go into more detail. A clear example are the current fraud detection systems. Historically they have been too rigid and developed with software with a lot of conditional statements that simply controlled the special cases of fraud already known. This made them inflexible and difficult to change; and naturally, they were prone to inaccuracies or even major errors.
With rules engines capable of processing events, more flexible and easy to update systems can be developed for of fraud detection and many others applications.
Event Levels
Going back to the previous diagram, one might think that the event cloud is a messy mass of information. But the truth is that events can be organised into very simple structures, for example, in layers or levels.
The idea of organising them in layers is similar to the organization of the OSI stack in telecommunications. The lowest level, the physical layer, transports events called packets (strings of bits). A specific set of these strings form a higher level event on the data link layer, which in turn are organised to form another event on the network layer and so on until events are generated at the highest level, the application layer. This can generate, following the example, the delivery of WhatsApp messages, which make our lives much more complicated.
The idea of event layers is also applicable to business events. It would seem logical to break down high level business events into lower level events.
An example would be the high level event “hiring of Gareth Bale by Real Madrid”. This will depend on a lot of lower level events: negotiation by both player’s managers, negotiation between club presidents, preparation of press releases to add pressure to the negotiations, search for bank funding, ignoring public opinion describing the deal as immoral,…
A higher level event can be broken down into lower level events, which can be present or absent.
These events in turn imply other lower level events, such as the use of accounting software, bank management software, text editing software (to draft the contract). If we continue with this breakdown, we can drill all the way down to the IT layer, having started with the business layer. A typical breakdown of business events will look like this:
Business Process Layer 1 → Business Process Layer 2 → … → Application Layer → Middleware Layer → Base Software Layer.
Organisation by event layers facilitates analysis and traceability of business processes.
If we have defined the dependencies between events at different levels, it is later easier to trace this from business errors down to problems in the database or even in the communication layer.
Obviously, business managers only worry about business layer events. If lower layers need to be analysed this is done by programmers, as we all know.
Event Flows
The first CEP applications that dealt with stock exchange events processed events in the order of arrival. The advantage of these systems was their ease of implementation. With such simple processing it was possible to do it in almost real time as the logic was also simple.
But such simplistic reasoning gives equally poor results. If on 11 September 2001 at 16:00 h GMT+1 dramatic drops in the Tokyo, London, Paris and Frankfurt stock exchanges arrive at the same time, this system would not have been able to detect the common underlying relation that could become a pattern. Who could see that it all lead back to Bin Laden?
An Event Flow is an orderly linear sequence of events happening within a specific time window. |
The Event Cloud includes the concept of event flow as a special case where events follow a linear order, one after the other.
When events are processed in the order of arrival we call it ESP (Event Streams Processing). The objective is to provide a system with the capability of processing very high rates of events in very short timeframes. The aim was, and still is, to conduct data processing that gives results in seconds. However, information as important as the possible relationship between these events, excluding the fact that they all happened at the same time, cannot be lost along the way.
Processing the event cloud is what we need
It is more common to have to deal with several event flows simultaneously, so that the events in each flow relate to the rest in very different ways.
As already mentioned, the events belonging to the same flow are interrelated by their date of arrival; however, it may also be the case that flow events are caused by events from another flow or that they form patterns with events from other flows. This leads to the basic conclusion that event flows must not be processed independently. Even an attempt to mix them in a single sequentially ordered flow is considered insufficient.
The causality relation doesn’t always have to be sequential. Imagine a string of WhatsApp messages in which a project manager and one of his programmers are happily discussing the functionality of a screen that provides an application report. The Project Manager sends a WM1 message proposing a 2D graph with pie charts. The Programmer replies with a WM2 message saying diagrams should be 3D, because it’s “a piece of cake”. The Project Manager gets encouraged and replies with a WM3 message saying that as well as 3D the values on each piece of the pie should be highlighted. The Programmer adds a WM4 message with a detailed legend on the top right of the chart.
Meanwhile, the Project Manager receives an EM1 message by email from the customer’s systems manager saying the report must be included in the application that same afternoon. The Project Manager replies with an EM2 message assuring him it will be ready. The Project Manager sends the Programmer a WM5 message telling him it is enough to come up with a text report, and make it quick.
We see here how the events of a second flow can clearly affect the events of the first flow by interrupting the sequential causality that had been established initially.
So then, which ideas are important? … Coming soon in the last chapter.
References
“Event processing for Business”. David C. Luckham
http://www.archelaus-cards.com/retail/archives/20090112.php
Programming Without a Call Stack – Event-driven Architectures Gregor Hohpe www.eaipatterns.com
Oracle Complex Event Processing (CEP) – Key Features and Benefits
http://forum.complexevents.com/viewtopic.php?f=13&t=183
http://inteligencialinguistica.com/blog/causalidad-vs-correlacion-primera-parte/