Getting Started with Oracle Event Processing 11g

In the world around us, every second of every minute of every hour, the human brain is bombarded with a limitless number of things that happen either at the same time or sequentially, or in a totally and seemingly erratic way that may not make sense immediately but as more of these things happen, we can start to understand their relevance and importance.

For example, we hear cheering in the distance, we see balloons flying in the air, music starts to play, police cars and trucks appear pulling brightly covered trailers with puppets and people waving on them, followed by ambulances, and today's date is July 4th. Individually, these events could mean anything, but together? It's probably an Independence Day Carnival Parade!

Our brain can easily determine this fact in the blink of an eye" and while not overly simple to define in computing terms, we could describe a "Parade Event Pattern" as follows:

Your brain is not restricted to sending information and just waiting until there is a response, or forced into following a series of fixed steps to get something done. As with this example, it is able to take the events happening now, their relevance to additional external factors such as today's anniversary date and understand a "parade" event pattern.

So as you learn more about Complex Event Processing, we focus on how this technology can take continuously flowing, never-ending information, from a potentially unlimited number of different places, and immediately understand how it relates to things happening right now and in the very near future, commonly known as Real-Time Situation Awareness .

The problem now in the world of computers is the proliferation of data. Information arrives from many different systems, in vast quantities, at different times, at different speeds, some of importance now to certain other systems, people or processes, and some stored for later recovery and determination. Why the proliferation now?

There are many issues involved, but here are just a few major ones:

As we evolve and the complexity of these systems "pour" more and more huge volumes of information at computer applications, we are reaching a "tipping point" where traditional point-to-point or request-reply-based solutions of the world break down and become unmaintainable and not extendable.

A company business can be influenced instantaneously from things (events) that can happen, not only in the "cozy" understandable world within its own environment but also from activities (events) from beyond, such as from "the Internet of things"—real-time sensor device that can measure and report on a multitude of situations, including "the impending danger from a sudden rise in temperature in a food storage facility" or "the global positioning system location of a shipping container which is having an unauthorized opening with movement detection sensed from within".

Immediate impact to a company's business can also come appear "out of nowhere" emanating from a change in global business conditions indicated from the ever-expanding social media outlets, for example, Twitter, instant messaging, and so on. Millions of people at the same time can all comment on the poor condition of a new product, highlighting an immediate need to change a product design. This will inevitably affect profits and will probably significantly affect the value of the business. So companies are now inevitably being manipulated by a wide range of both understood and misunderstood events.

In the past, probably going back over 15 years ago, business applications have had to conform to the methodologies, structure, and interfaces from the then available computing technologies (such as databases) where information must be inserted and statically placed. Only after this can users then analyze and respond. Traditional JEE Application Servers were generally implemented, expecting a client application to send an initial request and will then only process that request through, in most cases a significant amount of logic code, before it can respond back to the client. While these technologies enable, and will continue to provide benefit in more batch-orientated, less real-time approaches, newer lower latency and faster in-memory middleware products are now available.

Event-Driven (Architecture) based systems are intrinsically smarter, or better "equipped" to handle these types of situations, processing an entire business infrastructure as events that can be immediately interpreted and handled, spanning across the many departmental "silos" such as finance, manufacturing, and sales. These types of systems are also context aware and execute when they detect changes in the environment or business world, rather than occurring on a predefined (nightly) schedule or requiring someone to initiate an execution.

As the problems associated with Big Data grow substantially over the coming years in terms of the capture, management, and the ability to process the information within a tolerable amount of time, Event-Driven technologies (specifically Complex Event Processing) can provide Fast Data capabilities to apply a greater level of "intelligence" and decisioning to the originating data streams much closer to the "point of occurrence".

So the benefits of an Event-Driven technology approach is to turn that proliferation of data into real-time knowledge by firstly representing events (things that happen from anywhere) in standard ways, providing an ability to factor out events, route events, filter events, aggregate events, and correlate events intelligently, so that in most cases fragmented events can be evolved into holistic, solid, understandable business events, enabling the business to better view, control, and adapt to situations relatively instantaneously.

So how are Complex Event Processing Platforms used now to solve business problems? Certainly over the past few years, this technology is being used across most, if not all, of the different types of industries.

The financial services capital markets companies are using this technology for real-time algorithmic trading and real-time risk management types of solutions. As the stock markets stream their endless financial instrument data with values which can instantly fluctuate, there is an ever growing need to effectively handle this huge volume of information, understand its impact and potential risk, and then react as quickly as possible. The better the capability to evaluate and predict the consequences of the information, and the quicker the ability to respond to the results of this analysis, the more successful the business and the more money that can be made with less exposure to business risks and threats. This type of real-time trading information can be usually visualized using heat maps and scatter charts.

In the Electricity industry, customers are using the Complex Event Processing (CEP) platform for many new types of applications, which include Smart Meter, Smart Grid, and outage detection monitoring solutions. Sophisticated Demand Response (DR) solutions bring together system operators and the power generation companies, who contract with energy management and monitoring companies to provide energy usage load reduction services on demand. These technology companies that are using CEP-based applications contract with commercial and industrial businesses that are large consumers of energy, whom agree to curtail energy usage on demand. Streaming event devices are installed at client locations to measure energy usage and, in some cases, proactively control the load using continuous energy demand and usage data at minute or, even second, intervals. The generated profit revenue received from system operators is then passed back to the clients, relative to the number of associated load reduction dispatches.

Handling real-time events has a long history in the telecommunications industry, such as those generated by the various devices on the network, events from mobile phones, or perhaps streaming Call Detail Record (CDR) events indicating the time of calls made and whether some of these calls failed. Complex Event Processing platforms provide the technology for many new applications and solutions in this domain. As in other industries, Event-Driven platforms have a broad base of possible implementations. Some businesses have created powerful network management and monitoring solutions, which can detect hardware failure-related events continuing over certain time periods, or situations where equipment has not been issuing events for some time and in these circumstances alert messages are distributed and escalated.

In the context of an enterprise-level mobile telecommunication IT infrastructure, there are many different applications coming from many different suppliers. When the overall performance is not immediately meeting expectations, it's not easy to identify which component is the offending issue in the supply chain. Therefore these next-generation management and monitoring applications (based on Complex Event Processing) provide the capabilities to show the complete, holistic "picture", providing full visibility to the situation of a business through flexibility and fully integrated features, enabling agility for the infrastructure to react quickly to changing scenarios, and providing full operability enabled by a solution designed to meet business needs.

A very powerful capability of Complex Event Processing platforms which is being leveraged in the Transportation, Telecommunications, and Public Sector domain is real-time integrated spatial analysis.

A business can use this technology in applications where there is the need to monitor the movements of its assets and resources. Using, for example, GPS (global positioning systems) the movement patterns of someone, or something can be tracked in real time as it passes through boundary points (such as security checkpoints in an airport) to identify its route and, to some extent, predict where this person or object may subsequently move next. Also, this capability can be used to analyze a current position and its relationship to geofenced areas. A geofenced area being the definition of a geographical shape (polygon) defined or declared by a series of spatial coordinates.

When a resource gets near, inside, or enters and exits the geofenced area, various actions can be immediately performed, such as a warning message of an imminent exposure to a dangerous natural disaster, or offering a big discount on a second coffee at the person's current location or soon to be, position, based on his or her current movement pattern.

First Responder emergency services solutions can use integrated spatial technologies to not only monitor a fire or hundreds of simultaneous fires, but also dynamically track the movement on the fire, affected by weather conditions (wind) or igniting hazardous materials. These types of systems can evaluate immediately the relevance, importance, and applicability of all of the related assets (fire engines, police vehicles, and so on) close to these areas. For example, if a fireman does not move in certain number of seconds when close to a fire, this could indicate a serious life threatening situation.

There are many other types of business solution implementations using Complex Event Processing platforms that range from online retail monitoring systems, real-time data center infrastructure management, fleet vehicle transportation monitoring, traffic flow monitoring with variable toll charging and speed control, oil fields and rig monitoring/automation, and a host of real-time sensing device opportunities, where these devices can monitor the environment inside shipping containers, or air pollution situations. The scope and different type of applications that can now benefit from using Complex Event Processing technologies are evolving just as quickly as the world is changing, with a growing need to predict and pre-empt and in, some cases, prevent situations from even happening.

During the next few sections we will explore some of the basic principles and concepts commonly used in the creation of event-driven applications. These are the major "building blocks" for any solution that handles streaming event data.

An event stream

Generated by hardware sensor devices, distributed anywhere from the "Internet of things", computer applications, database triggers, or generated from any of hundreds of different sources, events arrive for processing in an event stream or streams. Event streams can have events that are continuously flowing at high volumes or arrive in sporadic intervals, but the events never end and are always time ordered, just like in the real world.

A market data feed in the financial services world, the GPS signals from your mobile telecommunications device and business events from a Service Orientated Architecture Application (SOA) are all examples of event streams.

In general terms, event streams can be simple, streaming, or high volume.

Traditional computing systems based on database or Java Enterprise Edition (JEE) infrastructures are not designed to effectively handle this type of continuously flowing event data, as the reading and writing demands to disk, or "send/reply" implementation paradigms involve increased and detrimental processing latencies or delays. So there is a need to evolve a new approach to handing these requirements and with an event-driven infrastructure it can "impose" itself "over" the event streams in memory using a defined window of time or number of events count.

An event type

The event types that flow "along" the event stream defines the properties associated with the event. Event type definitions can range in their levels of complexity, but in most applications can be declaratively defined with a simple notation.

Using the door event example discussed earlier in this chapter, a house event stream that is continuously monitoring things that are related to all doors in a building could have a specific door event type defined with a collection of property names and their associated values.

Event Processing Network

So now we have an event, probably thousands or millions of them that need to be effectively handled and processed. As these events continuously flow they need to be identified, have a response very quickly and are often "transported" only in memory, so using a database is not a recommended design option.

For this purpose, many Complex Event Processing platforms provide the Event Processing Network (EPN) (otherwise known as a Directed Flow Graph ).

Provided as the best approach for handling streaming event data, the EPN can be generally designed and modeled using various tooling offerings. The EPN is designed as a loosely-coupled collection of event nodes, each performing a unique action on the events as they pass through the network. Each event node subscribes to one or many other event nodes with the state (conditions/properties) held in the event definition itself.

This application model design approach provides the ability for extreme event processing in low latencies with a simple way of extending and/or changing the event handing as real-time situations happen. It also facilitates a mechanism (foreign stages) to enable new event nodes to be introduced into the solution either dynamically or statically during the actual deployment life cycle of the executing application.

A well-structured EPN will probably perform beyond expectations and set the foundation for easy extensibility, integration, and solution maintenance.

While many kinds of event nodes are evolving, most can be one or more of the following types:

• Event adapters provide the connectivity to event sources and sinks, and are relatively simple code implementations that normalize the incoming or outgoing data stream and convert this into event types that are processed downstream or upstream in the EPN. For example, an inbound event adapter can provide the connection to a TCP/IP socket and an outbound event adapter can provide an interface to a visual user interface.

  

• Event channels are the conduits that effectively handle the routing of events, these event nodes not only play an important role in ensuring that the various events are analyzed efficiently, but they also can have properties that can powerfully effect the performance of the application, such as controlling the amount of memory used for the events and the number of processing threads.

  

• Event cache and event POJO Bean nodes provide the in-memory persistence of long-term reference data and the solution-specific business logic written as a "Plain Old Java Object". These event nodes ensure that information needed for long periods of time can be managed, interrogated, and safely held in computing memory, and that any type of additional processing logic can be implemented. POJOs can sometimes act as event sources or event sinks. An example of using Event POJO Beans would be to include and enhance old legacy code, which has been mature and stable for a long period of time in other coded solutions, and would continue to provide additional value in the new Event Processing Network. One caveat when using this type of "old" code is to clearly understand the additional "cost", in terms of memory usage and processing load that will be incurred and how this will impact the overall performance of the new solution and this should be considered during the design phase.

  

• Event processors are the meta-containers for the powerful event analysis needed for any type of solution. There can be one or many event processor nodes in an application and they store the event processing language, which can be rules or queries that statically executes continuously on the flow of arriving events. The event processors are the core engine service of a Complex Event Processing solution, and the capabilities of such engines in most cases, dictate how successful the technology will be in delivering the desired business solution.

  

In most Complex Event Processing platform technologies, the Processor Event Node , or a similarly-defined construct (event engine), will execute the language of choice for the analysis of the events in an event stream.

For example, a car rental company might use the following business rule:

Offerings in the industry currently include; State-oriented, Inference rule, Script-orientated, and Agent-orientated SQL-idioms. Some people are familiar with the business rules approach and so decide to use the traditional "what-if-then" kind of analysis. Most others decide to leverage their SQL database skills and extend that knowledge to encompass the handling of streaming data in a way that is familiar to how they interact with data that is stored and processed in a database.

The benefits of a SQL-based event continuous query language extends the rigor of the relational model to event stream processing that can result in a more robust implementation with broader application.

These types of CEP language implementations can incorporate the well-known SQL '99 plus standards and relatively easily introduce the language extensions for the temporal and event count windowing requirements. For many, using this type of event handling approach provides now, and for the future, a single consistent language that can be used for all database and middleware application analysis processing.

Processor event node methodologies

The processor event node provides the direct analysis on the events and uses a number of various techniques.

Event filtering is applicable when thousands or even millions of events flow into an application and there is a need to ensure a time effective handling of the more important information. This can involve either removing or sending the events of no concern to another channel or path, where it can be handled separately. In this way only the events that indicate a relevance to the current application requirement are passed for further "heavy lifting" complex analysis. By using this capability the event load is more evenly spread through the application, making it far more efficient.

Event correlation and aggregation is generally employed after any event filtering has been performed and is a methodology to understand the relationship between different events and then join or merge these events together. For example, when thousands of events from a temperature sensor arrive providing individual values for each room in microseconds, one approach is to determine which rooms are of interest, then identify the sensors only in these rooms and finally calculate the maximum, minimum, and average temperatures over a one minute time period.

Event pattern matching enables the identification of a certain distinct occurrence in either a specific time window, that is, the last five minutes, or in the last number of events. For example, this can be an event pattern where one can identify an "order" event, followed by a "completed packaging" event, followed by a "truck loaded" event, followed by a "arrived at customer house" event, all for a specific item, in three hours. This could trigger a SMS message to the customer stating "your order has arrived". Event patterns can be without limit but are generally dependent on the semantics of the specific industry. They can incorporate "not events", where you can define an event pattern that expects event A, followed by event B, but not a C event, followed by a D event.

A synthetic or business event often represents the dynamic construction of an event from a collection of events or elements of events (fragmented events). In many cases, an event may arrive that has little meaning alone but when joined with contextual or reference data, it has significant meaning. Let's take again, for example, the temperature sensor in a room. This sensor may send an event that provides an ID of 005 and a value of 60. Now if we had previously saved information that indicates and ID of 005 refers to a sensor on the tenth floor of a building at San Pedro Square, in the kitchen, attached to the ceiling at the right corner, then by joining this information with the current sensor temperature value of 60 degrees Fahrenheit, we now have a much more concrete (business) event that can be passed to another piece of business logic or system for action.

Processor extensibility

With the implementation flexibility offered by the Event Processing Network, it is important not to restrict the event processor with a limiting language implementation which does not support specialized language extensions. These extensions are driven by the changing analysis demands from the advances in the various related technologies of the future, but are also focused on enabling additional industry and domain-specific capabilities that are required by specific users.

Some event processor implementations provide the ability for easy extensibility using a capability called data cartridges .

A data cartridge concept provides the notion of directly adding or plugging in new functionality to a Complex Event Processing system, so that event analysis can be added for any type of circumstance.

It would typically define a set of new object types and their behavior, and/or provide implementations for various extensibility interfaces. The purpose of data cartridges is to enable users to capture business logic and processes associated with specialized or domain-specific data in user-defined data types. It constitutes one or more of the following components: user-defined data types, implementation for these types, new operators and aggregate functions, and implementation of certain extensibility interfaces.

An example of data cartridge extensibility is to integrate specialized spatial or Java language analysis directly as part of the CEP engine service.

Event processor "Intelligence Injection"

Another topic to briefly cover in this section is the ability for event processors to be dynamically updated or changed with the rules or queries "on the fly" while the application continues to execute.

When events change in real life, your brain does not need to "reboot" in order to reassess the situation, it simply continues with the additional "event pattern" information to make a more informed decision. It may be implemented in different ways depending on the Complex Event Processing platform but most now, can provide this capability, and as these types of implementations evolve in the future, Event-driven systems potentially will have a self-awareness, self-learning, and a self-determination allowing them to adapt far more effectively to the changing dynamics of the world that surrounds us.

So as you can now understand, a Service Orientated Architecture design approach (send-reply paradigm) for many real-time, event-driven applications is perhaps not a good choice, and Complex Event Processing platforms have evolved over the years to address that growing need. However, in many ways an Event Driven Architecture (EDA) compliments a Service Orientated Architecture (SOA) and in many comprehensive industry solutions, these two implementation design patterns work together to solve overall requirements.

We call the combination of architectures as Event Driven SOA (ED-SOA).

Imagine the requirement for an airport to provide a complete system that will immediately identify missing luggage from a flight and in addition, the monitoring of passenger movements in real time to provide additional VIP services (such as a fast-path access through security checks).

The Complex Event Processing platform could be utilized to process the events from sensor devices that are reading the bag locations, using an event pattern that identifies immediately when bag A has passed by sensor 5, and sensor 6, but has not passed by sensor 7 within two minutes, with this event pattern applied to every bag on every conveyer belt in every part of the airport. At the same time, perhaps from movement events using Bluetooth technology, every "opted in" passenger location event (normally longitude/latitude data) is also sent to the Complex Event Processing platform, which has an event pattern, continuously analyzing each individual passengers position in the airport against general known transit times to his or her required gate.

This system could also get immediate events from each plane which, in addition to being able to transport people around the world, are also really big event "devices" that send information such as a "pushing back from the gate" event or "wheels up and departed" event. This valuable information could assist in the real time determination on whether the plane has in fact not left the gate yet even though the fixed departure time has been passed.

As all of this real-time activity is taking place, and the results are visualized on a SOA Business Activity Monitor technology or SOA Business Processes are initiated, other relevant SOA applications such as Ticketing and Reservations leveraging a traditional SOA Service Bus could also be included in the architecture as they can provide information about the "importance" of the passenger based on his or her class of service and which gate should be associated with each passenger.

While Complex Event Processing can be used as a standalone platform, particularly where extreme high event throughput is needed together with immediate or very low processing latencies, many complete solutions now need the combination of both implementation design pattern technologies.

The capability to look around the corner and predict what might happen next based on what is happening now is an extension to the event pattern matching over time or event windows used in ED-SOA solutions.

An example is when Complex Event Processing is used to monitor real-time CDRs (Call Detail Records) streaming in a telecommunications network and identifies subscriber calls that are dropping in a short period of time. Once the event-driven system has determined that a problem is happening now for a specific subscriber number, the next step is to "look back" over past persisted data, which could be large, and evaluate whether this person is more or less likely to change his or her telephone company.

The analysis of the potentially vast amount of historical data creating models that use a collection of algorithms to predict behavior is generally handled by other technologies such as data mining or real-time decisions. However, Complex Event Processing platforms can leverage these models and then use them in real time.

The value in this use case would be that the person experiencing the dropped calls could immediately get a call from customer support or a SMS text message that could automatically offer an incentive to stay with his current supplier.

Complex Event Processing Platforms are now evolving from the early adoption phase and becoming much more mainstream for many companies. There are a fair few choices on the type of event-driven platform you can invest your time in to evaluate, and deploy your next generation event-driven solutions.

In the next chapter, we will delve into the main components of the Oracle Event Processing Solution Platform, such as the EDA Java Application Container that has been optimized for performance and scalability. We focus on the evolution of the technology and its unique modular approach enabling this technology to be used in a wide variety of event-driven implementations.

More importantly, we will get you involved with the Oracle product directly, by stepping you through the process of creating an application and publishing (deploying) on your own installed version of the system, and we also show you the capabilities provided in a rich Internet application, how to monitor, and manage the system.

As you progress through the chapter, you will build up a solid foundation of knowledge that will be needed in the remaining sections of this book, and hopefully inspire you to become experts in Oracle Event Processing.