THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

StreamInsight Team

  • LINQPad for StreamInsight 2.0

    This has come up several times in the wake of our 2.0 release: The new LINQPad driver supporting StreamInsight 2.0 is now live! You can download it as usual through the LINQPad UI:

    lp20

    There are separate versions of the driver only because assembly references are not automatically forwarded for StreamInsight 2.0.

    Regards,
    The StreamInsight Team

  • White Paper/Case Study on ICONICS’ Use of StreamInsight for its Energy AnalytiX® Solution

    A couple of days ago, we released a new StreamInsight white paper/case study on TechNet and MSDN. The paper is joint work with ICONICS and discusses how ICONICS is using StreamInsight technology for its Energy AnalytiX® solution. The paper is available for download here in the Technical Articles section of the StreamInsight documentation.

    Today, businesses and organizations need to pay more and more attention to energy usage, as customers and the general public are becoming increasingly concerned about a respectful and sustainable use of resources. Organizations therefore need to carefully manage their use of energy and provide better visibility into their energy consumption. In this paper, we discuss how software solutions can help address these challenges.

    Besides providing some background on the drivers behind energy management, the paper discusses how organizations manage their use of energy with current product and service offerings from Microsoft and ICONICS. In the main body of the paper, a case study explains in depth how ICONICS Energy AnalytiX® is using Microsoft data platform components such as SQL Server StreamInsight to deliver market leading energy management solutions.

    Regards,
    The StreamInsight Team

  • StreamInsight V2.0 Released!

    The StreamInsight Team is proud to announce the release of StreamInsight V2.0! This is the version that ships with SQL 2012, and as such it has been available through Connect to SQL CTP customers already since December. As part of the SQL 2012 launch activities, we are now making V2.0 available to everyone, following our tradition of providing a separate download page.

    StreamInsight V2.0 includes a number of stability and performance fixes over its predecessor V1.2. Moreover it introduces a dependency on the .NET Framework 4.0, as well as on SQL 2012 license keys. For these reasons, we decided to bump the major version number, even though V2.0 does not add new features or API surface. It can be regarded a stepping stone to the upcoming release 2.1 which will contain significantly new APIs (that will depend on .NET 4.0).

    Head over here to download StreamInsight V2.0. The updated Books Online can be found here.

    Update: For instructions on how to make your existing application work against the new bits without recompilation, see here.

    Regards,
    The StreamInsight Team

  • StreamInsight Training Released

    Time for a life sign from our Team Blog, and a very exciting one: Richard Seroter, Principal Software Architect and Microsoft MVP, has put together a comprehensive Pluralsight course on StreamInsight. Richard has been a great StreamInsight advocate since its beginnings and knows the platform inside-out. You can watch the introductory module for free:

    ps_color_logo_250x55

    StreamInsightStill
    Watch Introduction to StreamInsight

    Get full access to the rest of this StreamInsight course and hundreds more courses for only $29/mo

    Free Trial! Subscribe Now

    Regards,
    The StreamInsight Team

  • New in V1.2: StreamInsight Performance Counters

    One of our goals in version 1.2 was to improve the administrative surface of StreamInsight. The newly introduced performance counters enable you to retrieve diagnostic information and monitor health and performance in real-time. Rafael has put together a great video introduction to perf counters in StreamInsight—Rafael, take it away:

    (You might want to click through to YouTube to watch it full screen.)

    Regards,
    The StreamInsight Team

  • New in V1.2: Default Window Output Policies

    Who knows what window output policies in StreamInsight LINQ are? Anyone? Well, you have been using them whenever you have used a window. Output policies specify the timestamps of the result of the aggregation that happens on the window. Even if you didn’t really care about that, you had to specify the output policy parameter for the window. In V1.2 we are introducing

    • Default policies so that you don’t have to specify them for every window, and
    • Another output policy for Hopping/Tumbling windows.

    Importantly, there is no breaking change. Whatever you used in 1.1 is still working in 1.2, with the same behavior. For count- and snapshot-windows, the single existing output policy is becoming the default:

    SnapshotWindow() is equivalent to
    SnapshotWindow(SnapshotWindowOutputPolicy.Clip)

    CountByStartTimeWindow(num) is equivalent to
    CountByStartTimeWindow(num, CountWindowOutputPolicy.PointAlignToWindowEnd)

    Set-based operations on top of snapshot windows always produce an event with the timestamps of the snapshot window itself, while operations on count windows produce a point event at the end of the window.

    The only existing output policy for hopping and tumbling windows in StreamInsight 1.1 was to produce the window size as result event lifetime. Now this turned out to be impractical in many cases, especially with overlapping hopping windows. Have a look at the following diagram:

    image

    Just like the windows, also the result events overlap. When using this result in a subsequent operation, like a join with another stream, this easily leads to unexpected results. In many cases, only a single aggregation result at each point in time is needed (instead of four concurrently valid payloads in the diagram above). To this end, we have introduced another output policy for hopping (and tumbling) windows, which produces a point event at the end of the window:

    image

    This policy is now also the default for hopping windows:

    HoppingWindow(size, hop) is equivalent to
    HoppingWindow(size, hop, HoppingWindowOutputPolicy.PointAlignToWindowEnd)

    TumblingWindow(size) is equivalent to
    TumblingWindow(size, HoppingWindowOutputPolicy.PointAlignToWindowEnd)

    This can now easily be converted into a continuous signal, for instance by using the ToSignal() macro that we discussed earlier. This signal has a payload at each point in time and hence is suitable to be joined with another stream, which is not necessarily synchronized in time. Typical example: join the result of a hopping aggregation with some reference stream:

    var result = from a in avg.ToSignal(e => e.SourceId)
                 from r in reference.ToSignal(e => e.SourceId)
                 select new { alarm = a.Value < r.Threshold };

    Dealing with point events instead of overlapping intervals in the context of windows helps avoiding unexpected side-effects and—most of the times—corresponds to the intended semantics.

    Regards,
    The StreamInsight Team

  • New in V1.2: LINQ Macros

    StreamInsight LINQ is powerful and expressive. At the same time, the learning curve for a new user can be steep, because the query operators work on a semantically basic level. This means that complex use cases need to be build bottom up, composing the built-in operators to create higher-level query modules, etc. Many such modules are used over and over again, like the conversion of a stream to a signal, or the left-outer join. In StreamInsight V1.1 it was possible to some extent to write your own extension methods over CepStreams, but due to the mechanics and capabilities of the StreamInsight LINQ provider, for many such methods this was often difficult if not impossible. In V1.2 We overhauled the LINQ infrastructure significantly to make the definition of LINQ macros much easier. This enables you to express common building blocks as extension methods just once, and then use and re-use them to compose more sophisticated computations. The user of these macros doesn’t need to think in terms of AlterLifeTime or Left-Anti-Semi-Join anymore.

    Let’s look at the first example, creating an event stream into a series of continuous, non-overlapping interval events. This is a typical preparation to implement a correlation (join) of two streams. In StreamInsight we can easily do this by extending the lifetime of each event and then clipping it right at the beginning of the next event (this is also described in the section “Point to Signal Conversion” on our MSDN pages):

    var result = source
                    .AlterEventDuration(e => TimeSpan.MaxValue)
                    .ClipEventDuration(source, (s, e) => true);

    Being a typical query pattern, I can easily convert this into an extension method:

    public static CepStream<T> ToSignal<T>(this CepStream<T> inputstream)
    {
        return inputstream
            .AlterEventDuration(e => TimeSpan.MaxValue)
            .ClipEventDuration(inputstream, (e1, e2) => true);
    }

    And use it as follows:

    var result = source.ToSignal();

    Now assume that my input is a multiplexed stream, containing events from different sources, identified by some payload field. Obviously, I want to convert each source into a separate signal, so the matching condition in ClipEventDuration() needs to change:

    var result = source
                    .AlterEventDuration(e => TimeSpan.MaxValue)
                    .ClipEventDuration(source, (s, e) => s.Id == e.Id);

    Ideally, I have an according extension method that allows me to pass the matching expression as a parameter:

    var result = source.ToSignal((s, e) => s.Id == e.Id);

    Here is how I can write such an extension method:

    public static CepStream<T> ToSignal<T>(this CepStream<T> inputstream, Expression<Func<T, T, bool>> matchExpression)
    {
        return inputstream
            .AlterEventDuration(e => TimeSpan.MaxValue)
            .ClipEventDuration(inputstream, (e1, e2) => matchExpression.Compile()(e1, e2));
    }

    The ability to compile expressions into delegates is new in V1.2. It also allows to create the equality predicate inside the macro:

    public static CepStream<T> ToSignal<T, K>(this CepStream<T> inputstream, Expression<Func<T, K>> keySelector)
    {
        return inputstream
            .AlterEventDuration(e => TimeSpan.MaxValue)
            .ClipEventDuration(inputstream, (e1, e2) => (keySelector.Compile()(e1)).Equals(keySelector.Compile()(e2)));
    }

    Now I can use the macro by just referring to the field that distinguishes my substreams:

    var result = source.ToSignal(s => s.Id);

    Our favorite example is the left outer join, which is not a native StreamInsight operator. In this operation, we want to join left and right streams, but also keep every event on the left side that does not join with the right side. We can model this as a plain inner join, a left-anti-semi-join (everything left without right), and the union thereof. Since Colin has already blogged about this pattern, I’ll not repeat it here and just refer to his posting.

    Here is another really nice one: De-duplication. The following extension method helps you to get rid of duplicate events (same time, same payload):

    public static CepStream<T> Distinct<T>(this CepStream<T> source)
    {
        return from x in source
                group x by x into g
                from win in g.SnapshotWindow()
                select new { g.Key, IgnoreMe = win.Count() } into x
                select x.Key;
    }

    The usage now becomes very simple:

    var result = input.Distinct();

    Macros make it possible to build a domain-specific set of subqueries on top of the StreamInsight LINQ language, so that defining computations and analytics becomes more accessible and straightforward. The new V1.2 LINQPad samples include several other useful macros. Note that LINQPad does not allow true extension methods in the same file – they would have to be compiled and the resulting DLL referenced. This is why in our LINQPad samples the macros are defined as regular old methods.

    Regards,
    The StreamInsight Team

  • New in V1.2: StreamInsight Checkpointing

    Over at his blog, Isaac Kunen posts:

    One of the big features in the new version is resiliency, which helps you build highly available systems that use StreamInsight. This post gives a high-level, no-code overview of what this functionality does and how it can be used.

    Click through to see the full post.

    Regards,
    The StreamInsight Team

  • New in V1.2: Primitive Event Types

    In a previous posting, we introduced nested types as one of the new features in StreamInsight V1.2. The second improvement in the type system represents the other end of the complexity spectrum, namely the ability to use "primitive" event types, without any struct or class. Remember the LINQ statement to compute the distances between two vehicles in that posting:

    var distance = from a in vehicleA
                  from b in vehicleB
                  select new
                  {
                      distance = Math.Sqrt(
                          Math.Pow(a.Loc.X – b.Loc.X, 2) +
                          Math.Pow(a.Loc.Y – b.Loc.Y, 2))
                  };

    Wouldn’t it be nice to be able to write this as:

    var distance = from a in vehicleA
                  from b in vehicleB
                  select Math.Sqrt(
                      Math.Pow(a.Loc.X – b.Loc.X, 2) +
                      Math.Pow(a.Loc.Y – b.Loc.Y, 2));

    Yes, it would – and in V1.2 you can! The type of this is now simply a CepStream<double>, and you can use it like this:

    var result = from d in distance
                 where d > 9
                 select d;

    Beauty! But wait, there is more: in V1.2, we also enabled the LINQ Let clause. We can consolidate the above statements nicely into a single one:

    var withLet = from r in vehicleA
                 from l in vehicleB
                 let d = Math.Sqrt(
                     Math.Pow(r.Loc.X - l.Loc.X, 2) +
                     Math.Pow(r.Loc.Y - l.Loc.Y,
    2))
                 where d > 9
                 select d;

    You can find this example attached to run in LINQPad.

    See how we can create a very concise StreamInsight sample over a stream of int in LINQPad:

    var source = Enumerable.Range(1, 100).ToPointStream(
                    Application,
                    ev => PointEvent.CreateInsert(
                        DateTimeOffset.Now.AddMinutes(ev), ev),
                    AdvanceTimeSettings.StrictlyIncreasingStartTime);

    var result = from win in source.TumblingWindow(TimeSpan.FromMinutes(5))
                select win.Avg(e => e);

    result.ToPointEnumerable().Where(e => e.EventKind == EventKind.Insert).Dump();

    Alright, this query doesn’t really make a lot of sense, but you get the idea. This sample is also included in the set of new LINQPad samples – just update your LINQPad driver to get them.

    Regards,
    The StreamInsight Team

  • New in V1.2: Nested Event Types

    One of the focus areas for us in V1.2 was the LINQ surface of StreamInsight. We have straightened out many of the kinks people complained about in previous versions and have added new features. Too many improvements for a single blog post, so today we’ll start with new features in the event type system.

    One of the big limitations in StreamInsight so far was the requirement to express event types as a “flat” list of fields. What we are introducing with our current version is the ability to use nested structures as event types. You can now use fields that are instances of another class or struct, and you can define such nestings with arbitrary depth. Let’s look at an example:

    public class Location
    {
       public double X { get; set; }
       public double Y { get; set; }
    }

    public class Vehicle
    {
       public string Id { get; set; }
       public Location Loc { get; set; }
    }

    Here, we are planning to use Vehicle as our event type, with a field of type Location. We can create our payloads in the input adapter (or as a .NET sequence item) as expected:

    new Vehicle { Id = "A", Loc = new Location() { X = 2, Y = 3 } }

    Within LINQ query statements, you can refer to these nested fields:

    var distance = from a in vehicleA
                  from b in vehicleB
                  select new
                  {
                      distance = Math.Sqrt(
                          Math.Pow(a.Loc.X – b.Loc.X, 2) +
                          Math.Pow(a.Loc.Y – b.Loc.Y, 2))
                  };

    (We are assuming that these two input streams actually contain overlapping events, otherwise the join wouldn’t produce anything.)

    Now we’d like to clarify a few things here:

    • The atomic fields in the structures still need to be of one of the StreamInsight-supported types. Collections as field types are not supported yet.
    • Nested structures always need to contain actual instances, they cannot be null.

    How do you deal with nested fields in untyped adapters? Well, no change here. You still use the GetField/SetField methods of the event, passing the ordinal of the desired field. You get the ordinal for a field from its name through the CepEventType dictionary. The name for a nested field is for example Loc.X, just like in a programmatic reference.

    Attached is a LINQPad sample that demonstrates nested types – you will also find it as part of the samples that come with the plugin. The sample also uses a macro for a common streaming design pattern, more about that in a later post. Don’t forget to update the plugin, btw!

    Regards,
    The StreamInsight Team

  • Updated LINQPad Driver and Samples

    With the release of StreamInsight V1.2 we also updated our LINQPad driver. It is now aware of StreamInsight versions, so that a sample can fail gracefully if the required library version is not found.

    You can install the new driver (of update your existing one) as follows: In LINQPad click on Add connection:

    image

    Next, you click on View more drivers:

    image

    Scroll down to the Microsoft StreamInsight Driver section and click on Download & Enable Driver:

    image

    Voila:

    image

    You can now simply cancel the driver selection window.

    The new driver allows to specify various connection properties:

    image

    To get the new samples, click on Download more samples:

    image

    Scroll down to Microsoft StreamInsight Samples and click on Download into LINQPad:

    image

    If you had downloaded samples for an earlier version, confirm that you want to overwrite them. The samples are now available in the StreamInsight subfolder:

    image

    We will discuss the features that are demonstrated by the new samples (What’s New in 1.2) in an upcoming series of blog postings.

    Regards,
    The StreamInsight Team

  • StreamInsight V1.2 Released!

    The StreamInsight team is proud to announce the release of V1.2! Don’t be fooled by the minor number increment - we've made a bunch of major improvements:

    • Resiliency against planned and unplanned downtime through checkpointing.
      Custom solutions to work around the problem of resiliency have typically been expensive to implement, and limited in their ability to prevent data loss and guarantee correctness of output after recovery. The new resiliency features in StreamInsight let you run your critical applications with tolerance to system failures and minimal operational downtime. StreamInsight resiliency supports recovering query state from the failure by checkpointing system state at specific points in time during normal operations. After a failure, the query is made operational again by recovering the query state from this persisted store.
    • Better support for predictive modeling and pattern matching scenarios through new user-defined stream operators.
      A user-defined stream operator, or UDSO as the team affectionately calls them, lets the user interact with the event stream in a procedural way. Unlike previous extensibility interfaces, they are not bound to a window, and can maintain their own state. A user-defined stream operator ingests a single event at a time and can produce one or more result events for each such input.
    • Improved administrator experience through performance counters.
      In previous versions, StreamInsight runtime diagnostics needed to be obtained from the engine’s diagnostic views. This new feature provides performance counter information, published through the Windows performance counter infrastructure, which can help monitor the health and performance of StreamInsight applications using well-established tools, such as Performance Monitor.
    • Improved administrator experience through the administrative event log
      The new administrative event log records administrative events and errors in the StreamInsight engine, which can be discovered, explored, and used via system tools such as Event Viewer. These events register state changes that can help system administrators understand the behavior of a StreamInsight application and react to state changes in StreamInsight components such as adapters, queries, and the engine, or investigate the unexpected shutdown of a query.
    • Improved development experience through support for nested event type structures and additional LINQ expressions.
      Previously, StreamInsight event types needed to be flat lists of payload fields. The new version of StreamInsight enables users to express event types as nested structures, reducing the friction between your native event schemas and their representation in the StreamInsight framework. Users can work with nested event types in the input and output adapters as well as in StreamInsight LINQ. A number of other improvements—like n-way joins, nested queries, and support for query macros—enhance the developer experience.

    The English as well as localized setup packages are already online on our download center page.

    In order to get the full experience, we recommend that you also download the new samples:

    1. We added one checkpointing and one UDSO sample to the StreamInsight product team sample package on CodePlex.
    2. We updated our LINQPad driver and added a set of samples there as well to demonstrate the LINQ improvements and UDSOs. You can update to the new driver and samples by simply adding a new connection and downloading the driver again (we'll follow up with a more detailed post on how to do this).

    We’ll post further articles about the new features and how to use them over the next few weeks, so stay tuned.

    Come and grab it while it’s hot, and let us know what you think!

    Regards,
    The StreamInsight Team

  • FIX adapter for StreamInsight

    Over the last couple of month, Rapid Addition, a leading FIX and FAST solutions provider for the financial services industry, has been working closely with the StreamInsight team to enable StreamInsight Complex Event Processing queries to receive input feeds from Rapid Addition’s FIX engine and to send result events back into FIX. Earlier today, Toby Corballis from Rapid Addition blogged about these capabilities here on HedgeHogs.

    We are very excited to demonstrate these capabilities at the SIFMA conference in New York. The session will take place tomorrow, Tuesday, 11am – 12noon, at the Hilton Hotel New York, 1335 Avenue of the Americas, East Suite 4th floor. Torsten Grabs from the StreamInsight team will join the RapidAddition and local Microsoft teams for the session.  If you are interested in attending the session please register at http://bit.ly/c0bbLL.

    We are looking forward to meeting you tomorrow at SIFMA!

    Best regards,
    The StreamInsight Team

  • Finalizing StreamInsight V1.2

    You might have heard the news from TechEd in Atlanta already: we are getting closer to the next release of Microsoft StreamInsight. It will be a minor release called StreamInsight 1.2. This release will focus on aspects that we heard loud and clear in the feedback we got from customers. Here are the highlights of the upcoming release:

    • Resiliency against planned and unplanned downtime through new check-pointing capability.
    • Better support for predictive modeling and pattern matching scenarios through new user-defined operator.
    • Improved administrator experience through performance counters.
    • Improved development experience through support for nested event type structures and many additional LINQ expressions.

    We are planning to release StreamInsight 1.2 this summer – the English version first and the other languages a few weeks after the English release. As soon as the bits are out, we will blog about each of these features in more detail.

    Regards,
    The StreamInsight Team

  • StreamInsight Talk at TechEd 2011

    Torsten Grabs, one of our fearless leaders, has given a well-received presentation at TechEd 2011 in Atlanta this week. He talked about the StreamInsight value proposition, its event-processing paradigm, and ran a bunch of great demos from various verticals (you can find one of the samples here in a previous posting). Also, Torsten presented a glimpse into our upcoming version 1.2 as well as our plans for the Cloud. You can watch the talk and download the presentation on the Channel9 site: Microsoft StreamInsight: Introduction to Complex Event Processing with SQL Server 2008 R2 StreamInsight (The video frame is too big to embed here with our current blog theme).

    Regards,
    The StreamInsight Team

Powered by Community Server (Commercial Edition), by Telligent Systems
  Privacy Statement