STREAM On

Jennifer Widom, a Stanford computer science professor came by and gave a fun talk/demo on Monday about her work with the STREAM group. There were a couple interesting aspects to the talk, the first was some of the data stream extensions that they've added to SQL, dubbing it CQL. The biggest change was in the FROM list, instead of just specifying a data source, its a source + a window of time. So you might say,

...
FROM tablename [Range 1 Day]
...

in order to specify over how much to operate on. There were other constructs, like specifying inserts and deletes which I unfortunately didn't quite grasp their real usefulness.

The really sexy stuff was the second part of the talk. Remember this is about data stream processing. You don't run a query and have it finish. Instead you register a query with the system and get either periodic updates or continuous results depending on what you asked for. New queries are merged in with the existing query plan in an optimal way. So if two queries require a particular join, its only done once for both of them before splitting off to their respective operators.

Unfortunately, this being research, they've focused on the sexy stuff, and swept a lot of the details under the rug. There were a number of questions about the inevitable overload of data, which the system pretty much drops it on the floor for the moment. But all in all, some very neat technology. Worth a look.