Kinect your Metro Style App

As part of our Software Design project in Imagine Cup competition, we develop a demo Metro Style Application controlled by Kinect.

The main idea is to tell Metro Style App communicate with external environment via Kinect Sensor (at present with Kinect for Xbox 360, Kinect for Windows in a future blog post).

Kinect SDK is implemented in only .Net and C++ code that’s not supported on Metro Style Application.

So, we introduce middleware architecture with Publish-Subscribe pattern to handle communication between Kinect Sensor and Metro Style App.

Foremost, a short introduction of Publish-Subscribe pattern from Sacha Krakowiak (http://proton.inrialpes.fr/~krakowia/):

The PUBLISH-SUBSCRIBE pattern is a more general form of OBSERVER. It defines two “roles”, Publisher (event source) and Subscriber (event sink). A given entity may take up either role, or both.

Events generated by publishers are selectively notified to subscribers. The selection is achieved by defining event classes, i.e. sets of events that satisfy some conditions, to be discussed later. A subscriber may register its interest for a class of events by subscribing to that class. When a publisher generates an event E, all subscribers that previously subscribed to an event class to which E belongs receive an asynchronous notification of the occurrence of E. Reacting to this notification is locally arranged by each subscriber (e.g. by associating a specific handler with each class of events, etc.).

Figure 1-Solution architecture

We have use the EventAggregator implementation of PRISM project in codeplex http://compositewpf.codeplex.com. So, inside middleware part, we have two services loosely-coupled that one handle Kinect detected gesture and transfer it to a second part that command a Metro Style Application. Communication was based on W3C Web Socket protocol which supported by Metro Style App and .Net App. In Metro side, a HTML5 Web Worker was used to handle commands and manipulate metro application.

Our Metro Style Application is based on “Split Application Template” from Visual Studio 11 using JavaScript and run on Windows 8 Consumer Preview.

This video show a demo of our application:

Now, we will be showing some snippet code from Metro App and middleware part:

Middleware Part:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Net.WebSockets;
using System.Net;
using System.Threading;
using Microsoft.Practices.Composite.Events;

namespace ServerRegistry
{
    public class Registry
    {
        public WebSocketServer ServerWebSocket { get; set; }

        public WebSocketClient ClientWebSocket { get; set; }

        public string ServerWebSocketUri { get; set; }

        public string ClientWebSocketUri { get; set; }

        private IEventAggregator eventAggregator=null;

        public Registry()
        {
            eventAggregator = new EventAggregator();

            ServerWebSocketUri = "http://localhost:8052/server/";
            ClientWebSocketUri = "http://localhost:8051/client/";

            ServerWebSocket = new WebSocketServer(eventAggregator);
            ClientWebSocket = new WebSocketClient(eventAggregator);

            Task.WhenAll(ServerWebSocket.Start(ServerWebSocketUri), ClientWebSocket.Start(ClientWebSocketUri));
        }
    }
}

HTML5 Web Worker:

function start() {
    if (streamWebSocket) {
        log("Already started.");
        return;
    }
    log("Starting");

    var webSocket = new Windows.Networking.Sockets.StreamWebSocket();
    webSocket.onclosed = onClosed;

    var uri = new Windows.Foundation.Uri("ws://localhost:8051/client/");

    log("Connecting to: " + uri.absoluteUri);

    webSocket.connectAsync(uri).then(function () {
        log("Connected");
        streamWebSocket = webSocket;
        dataWriter = new Windows.Storage.Streams.DataWriter(webSocket.outputStream);
        dataReader = new Windows.Storage.Streams.DataReader(webSocket.inputStream);
        // When buffering, return as soon as any data is available.
        dataReader.inputStreamOptions = Windows.Storage.Streams.InputStreamOptions.partial;
        countOfDataSent = 0;
        countOfDataReceived = 0;

        //  Connect to the server
        Connect();

        readIncoming();

    }, function (e) {
        log("Failed to connect: " + getError(e));
    });
}

function Connect() {
    try {
        dataWriter.writeString(data);
        dataWriter.storeAsync();
    }
    catch (e) {
        log("Sync write error: " + getError(e));
    }
}

function readIncoming(args) {
    // Buffer as much data as you require for your protocol.
    dataReader.loadAsync(100).then(function () {
        var incomingBytes = dataReader.readString(dataReader.unconsumedBufferLength);
        postMessage(incomingBytes);

        readIncoming(); // Start another read
    }, readError);
}

Full code on codeplex.com: http://kinectmetroapp.codeplex.com/

Contact Information:

Advertisements
  1. Viewing Kinect Data in the New Windows 8 UI « Clatter from the Byte Kitchen

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: