Using Azure Stream Analytics to tap into an Event Hub data stream

The pre-requisite is to make sure that you have requested for Stream Analytics preview if you have not already done so.

1. Create a Stream Analytics job. Jobs can only be created in 2 regions while the service is under preview.

sa-1

2. Add an input to your job., This is the best part because we get to tap into an Event Hub to analyse the continuous sequence of event data stream and potentially transform it by the same job.

sa-2-addinput

3. Select Event Hub.

sa-3-addeh4. Configure the event hub settings. You could also tap into an event hub from a different subscription.

sa-4-ehsettings

5. Due to my event hub’s event data being serialized in JSON format, this is exactly what I will select in the following step.

sa-5-serializationsetting

Under the Query tab, I just insert a simple query like the following.

SELECT * FROM getfityall 

I’m not doing any transformation yet, I just want to make sure that the event data sent by my Raspberry Pi via Event Hub’s REST API is done properly.

Next on the list of steps is to setup the output in the job.

6. Add an output to your job. I’m using a BLOB storage just to keep things simple so that I could use open the CSV file in Excel to take a look at the data stream.

sa-6-output

7. Setup the BLOB storage settings.

sa-7-blobsettings

8. Specify the serialization settings. I’m choosing CSV for obvious reason stated above.

sa-8-serialization

As I pump telemetry data from my Raspberry Pi, I could see my CSV file created/updated. Just go to the container view in your BLOB storage, and download the CSV file.

sa-9

Below is what my event data stream looks like.  It shows event data points captured from two Raspberry Pis, one using the MPL3115 temperature sensor (part of the Xtrinsic sensor board), and another using MCP9808 temperature sensor. The fun begins as I could write some funky transformation logic in the query and do some real-time complex event processing.

streamanalytics-temperature

Send accurate temperature data from Raspberry Pi to Azure Event Hub

This is a follow-up post to my previous one which is about sending accurate temperature data from the MCP9808 temperature sensor board to an Azure Event Hub. This is done through the MCP9808 Python library provided by Adafruit, and also one which I have repurposed from the Xtrinsic sensor board. This is the updated version. Inside the ~/Adafruit_Python_MCP9808/examples directory, I made a copy of the simpletest.py script as send2eventhub.py.

First step is to import eventhubms and socket.

import socket
import eventhubms

The parts which I had modified are the following:

# Loop printing measurements every second.
print 'Press Ctrl-C to quit.'
hubClient = eventhubms.EventHubClient()
parser = eventhubms.EventDataParser()
hostname = socket.gethostname()

while True:
temp = sensor.readTempC()
message = 'Temperature: {0:0.3F}'.format(temp)
body = parser.getMessage(message,"MCP9808")
print "\n"+body+""
hubStatus = hubClient.sendMessage(body,hostname)
print "[RPi->EventHub]\t[Data]"+message+"\t[Status]"+str(hubStatus)
time.sleep(1.0)

The event hub client is no different that the one I described in my previous post for sending data from the Xtrinsic sensor board. The event data is sent to the event hub via its REST API. It’s pretty simple.

Then to make sure that the event data is sent correctly and can be consumed from the event hub, I made use of Azure Stream Analytics which is the simplest way to set the event hub as an input. Otherwise you would have to write code to either directly receive events from the event hub or use the EventProcessorHost like how I used in my scalable event hub processor in an Azure worker role.

Since this is a separate topic by itself, I will be writing another post about how to create a new Stream Analytics job to add an event hub as the data stream from which events data will be consumed and transformed by the Stream Analytics job.

Using more accurate temperature sensor with my Raspberry Pi

When I stumbled upon the MCP9808 precision temperature sensor, I was sold based upon its promise of up to 0.25 degrees Celsius accuracy. Just like my Freescale Xtrinsic sensor board, there’s a Python library that that allows me to use the MC9808 temperature sensor with my Raspbbery Pi.

My one and only challenge was doing a proper soldering work because the MCP9808 temperature sensor board comes as a breakout board. I referred to this tutorial to prepare the header strip (with the pins) that comes with the sensor board, inserted it into my breadboard, placed the breakout board over the pins, and solder! It was tough. Being a software guy, there is a big mental hurdle because there is not much undo feature or “let’s try to step into this code to see if it works or fails”. It’s a big challenge to hold a soldering iron and try not to apply too much heat when trying to apply solder onto the pins, else the sensor board would be toast.

The end result: I did a better job with the first breakout board (I bought 3 sensor boards as contingency), and the other two boards were not so lucky under the soldering iron.

Here’s what the MCP9808 sensor looks like on my breadboard with the 4 connections to my Raspberry Pi through the very usefulT-Cobbler Plus connector.

mcp9808

Here’s a photo of the GPIO cable nicely coming out of my Raspberry Pi clear case and connecting to the T-Cobbler Plus.

mcp9808-whole

Here’s a “creative way” of making sure the clear case cover still protects most of my Raspberry Pi even with the protruded Xtrinsic sensor board. Priceless 🙂

xtrinsicsensorboard-rubberband

In my follow up post, I will describe how I send the temperature data stream to an Azure Event Hub, and then do simple analytics of the data on transit using Azure Stream Analytics.

Installing Windows Developer Program for IoT image on Intel Galileo Gen 2

In order to setup and install the Windows image onto Intel Galileo Gen 2, the best way is to follow the setup instructions in the Windows Developer Program for IoT.
Please be very careful with the folder where you will save the .cmd and .wim file. The best is to store the downloaded files in a simple to find folder, and a folder without any space in its name. Otherwise the .cmd file will not execute properly.
You may also want to rename the .wim file to a simpler name like galileo_v2.wim. When the .cmd file executes correctly, you should see the following:

C:\Temp>apply-BootMedia.cmd -image galileo_v2.wim -destination D:\ -hostname galileofai -password xxxxxx

C:\Temp>rem **************************************************************************

C:\Temp>rem ** Copyright (c) Microsoft Open Technologies, Inc.  All rights reser
ved.

C:\Temp>rem ** Licensed under the BSD 2-Clause License.

C:\Temp>rem ** See License.txt in the project root for license information.

C:\Temp>rem **************************************************************************

**** Temporary changing time zone to 'Pacific Standard Time'
**** Fat32 is local time based, and the images are created in a pacific time zone. If there is a mismatch Windows will bug check after 5 minutes.
**** Set-up work folder: C:\Users\hoongfai\AppData\Local\Temp\apply-BootMedia-15483
**** Retrieveing C:\Temp\galileo_v2.wim
****          to C:\Users\hoongfai\AppData\Local\Temp\apply-BootMedia-15483

Deployment Image Servicing and Management tool
Version: 6.3.9600.17031

Mounting image
[==========================100.0%==========================]
The operation completed successfully.
**** Customizing image C:\Users\hoongfai\AppData\Local\Temp\apply-BootMedia-15483\galileo_v2.wim
****        mounted at C:\Users\hoongfai\AppData\Local\Temp\apply-BootMedia-15483\galileo_v2.wim.mount

Deployment Image Servicing and Management tool
Version: 6.3.9600.17031

Saving image
[==========================100.0%==========================]
Unmounting image
[==========================100.0%==========================]
The operation completed successfully.
**** Applying image C:\Users\hoongfai\AppData\Local\Temp\apply-BootMedia-15483\galileo_v2.wim
****             to D:\

Deployment Image Servicing and Management tool
Version: 6.3.9600.17031

Applying image
[==========================100.0%==========================]
The operation completed successfully.
**** Mounting D:\\Windows\System32\config\SYSTEM
****       to HKEY_USERS\Galileo-15483-SYSTEM
**** Setting hostname to galileofai
**** Restoring time zone to 'Pacific Standard Time'
****
****   Successfully applied C:\Temp\galileo_v2.wim
****                     to D:\
****
****              hostname: galileofai
****              timezone: Pacific Standard Time
****              Username: Administrator
****              Password: xxxxxxxxxx
****
**** Done.