Bridging network adapters to share Internet connection with your RPi2/Windows 10 IOT Core

In my previous post, I shared about the workaround in order to share Internet connection via ICS when the option is disabled due to domain group policy. I learned that there is an easier option to share the Internet connection of your Wi-Fi adapter to devices connected to your Ethernet adapter, like a Raspberry Pi running Windows 10 IoT Core. Here are the steps:

  1. Open Network and Sharing Center.
  2. Change adapter settings.
  3. Select both your Wi-Fi and Ethernet adapter.
  4. Right-mouse click and select the option to bridge
  5. Make sure that the Internet Protocol Version 4 (TCP/IPv4) properties are set to “Obtain an IP address automatically”.
  6. In order to find your Windows 10 IoT Core device’s IP address, run Windows10IoTCoreWatcher. Windows10Io
7. Right-mouse click on the board item, and select “Copy IP address”.
8. Follow the PowerShell documentation here to use PowerShell to connect to your running device. You can also follow the instructions here to use SSH to connect to your device.
If this method fails, please fall back to the ICS setup workaround.

Windows 10 IoT Core / Raspbian on Raspberry Pi 2 using Windows 10’s Internet Connection Sharing (ICS)

You just got yourself a Raspberry Pi 2 (RPi 2). You could be running Raspbian or Windows 10 IoT Core. You don’t have access to a hub/switch/router to connect the RPi 2 for Internet connection. The next best solution is by connecting the RPi 2 to your PC via Ethernet and sharing your Wi-Fi’s internet connection via Internet Connection Sharing (ICS). When you go to the Wi-Fi adapter properties, you got some bad news:WiFi-ICS-disabled

What do you do? Here’s a workaround which is definitely NOT endorsed by your friendly network administrator, but it works. NOTE: This workaround is NOT permanent and it is not meant to flout your network administrator’s group policy because they are rules for good reasons; security, etc. 

  1. To enable sharing on the WiFi adapter, run the following command in a Command Prompt run as Administrator.
netsh wlan set hostednetwork mode=allow

 

  1. Run regedit. Go to Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Network Connections. Edit NC_ShowSharedAccessUI, and enter 1 in value data.
  1. Go back to Wi-Fi adapter properties, now you will see the Sharing tab. In case you don’t see the Sharing tab, this could be due to the reason that you have not connected your Ethernet adapter (for those that comes in a USB dongle). You need at least two network adapters to be present in order to do ICS. Check the box that says “Allow other network users to connect through this computer’s Internet connection”.

WiFi-ICS-avail

  1. Go to your Ethernet adapter properties. Check out the Internet Protocol Version 4 (TCP/IPv4) Properties. You will see the following preconfigured for you. Do not change these settings.

ethernet-ipv4settings

 

  1. Connect the network cable between your Raspberry Pi 2 and your Windows 10 machine via the Ethernet port.
  1. When you start up your Windows 10 IoT Core on your Raspberry Pi 2, you will see that the IP address is dynamically set to an IP address like 192.168.137.2. Voila, this means that you have Internet connection shared with your RPi 2.
RPi2-win10-dashboard

7. Follow the PowerShell documentation here to use PowerShell to connect to your running device. You can also follow the instructions here to use SSH to connect to your device.

From <http://ms-iot.github.io/content/en-US/win10/SetupRPI.htm>

  1. To make sure ICS is enabled properly, just ping any Internet site.

pinganysite

To start sending events from Windows 10 IoT Core to Azure IoT Hub:

In your Visual Studio 2015 UWP project, go to your project properties, and configure the remote machine IP.
vs2015-rpi2-props

Configure remote machine IP as 192.168.137.2, or any other IP address which you got from Step 6. Run your project.

Check Device Explorer for event has been received at the IOT Hub.deviceexplorer

Finally, a word of caution. If you don’t see ICS sharing available in your Wi-Fi adapter settings anymore, this is because the group policy has been re-applied to your machine. That’s ok, it’s meant to protect your machine after all. When you need to enable ICS for another instance, just re-do the steps above.

Using Azure Stream Analytics to tap into an Event Hub data stream

The pre-requisite is to make sure that you have requested for Stream Analytics preview if you have not already done so.

1. Create a Stream Analytics job. Jobs can only be created in 2 regions while the service is under preview.

sa-1

2. Add an input to your job., This is the best part because we get to tap into an Event Hub to analyse the continuous sequence of event data stream and potentially transform it by the same job.

sa-2-addinput

3. Select Event Hub.

sa-3-addeh4. Configure the event hub settings. You could also tap into an event hub from a different subscription.

sa-4-ehsettings

5. Due to my event hub’s event data being serialized in JSON format, this is exactly what I will select in the following step.

sa-5-serializationsetting

Under the Query tab, I just insert a simple query like the following.

SELECT * FROM getfityall 

I’m not doing any transformation yet, I just want to make sure that the event data sent by my Raspberry Pi via Event Hub’s REST API is done properly.

Next on the list of steps is to setup the output in the job.

6. Add an output to your job. I’m using a BLOB storage just to keep things simple so that I could use open the CSV file in Excel to take a look at the data stream.

sa-6-output

7. Setup the BLOB storage settings.

sa-7-blobsettings

8. Specify the serialization settings. I’m choosing CSV for obvious reason stated above.

sa-8-serialization

As I pump telemetry data from my Raspberry Pi, I could see my CSV file created/updated. Just go to the container view in your BLOB storage, and download the CSV file.

sa-9

Below is what my event data stream looks like.  It shows event data points captured from two Raspberry Pis, one using the MPL3115 temperature sensor (part of the Xtrinsic sensor board), and another using MCP9808 temperature sensor. The fun begins as I could write some funky transformation logic in the query and do some real-time complex event processing.

streamanalytics-temperature

Using more accurate temperature sensor with my Raspberry Pi

When I stumbled upon the MCP9808 precision temperature sensor, I was sold based upon its promise of up to 0.25 degrees Celsius accuracy. Just like my Freescale Xtrinsic sensor board, there’s a Python library that that allows me to use the MC9808 temperature sensor with my Raspbbery Pi.

My one and only challenge was doing a proper soldering work because the MCP9808 temperature sensor board comes as a breakout board. I referred to this tutorial to prepare the header strip (with the pins) that comes with the sensor board, inserted it into my breadboard, placed the breakout board over the pins, and solder! It was tough. Being a software guy, there is a big mental hurdle because there is not much undo feature or “let’s try to step into this code to see if it works or fails”. It’s a big challenge to hold a soldering iron and try not to apply too much heat when trying to apply solder onto the pins, else the sensor board would be toast.

The end result: I did a better job with the first breakout board (I bought 3 sensor boards as contingency), and the other two boards were not so lucky under the soldering iron.

Here’s what the MCP9808 sensor looks like on my breadboard with the 4 connections to my Raspberry Pi through the very usefulT-Cobbler Plus connector.

mcp9808

Here’s a photo of the GPIO cable nicely coming out of my Raspberry Pi clear case and connecting to the T-Cobbler Plus.

mcp9808-whole

Here’s a “creative way” of making sure the clear case cover still protects most of my Raspberry Pi even with the protruded Xtrinsic sensor board. Priceless :)

xtrinsicsensorboard-rubberband

In my follow up post, I will describe how I send the temperature data stream to an Azure Event Hub, and then do simple analytics of the data on transit using Azure Stream Analytics.

How to send sensor data from Raspberry Pi to Azure Event Hub using a Python script

This post follows what I intended to do which is to pump sensor data consisting of temperature and altitude readings from the Xtrinsic sensor board to my Azure Event Hub named getfityall. The sensor board comes with some Python scripts already as you have seen in my earlier posts. Coupled with Microsoft Azure Python packages from the SDK, I could easily reuse a very nifty Python script I found from Kloud’s blog, a very competent Microsoft cloud partner, to send sensor data to my Azure Event Hub. The repurposed Python script looks like the following:

import sys
import azure
import socket

from azure.servicebus.servicebusservice import (
  ServiceBusSASAuthentication
  )

from azure.http import (
  HTTPRequest,
  HTTPError
  )

from azure.http.httpclient import _HTTPClient

class EventHubClient(object):

  def sendMessage(self,body,partition):
    eventHubHost = "youreventhubname.servicebus.windows.net"

    httpclient = _HTTPClient(service_instance=self)

    sasKeyName = "yourprocessorname"
    sasKeyValue = "yourprocessoraccesskey"

    authentication = ServiceBusSASAuthentication(sasKeyName,sasKeyValue)

    request = HTTPRequest()
    request.method = "POST"
    request.host = eventHubHost
    request.protocol_override = "https"
    request.path = "/youreventhubname/publishers/" + partition \
    + "/messages?api-version=2014-05"
    request.body = body
    request.headers.append(('Content-Type', \
    'application/atom+xml;type=entry;charset=utf-8'))

    authentication.sign_request(request, httpclient)

    request.headers.append(('Content-Length', str(len(request.body))))

    status = 0

    try:
        resp = httpclient.perform_request(request)
        status = resp.status
    except HTTPError as ex:
        status = ex.status

    return status

class EventDataParser(object):

  def getMessage(self,payload,sensorId):
    host = socket.gethostname()
    body = "{ \"DeviceId\" : \"" + host + "\""
    msgs = payload.split(",")

    for msg in msgs:
      sensorType = msg.split(":")[0]
      sensorValue = msg.split(":")[1]
      body += ", "
      body += "\"SensorId\" : \"" + sensorId \
           + "\", \"SensorType\" : \"";
      body += sensorType + "\", \"SensorValue\" : " \
           + sensorValue + " }"
    return body

I saved and named this Python script as sendtelemtry.py.

Then in the mpl3115a2.py script, add the following import statements:

import socket
import sendtelemetry

At the end of the script, replace the Python code with the following:

mpl = mpl3115a2()
mpl.initAlt()
#mpl.initBar()
mpl.active()
time.sleep(1)
while 1:

        #print "MPL3115:", "\tAlt.", mpl.getAlt(), "\tTemp:", mpl.getTemp()
        hubClient = sendtelemetry.EventHubClient()
        parser = sendtelemetry.EventDataParser()
        hostname = socket.gethostname()
        message = "temperature:"+repr(mpl.getTemp())+",altitude:"+repr(mpl.getAlt())
        body = parser.getMessage(message,"mpl3115")
        hubStatus = hubClient.sendMessage(body,hostname)
        print "[RPi-&gt;AzureEventHub]\t[Data]"+message+"\t[Status]"+str(hubStatus)
        time.sleep(0.1)

Then execute this script by doing

sudo python mpl3115a2.py

The Azure Event Hub REST API returns HTTP status code to indicate the result of the send event REST call. A HTTP status code of 201 means success. Read more about the send event REST call here. You can monitor your Azure Event Hub dashboard to see the incoming messages.

In the next post I will share more about what I’m doing with the preview feature of Azure Stream Analytics to do a toll-gate analysis of event data in transit. Thus far I had only been doing descriptive analytics of data at rest. This ought to be interesting because it requires a different understanding of what I would like to analyze.

Fresh raspberry pi at my service

I am super glad that the items I ordered from Element14 arrived overnight. I got not one but 3 MEMS Sensor board, and it is working well. Proof that I somewhat “overused” the previous board so much so that the temperature and altitude reading stopped working.

pi@raspberryclay ~/rpi_sensor_board $ sudo python mpl3115a2.py
MPL3115: Alt. 100.24 Temp: 28.144
MPL3115: Alt. 100.24 Temp: 28.144
MPL3115: Alt. 100.24 Temp: 28.144
MPL3115: Alt. 100.24 Temp: 28.144
MPL3115: Alt. 101.32 Temp: 28.128
MPL3115: Alt. 101.32 Temp: 28.128
MPL3115: Alt. 101.32 Temp: 28.128
MPL3115: Alt. 101.32 Temp: 28.128
MPL3115: Alt. 101.32 Temp: 28.128
MPL3115: Alt. 101.32 Temp: 28.128
MPL3115: Alt. 101.32 Temp: 28.128

I got a new Raspberry Pi too, it’s for my co-worker, Clayton. It’s pretty simple to clone the micro-SD card from raspberryfai and write it on to the new one. Just use the Win32DiskImager. It works like a charm.

Just to make sure my ISS agent which sends the right telemetry data which consists of MEMS sensor temperature/altitude reading, here’s a good OData feed of the data all captured in the cloud.

raspclayiss

I’m thinking of the next steps in my experimentation such as:

  • Revert to plain old way of sending telemetry data from my Raspberry Pis to Azure Event Hubs using a friendly AMQP client/library.
  • Do real-time analytics of multiple data streams from Fitbit, Strava, Raspberry Pi (temperature/altitude). I’ve been mucking around with the new preview of Azure Stream Analytics to some success.

I have 2 more packages of nice toys to be delivered. I’d ordered an Arduino Starter Kit, and 2 Intel Galileo 2, among other nifty breakout boards, sensors, and kits. Stay tuned.

 

Sending sensor board data from Raspberry Pi to Intelligent Systems Service (ISS)

I have been mucking around with the Xtrinsic-sense board which is an add-on to my Raspberry Pi. In my previous post, I had executed the Python scripts that came with the sensor library that I cloned out of this github. There is a good article that explains about how to enable Xtrinsic-sense board in Raspbian.

However my aim is to get the Intelligent Systems Service (ISS) sample C application to invoke the sensor board shared library to retrieve the altimeter and temperature values, assign that to a string property within the data model, then send it to ISS. The sensor board library comes in the form of a shared library called sensor.so.

Hence I started on a journey to try to invoke the functions within the sensor.so out of mirroring the Python code, but in the sample C application. Now I must admit that Python coding skills is non-existent, and my C coding skills were really rusty. The last time I wrote a C/C++ application was in 2000/2001! What’s surprising is that I haven’t completed forgotten C, it’s like riding a bike, suddenly remnants of makefile, linking with dynamic/static libraries, include files, and function pointers in C slowly came back to memory.

But I’m still hopeless in Python, I gave up because I didn’t understand what the following code does except that it does some byte shifting, but for what reason, it beats me.

def getTemp(self):
    t = self.readTemp()
    t_m = (t >> 8) & 0xff;
    t_l = t & 0xff;

    if (t_l > 99):
        t_l = t_l / 1000.0
    else:
        t_l = t_l / 100.0
    return (t_m + t_l)

I knew there has to be an easier way to invoke the sensor board library in C, but where’s the sample code? After much searching, I found a reverse-engineered sensor.so with C source code! How amazing, thanks to this bloke named Lars Christensen. It was just hot off the oven, to be specific just last Saturday 13 Sept. To get started, all I had to do was to clone his source from github. In my raspberrywifai SSH session, I issued the following command (provided you have already installed git too).

git clone https://github.com/larsch/rpi-mems-sensor.git

There was a slight problem. The mpl3115a2.h header file was empty. No worries, just go to the github repository at https://github.com/larsch/rpi-mems-sensor, click on mpl3115a2.h, copy the content, and paste it in the file in your working directory on RaspPi.

You also need to download and extract the bcm2835 library. Then just follow the steps on how to make it. This creates a static library called libbcm2835.a. Next, make the reverse-engineered sensor board library so that you create a new sensor.so, and libmemssensor.a. I was only interested in the static libraries of libbcm2835.a and libmemssensor.a so that I can compile test.c. There wasn’t a proper make statement for test so I just issued my own gcc command.

gcc -Wall -I /home/pi/rpi-mems-sensor  -L. -lmemssensor -lbcm2835 -static test.c -o test libmemssensor.a libbcm2835.a

Do not just run ./test because initializing bcm2835 requires elevated permission. If you do that you will get the following error:

pi@raspberryfai ~/rpi-mems-sensor $ ./test
bcm2835_init: Unable to open /dev/mem: Permission denied

So do this instead:

sudo ./test

And voila! I get the altimeter and temperature reading.

raw alt = 11968
alt: 46.750000, temp: 23.812500
raw alt = 11968
alt: 46.750000, temp: 23.812500
raw alt = 11968
alt: 46.750000, temp: 23.812500
raw alt = 11968

The Python code which I just couldn’t comprehend is the equivalent of the following C code:

 

double getTemp() {
    int t = MPL3115A2_Read_Temp();
    int t_m = (t >> 8) & 0xFF;
    int t_l = t & 0xFF;
    if (t_m > 0x7f) t_m = t_m - 256;
    return t_m + t_l / 256.0;
}

It still beats me what it does, but that’s alright, I just wanted this code to work within the ISS sample C application. Next I embedded this code within the DATAPROVIDER_RESULT DATA_PROVIDER_INIT function. I didn’t want to create a new data model so I reused the sample data model, and chose to assign the altimeter and temperature reading to the stringProperty, as shown in the following code:

// setting StringProperty which contains the alt and temperature
bcm2835_init();
MPL3115A2_Init_Alt();
MPL3115A2_Active();
sleep(1);
char tempinfo[1024];
snprintf(tempinfo, sizeof(tempinfo), "alt: %f, temp: %f\n", getAlt($
stringProperty = tempinfo;

Compile (also includes modifying the makefile to ensure the sensor board header files and static libraries are correctly linked), and then run the ISSAgent app, and wait for it……..

Info: Microsoft Intelligent Systems Agent Console
Info: Using device endpoint: getfityall.device.intelligentsystems.azure.net
Info: Registering device: Name=raspberrywifai, FriendlyName=raspberrywifai, ModelName=Contoso.Device, Description=raspberrywifai
Info: x-ms-activity-id: 557374f8-2342-4bfb-bb7b-ce19199fa77e
Info: Getting endpoints for device raspberrywifai
Info: x-ms-activity-id: 229227bf-483b-4914-a613-e447bb7caca5
Info: Got ingress queue namespace=uswepm01c02x0sb20-0.servicebus.windows.net, name=getfityall/raspberrywifai
Info: Got commands queue namespace=uswepm01c02x0sb20-0.servicebus.windows.net, topic path=DeviceBank1, subscription name=raspberrywifai
Info: Got per device token=[secret]
raw alt = 11152
Info: Uploading message 2e5f2634-bee2-4e88-b1c3-3ed991f25f41 for device raspberrywifai (payload={"@iot.devicename":"raspberrywifai", "value":[ {"@odata.context":"Contoso\/$metadata#Device\/$entity", "@odata.id":"Device('raspberrywifai')", "@Microsoft.IntelligentSystems.Vocabulary.V1.Timestamp":"2014-09-17T00:34:41Z", "structProperty":{"simpleField":17, "structField":{"int32Field":10, "int64Field":34, "doubleField":40.000000000000000, "stringField":"Wed Sep 17 10:34:40 2014\u000A", "guidField":"0F1E2D3C-4B5A-6978-8796-A5B4C3D2E1F0", "binaryField":"AAAA", "dateTimeOffsetField":"2014-09-17T10:34:40Z", "booleanField":true}}, "stringProperty":"alt: 43.562500, temp: 22.812500\u000A", "dateTimeOffsetProperty":"2014-09-17T10:34:40Z"} ]}) ...Info: Success

Or more specifically:

"stringProperty":"alt: 43.562500, temp: 22.812500\u000A"

Woo hoo! I got it. Now moving on to the next steps. This opens up other possibilities such as sending temperature as an event on a pre-determined interval, and also set up alarm when temperature falls below or goes above a certain value. The fun begins! :)

How to get Raspberry Pi to work with TP-Link TL-WN721N USB Wi-Fi adapter

Wow! It’s taken me over a week to find just the right combo of advice from a number of really helpful blogs to help me setup raspberryfai with mt TP-Link TL-WN721N USB WiFi adapter. I tried so many different steps that I couldn’t remember what I’d done. I thought that the wicd-curses utility would work like a charm but it hung my RasPi a few times.

Now that it’s setup, I shall call it raspberrywifai. :)

The blog which was most helpful was this post written almost 2 years ago. Just follow the steps especially on how to get the firmware for the TP-Link USB adapter. The only exception was that the following command didn’t work in my case,

pre-up wpa_supplicant -Dwext -i wlan0 -c /etc/wpa_supplicant.conf -B

Instead I just have to change it to the simpler form of (my wpa_supplicant.conf file is kept in its own folder):

wpa_conf /etc/wpa_supplicant/wpa_supplicant.conf

In order to get the psk value, you can’t just type in your WPA2 passphrase, you need to run the following command:

sudo wpa_passphrase ssid passphrase

To be sure that it is indeed connected and seen as a USB device on my RasPi, I run lsusb, a command I learned from  this post.

Now that raspberryfai has become raspberrywifai, I can bring it out for a spin this weekend.

Intelligent Systems (at your) Service

I was really excited that my application for the Microsoft Azure Intelligent Systems Service (ISS) Limited Public Preview (LPP) had been approved (my apologies for all the 3-letter acronyms which I will be repeating all over my posts from now on). What the ISS LPP entitles me are the following:

  • Access to the ISS service which is enabled for use with my Azure account. The ISS service icon is now available in the list of services (on the left in my Azure Management Portal) and it looks like this:

0-issazure

  • Download the ISS SDK along with a couple other utilities such as DeviceMonitoring and the Contoso Home Automation -part 1 sample/demo (to be checked out soon)
  • Participate in a private forum.  According to the welcome email, “You are more likely to get a response in a reasonable timeframe, since you are not reliant on an individual being online and able to answer your question”, which sounds great.

I went ahead to create an ISS service for GetFitY’all. However I could not share too much details and the screen clipings because according to confidential information clause in the EULA,  the software and service, including the user interface, features and documentation, are confidential and proprietary to Microsoft and its suppliers.

The real fun happens inside my raspberryfai. The RPi runs Raspbian, a customized version of Debian used to run on the RPi. I used scp to copy the ISSAgent_C_Samples folder into my raspberryfai. Then I compile the sample ISS agent and run it. The agent sends messages to my ISS account. My next step is to embed the ISS managed library into my GetFitYall device gateway, which was implemented as a WebJob described in one of my previous posts, and do the same in sending the activity data points to my ISS account. Previously I implemented a simple message pump functionality in the WebJob to send activity data points (pulled from Fitbit and Strava APIs) asynchronously to an Azure Event Hub via AMQP. Then I have Azure Worker Role instance(s) to ingest the event hub messages by persisting into respective Azure Storage Tables.

raspberryfai at your service

My apologies for the corny title of this post, but since Fai rhymes with Pi, I couldn’t resist not naming my li’ juice fruit as RaspberryFai. See my SSH session below:

raspberryfai

 

I lost a couple days of precious time in tinkering with my Raspberry Pi (RPi) because I got some internet connection issues but it’s been resolved since yesterday. First thing I did was to ensure I could make an SSH connection to my “headless” Pi that’s sitting next to and connected to my ADSL router. Next on the list is to download all the essential packages that I need to get started. The apt-get tool works like a charm once you have internet connection. No need to hunt for all the dependencies, just know what I want and apt-get install away.

I’d installed the following packages so far:

  • wicd-curses – easy way to get WiFi enabled on my raspberryfai. My TP-Link TL-WN721N WiFi USB dongle works well with RPi. It’s not exactly compact in size but then I don’t want to get more gadgets yet so this will suffice at the moment.
  • mono
  • C# interactive shell – both mono and this fromhttp://www.raspberry-sharp.org/
  • build-essential
  • node-js – by following this setup tutorial.
  • node-red: while i’m at it, I may as well check out node-red which is presumably the simplest form of an open source visual editor for wiring the internet of things produced by IBM

Before I end up downloading a bunch of non-essential packages, I better start wiring a quick IoT scenario using raspberryfai.