Having fun with ELL

ELL is a new Open Source library from Microsoft Research.  I’m having lots of fun working on it and learning all about Convolutional Neural Networks.  The first time you train one and try it out it definitely feels like magic.  How can a bunch of numbers correctly classify the type of object in a photo?  Granted we’re talking millions of numbers, so it’s not that magical, but it is still fun.

Now, of course, I have to combine this with my love of DGML graphs. So in this post I will outline how I’ve used ELL to execute neural networks trained on the MNist dataset.  MNist is a classic benchmark for neural networks and it provides a dataset for “hand written digits”.  This dataset is small since each image is only 28×28 pixels, which makes for a great “first time developer experience” with neural networks, since you don’t have to wait for terabytes of images to be downloaded!

For example, here’s one of their test images which contains a hand written digit “8”:

It turns out the Microsoft CNTK library is an excellent library for training  Neural networks and there is a CNTK sample that can be used to train an MNist model.  See install_mnist.py.  After you run that you will have the MNist dataset and then you can run SimpleMNIST.py to train a CNTK model on that dataset.   On my machine I get a file named mnist.cntk which is about 640kb.

So the next thing I want to do is run this on a Raspberry Pi, so I used the ELL library to do that.  If you follow the Getting Started tutorial you will see how easy it is to build ELL, import a model from CNTK and build a cross-compiled module that will run on the Raspberry Pi.  So in my case I simply do this (after building the ELL repo):

python d:\git\ell\ell\tools\importers\CNTK\cntk_import.py mnist.cntk

This gives me an “mnist.ell” model, so what is inside this model? You can use the ELL print tool with -dgml and get this picture:

Of course, this is a very simple neural network.  Try the DGML graph on some of the more complex deep neural networks.   You can then compile this model using the ELL model compiler:

python d:\git\ell\ell\tools\wrap\wrap.py mnist.ell --target pi3

This gives me a ‘pi3’ folder already setup with CMakeLists.txt for building on the pi.  You can also copy Demo.py and DemoHelpers.py from ELL\tools\utilities\pythonlibs to get a quick start in using this python module.  Once on the pi do the using cmake build thing outlined in the ELL tutorials, and then we can test it out.

To test it out we can run this python script:

import cv2

import numpy as np

import sys

sys.path += [ “d:/Temp/MNist/host” ]

sys.path += [ “d:/Temp/MNist/host/build” ]

sys.path += [ “d:/Temp/MNist/host/build/Release” ]

import mnist

import requests

r = requests.get(“http://www.lovettsoftware.com/images/eight.jpg”)

nparr = np.fromstring(r.content, np.uint8)

image = cv2.imdecode(nparr, cv2.IMREAD_COLOR)


gray = cv2.cvtColor(image,cv2.COLOR_RGB2GRAY)

output = mnist.predict(gray.ravel())

result = np.array(output)



which prints the correct output, namely “8”.

If you want to get adventurous you can run the ELL model over all the tests images in the MNist test dataset using this python code:

import sys

import cv2

import numpy as np




import mnist

input_shape = mnist.get_default_input_shape()

input_size = input_shape.rows * input_shape.columns
* input_shape.channels

output_shape = MNist.get_default_output_shape()

output_size = output_shape.rows * output_shape.columns
* output_shape.channels

nogui = False

failed = 0

total = 0

args = sys.argv[1:]

if len(args) == 1 and args[0] == nogui:

    nogui = True

def getLabel(labels):

    a = labels.split(‘ ‘)

    if a[0] == ‘labels’:      

        for i in range(10):

            if a[i+1] == “1”:

                return i

    return 0


def getImage(features):

    a = features.split(‘ ‘)

    data = []

    if a[0] == ‘features’:       

        for i in range(len(a) – 1):


    return np.array(data).reshape((28,28))


def Test(index, line):

    global total, failed, output_buffer

    parts = line.split(‘|’)

    passed = True

    if len(parts) == 3:

        labels = parts[1]

        features = parts[2]

        answer = getLabel(labels)

        image = getImage(features)

        if not nogui:

            cv2.imshow(“test”, image)

        output = mnist.predict(image.ravel())

        prediction = np.argmax(output)

        result = “passed”

        if prediction != answer:

            result = “failed”

            failed += 1

            passed= False

        total += 1

        print(“%d: predicted %d, answer is %d, test %s”

             % (index, prediction, answer, result))

    return passed


def WaitForKey():

    key = cv2.waitKey(1) & 0xFF

    while key == 255:

        key = cv2.waitKey(1) & 0xFF

    return key


def RunTest(filename):

    global total, failed

    index = 1

    with open(filename) as f:

        for line in f.readlines():

            line = line.strip()

            passed = Test(index, line)

            index += 1

            if not nogui and not passed and WaitForKey() == 27:




    print(“Total tests %d, failed = %d, pass rate = %f” %

        (total, failed, (total – failed) / total))




Then the output from this is:

  • Total tests 10000, failed = 259, pass rate = 0.974100

Lots ‘o fun!




Visual Studio’s lesser-known Xml Schema features

John Tasler made me write this blog.  There I was quietly responding to various emails when John said to me, he said “hey, that would make a great blog”.  So don’t blame me, it was his idea 🙂

What was the question you ask?  Well Ming wanted to know how do you make Visual Studio automatically apply an xsd schema to a newly created XML file ?

Answer: You have lots of options.

 If your schema lives in %DevEnvDir%\..\..\Xml\Schemas, or anywhere in your currently loaded “Solution”, then all you need to do is reference the target namespace of your schema like this:


But if your schema needs to live some place else (like in the cloud) then you can reference it using xsi namespace trick like this:

But for this to work you have to enable downloads in the XML editor tools options miscellaneous tab (it is off by default):
And you should be sure the owner of the site that publishes this schema is happy with you doing that.  Especially if you are going to encourage lots of users to do the same.Lastly if you have a totally ad-hoc XML and schema location elsewhere you can set one or more schemas to use on a per-document basis using the document Properties windows like this:
If you are building a project item wizard then, you could automate this step if you want using VS automation api.
SchemaCatalogs:  But what a lot of people never think to ask is what if I have multiple schemas of different versions that use the same target namespace?
Well for functional completion on this topic, this is where the  SchemaCatalog file comes in:

<?xml version=1.0 encoding=utf-8?>
<SchemaCatalog xmlns=http://schemas.microsoft.com/xsd/catalog>


See %DevEnvDir%\..\..\Xml\Schemas\catalog.xml for an example.  It can “associate” schemas from other VS directories using %InstallRoot%, and it can associate by file type with conditional expressions.

AirSim is “live” !

So if you love flying quadrocopters, as I do, then you’ll probably also love flying them in a simulator.  I recently worked on a cool new Simulator with a colleague in Microsoft Research and it was a blast.  The Unreal Game engine is astounding, I love what it can do with rendering reflections, grass blowing in the wind, shadows created by leaves on the trees that also move as the wind blows.  Amazing stuff.

We built the AirSim simulator because we needed a research platform where we could safely test out reinforcement learning algorithms for vision based control of the drone.  Well, obviously we don’t want to have to crash a real drone while it is “learning”, but the problem with existing simulators is they didn’t look real enough.  So we built a new drone simulator using Unreal engine and it works great!

MyMoney.Net is now open source…

A good friend of mine, JP Duplessis and I have been tinkering away on an app for the last 10 years or more, it started out as a replacement for the then cancelled Microsoft Money app.  We built it entirely in C#, it started out as WinForms, we ported it to WPF and XAML and then fixed bugs.

We wanted to open source it but I finally got around to doing it today on GitHub.  See https://github.com/clovett/MyMoney.net/wiki.


The app is really designed for programmers who want to tinker and add stuff.  JP added a whole module to help deal with rental properties.  The data is stored in sqllite, so it is super easy to extend and add whatever you want.

The surprising thing is how small financial data really is.  For a family of 5 with data all the way back to 1997 (which I exported in QFX format from Microsoft Money and my wife’s data from Quicken) the entire database is just over 6 megabytes.   In today’s hard drive world, this is miniscule.

So MyMoney.Net loads up the entire database into memory, which means it is super snappy to pivot, slice and dice and jump around your accounts.  You might wonder how long it takes to load, ooh, about 1.2 seconds 🙂

So anyway, I hope a few other people can enjoy the app, and contribute on GitHub.  I look forward to many interesting pull requests.




I love Mylio

A good friend of mine named JP Duplessis joined a startup named Mylio and they built some really amazing photo management software.   I’ve been using it for over a year now and I have to say, I really like it.

I was tired of the old Windows Photo Apps that are so, well, um, boring and feature limited and I was tired of my wife losing her MacBook and all her photos on it.  

You should have seen the moment when we set it up in my kitchen and I could finally sync photos off her MacBook onto my Windows PC for the first time.  I thought I died and went to heaven !!

I then added a bunch of photos from my son who is an avid photographer.  He went on a hike in California (which is where he lives) and I would have never seen this photo, not in a million years of begging and pleading for photo backups.  He works really hard in a startup and doesn’t have time for facebook.  But now, like magic using the Mylio Cloud Sync feature, it pops up on my machine and I can enjoy a whole series of amazing shots of this Tiger Swallowtail butterfly. 

So I’m now using Mylio to sync over 60,000 photos across 9 machines (I have a PC, laptop, a PC at work, and a backup NAS drive, my wife has MacBook, iPad, and iPhone, and my son has Android phone and a MacBook running Windows).  I hear some users have over 1 million photos and even more machines, and that makes me feel good about the scalability of this app.  

Nothing else came even close to “managing” this many photos, and Mylio is incredibly snappy.  I can slice and dice and pivot through folders, albums, people, ratings, all to my heart’s content and I’m never waiting for a photo to load or for thumbnails to be generated.  This snappiness is probably my favorite feature of the app.  Who has time to wait for photos to load, like on all those “cloud” based photo sites?  I don’t. 

I’ve also used the built in photo editing features, and I love that all edits are non-destructive (they are stored in a separate *.xmp file).  So any user can go back and reset the edits and try again to get a better result and as the app improves, even existing edits can look better after a Mylio update.

Of course, most of my photos are huge RAW files from a fancy Canon or Nikon camera and Mylio handles all those natively, no stupid OEM drivers necessary, which is fantastic.  They must have done a ton of work to pull that off and I can “share” these as smaller png files on Facebook with a single button click.

So Mylio does everything I need with photos, in a fast, smooth and well-designed user interface.


WinRT Extensions using RuntimeBroker

Have you ever wanted to do something naughty from your lovely well behaved sandboxed WinRT application?  Do you need to call some Win32 API that you don’t have permission to do from WinRT?

Well WinRT itself does this using a separate service called the RuntimeBroker.  Many WinRT objects use RPC to call into this broker process and do those things that need elevated priviledges.

I built a sample application that shows how to do this.  Essentially it goes something like this:

  1. Create a static C++ library containing a MIDL interface and compile it using /winrt to make it a windows runtime (WinRT) interface. 
  2. This generates RPC proxy stuff needed to marshal the interface across processes.  Compile this into a new C++ DLL called “YourInterface.ProxyStub.dll”.  You will register this dll using regsvr32 and this is needed for RPC to work.  This proxy stub and the implementing broker dll will need to live in a location your WinRT app can access, like c:\windows\system32.
  3. Next create a Universal C++ DLL called “YourInterfaceBroker.dll” – this will contain the implementation of the interface and is designed to run in the RuntimeBroker process.  This DLL will be able to call out to Win32 and do interesting things.
  4. Register the broker using a custom registry script.
  5. Lastly create a C# Blank Universal WinRT application to test out your new interface, referencing the *.winmd that was generated by step 1.  When you instantiate your new runtime class it is loaded into the RuntimeBroker process, and your C# app gets back the client proxy to it so each call to that object is marshalled automatically to the RuntimeBroker. 

 See the sample code: WinRTServiceSample.zip.

MIDL Interface Library

In my sample I started with this static C++ library, I added a MIDL file called Microsoft.Service.Sample.idl, this defines the name of the *.winmd so its namespace has to match the filename.  I then used the MIDL Tab in the C++ properties to turn on “Enable Windows Runtime” /winrt compile option.  The *.winmd file is placed in the project output directory. I built it once and this also generates a dlldata.c and *_i.c and *_p.c files.  I added those files to the project.  To compile them it is easier if you just turn off the C++ option for precompiled headers.  I also added the C++ preprocessor defines: WIN32;_WINDOWS;.  So now you have a static library containing the RPC stubs needed later.  For the purposes of this sample app I defined the following simple WinRT interface and runtime class:

#ifndef NTDDI_WIN8
#define NTDDI_WIN8                          0x06020000
namespace Microsoft
    namespace Service
        namespace Sample
            interface IMyService : IInspectable
                HRESULT DoSomethingFun();
            runtimeclass MyService
                [default] interface IMyService;



I then added a C++ Win32 DLL project called MyService.ProxyStub.  This dll has to include the above files again, but this time with the C++ preprocessor definition “REGISTER_PROXY_DLL”, and you need to generate a new GUID for your runtime class id (different from the one in the MIDL file) in a new ProxyEntry.c file as follows:

// {62C41977-4CCC-4D4E-9C8C-A842DB7182E6}

#definePROXY_CLSID_IS { 0x62c41977, 0x4ccc, 0x4d4e,{ 0x9c, 0x8c, 0xa8, 0x42, 0xdb, 0x71, 0x82, 0xe6 } };












This is also easier to compile if you turn off precompiled headers.  You also need to add a Modile Definition file for the DLL containing:

    DllGetClassObject           PRIVATE
    DllCanUnloadNow             PRIVATE
    DllRegisterServer           PRIVATE
    DllUnregisterServer         PRIVATE


You’ll also need to add a VC++ Include Directory pointing at your interface static lib directory where the dlldata.c and *_i.c and *_p.c files got generated.  You will also need to add the linker input “rpcrt4.lib”.  This generates the proxy stub needed for RPC to work.  This dll will be registered using regsvr32.


Next I added a Universal C++ DLL project called “MyServiceBroker”.  I added a reference to the static library project, I included #include<ObjIdl.h>in the pch.h file and again added the VC++ include directory where the *_h.h file was generated.  I wanted this DLL to work on the IoT SKU of Windows 10, which is why I used Universal DLL, but I want to run this outside an app container so I removed the following from the *.vcxproj file: <AppContainerApplication>true</AppContainerApplication>.  I also added the following linker instructions to each <Link> section:
    <IgnoreSpecificDefaultLibraries>comsuppwd.lib; comsuppw.lib; kernel32.lib; ole32.lib; user32.lib</IgnoreSpecificDefaultLibraries>
AdditionalDependencies>Wlanapi.lib;mincore.lib;onecoreuap.lib; WindowsApp.lib;</AdditionalDependencies>

Lastly I added the a class that implements my new interface as follows:


namespace Microsoft
namespace Service
namespace Sample
classCMyService : public Microsoft::WRL::RuntimeClass<ABI::Microsoft::Service::Sample::IMyService, Microsoft::WRL::FtmBase>
InspectableClass(RuntimeClass_Microsoft_Service_Sample_MyService, PartialTrust);



constCMyService& operator = (constCMyService&);


Then I added an empty implementation the the .cpp file.  I also replaced the contents of the dllmain.cpp file with the following:


using namespace Microsoft::WRL;

STDAPI DllGetActivationFactory(_In_HSTRINGactivatableClassId, _COM_Outptr_IActivationFactory **factory)
return Module<ModuleType::InProcDisableCaching>::GetModule().GetActivationFactory(activatableClassId, factory);

DllGetClassObject(_In_REFCLSIDrclsid, _In_REFIIDriid, _Outptr_LPVOIDFAR* ppv)
return Module<ModuleType::InProcDisableCaching>::GetModule().GetClassObject(rclsid, riid, ppv);

STDAPI DllCanUnloadNow()
if (Module<ModuleType::InProcDisableCaching>::GetModule().GetObjectCount() == 0)
        hr =
return hr;

STDAPI_(void) DllAddRef()

STDAPI_(void) DllRelease()

STDAPI_(BOOL) DllMain(_In_opt_HINSTANCEhinst, DWORDreason, _In_opt_void*)
if (reason == DLL_PROCESS_ATTACH)
return TRUE;


First you need to copy your *ProxyStub.dll and your *Broker.dll to a place that the WinRT app can access (like c:\windows\system32) otherwise you will get E_ACCESSDENIED when trying to call object from WinRT.

To register the broker I created a custom registry script containing the following.  Notice the GUID used here matches the one I generated for the ProxyEntry.c file earlier.

Windows Registry Editor Version 5.00



Running this from administrator command prompt makes it possible for WinRT to construct your runtime class.  You may need to take ownership of the WindowsRuntime part of the registry in order to do this.  There is a tool floating around called TakeRegistryAdminOwnership.exe that can help with this.

You may also need to install the VC++ debug C++ runtime bits (vcruntime140d.dll and ucrtbased.dll) if you have not already installed that on your test machine.  Use depends.exe to find out what else the proxy and broker are dependent on if they still refuse to load.

C# App

To test out my new brokered service I created a blank C# windows application and added a reference to the generated Microsoft.Service.Sample.winmd file, then added the following lines:

            var service = new Microsoft.Service.Sample.MyService();

When you run these lines the MyServiceBroker.dll is loaded into the RuntimeBroker.exe process (you can attach to that process using your VS debugger and watch it happen and you can debug your broker that way also).  You will see RPC stuff on the stack when the DoSomethingFun call comes in.  Don’t worry about terminating the RuntimeBroker.exe process, it will auto-restart.


If you need to have your broker do something async and call back to your C# application later you can easily define a new interface, implement that in C# and pass it to your broker as an argument, then the broker can call that interface.  This RPC interface will “timeout” after a minute, so you may need to repeat that process periodically or do a heartbeat kind of thing to keep it alive.  You could build a WinRT component project that wraps all this and exposes a new “Client” interface that turns all this callback stuff into C++/CX events so that it is easier to consume in your C# applications.  Your “Client” interface could also expose C++/CX properties and events so you have a nice modern interface to your new broker. 



Magic LED Lights Rock

I had some fun over the Christmas holiday playing with my new toy, Magic LED Lights.  These are controlled from your phone using Bluetooth Low Energy.  But as is often the case there was no Windows Phone app for these, so hey, it was the holidays, I had some time, so I built one.  Actually, I built a Windows Phone app and a Windows 8 app.  It is implemented in C# and most of the code is shared between the two apps.

See Demo Video of bluetooth lights and Wifi Demo Video for wifi lights.  So far the light bulb has worked perfectly!   You can get more information from the manufacturer at www.ledmagical.com.  

Of course, this is just the beginning.  Imagine the lights turn on when I walk into my office.  Or the XBox controls the lights so they turn red when I get shot in Halo. There are many fun applications of this really simple but very cool new tool.

Update to XML Notepad

As  XML has exploded across the planet I continue to get many requests for improvements to my XML Notepad of 2007.  The tool has been downloaded over a million times, so I figured it’s time to show it some love and fix some bugs.  In the process I updated it to NET 4.0 using VS 2013.  I was amazed how everything from back then still works, including the Unit Tests.  That is pretty amazing platform compatibility.  Windows is a great platform and Visual Studio is still best of breed for software development, and I still love C#.

I also posted the source code on github.com/microsoft.xmlnotepad

See Download

Seismograph App

I built a simple app called Seismograph to visualize the accelerometer data on my phone.  The reason I built this is because I wanted a simple way to check the vibration of the motors on my quadrocopter.  I saw some really nice apps for android, but all the windows phone accelerometer apps were crap.  So I built my own 🙂 

I then ported this Seismograph for Windows RT so I could also run it on my Surface device.  When I did that port I discovered that the same approach to graphing the data didn’t work well on a bigger screen, it slowed down way too much.  So I had to re-implement the scrolling graph control.  Previously I was building a new PathFigure with the new points scaled to fit the screen, and so each frame was a new Path object and that’s how it would scroll across the screen.  Turns out the performance is a lot better if I reuse the PathFigure and only add to the Path object and never remove any points, and rarely re-scale the points.  But in order to scroll then I needed to adjust the canvas position of the path to the left, well turns out I can do the scrolling at a higher rate than the point adding which results in a very smooth scroll which looks great.  The problem is I would run out of memory if I never removed points, so what I do is cycle two Path objects, when one is completely offscreen on the left, I recycle it and use it on the right, and this is how I get really nice performance on the Surface.  So I back ported that to the phone and it looks great there too.

Privacy Policy

The app records the accelerometer sensor data so that you can choose to save it locally on your device either in a bitmap image or in an XML file.  The app does not send your data anywhere else, so the data is not shared with anyone.  You can choose to send the bitmap image or the XML file to someone else, but that is up to you.  The app does not even know if you have shared your data with anyone.