Event Planning using BING API

by clovett26. May 2013 19:10

I received an email from a friend asking for help figuring out a fun math problem.  He works for FIRST and the problem is figuring out how to match up over 400 teams in Washington State with 5 separate event locations, minimizing the driving distance for each of those teams.  Each event was happening on the same day, and each team gets to compete at only one of these events.

So I quickly cranked out a nice little Silverlight Application to do the job.  And I hooked up to the BING Mapping API to calculate the actual driving distances from each team to each event location.  Silverlight and BING worked beautifully together.  So far so good, but now for the hard part.  How do you calculate the optimal assignment of teams to events that results in minimal total driving distance?  I hacked out a brute force solution that first assigned every team to their closest venue, but then there were additional constraints that got more interesting.  Each event has a minimum and maximum capacity, so if an event was over populated, some teams had to move to the second or third closest venue.  This wasn’t too hard to code, but I quickly realized, the solution I was getting was not necessarily the best.  For example, in the diagram below, let’s say the maximum capacity for each event is 3 teams, which means team “i” needs drive all the way to “C”.  Clearly this is silly. 

 

 

It is better to move “c” to “B” and “f” to “C”, making room for “i” to join “A”, like this:

But this is a tricky “ripple” effect that is hard to find programmatically without having to essentially search all combinations of teams to events, but then I quickly realized it would take about 100 years to execute that kind of exhaustive search on my machine because 400 factorial is a large number.

So this is when I contacted Microsoft Research.  Lev Nachmanson works there and is a good friend of mine and a powerful magician when it comes to mathematics.  So I presented the problem to him, and he said he’d think about it.

In a short while I got an email back saying he had a solution.  He explained it to me over the phone, I implemented it, and it worked, finding the solution in less than a second.  I was very impressed, but I also a bit skeptical.  I couldn’t believe it could be so simple.  I went to his office to get a better understanding of how it worked.

This is how he explained it to me. The global task is to drive to the minimum the sum of the distances from teams to the events they are assigned to. To initialize a solution assign each team to their closest event, but then think of the remaining optimization problem as a type of traffic flow problem.  We will be shifting the teams between the events, so we need to consider a graph for doing this. The edges of this graph are AB, AC, and BC. (It is called a complete graph.) For example, above we moved “c” along edge AB, "f" along BC and "i" along AC. To find a solution we need to consider all paths on the complete graph that don’t repeat an edge. A path can be a cycle, like ABCA, but every edge in it should appear at most once.

To run one iteration, take a path as described above, then take the first edge of the path and shift a team along it such that shifting it along this edge “makes the least damage”. Here the total sum can even grow. Take the second edge and do the same, and do it for every edge of the path. When we are done with the path, look at the total sum. If we are lucky and it went down, we accept the move, otherwise, we discard it. When no path gives us a gain we are done!

Then how many possible paths do we have? Well in this case with 3 events there is AB, AC, BA, BC, CA, CB, ABC, ACB, BAC, BCA, CBA, CAB  plus all the cyclic versions of these: ABA, ACA, BAB, BCB, CAC, CBC, ABCA, ACBA, BACB, BCAB, CBAC, CABC, a total of 24. It is not a huge amount of paths. The last thing to do for running the algorithm fast is to know how efficiently to find a team which can be shifted along an edge with the minimal damage. The best data structure for this is a priority queue.

The end result was that on my desktop machine I could use this algorithm to find the optimal solution in sub-second times, which is fantastic.  Super thanks to Lev!   I just love it when a nice elegant algorithm makes software work really well.

Tags:

C# | First | Research | WPF

Epic Crash

by clovett17. March 2013 17:53

The GoPro Hero 3 camera and the Turnigy Heavy Aerial Lift quadrocopter frame are both amazing.  The other day my quadrocopter decided to take off by itself because the barometric sensor stopped working for some unknown reason and so it took off straight up into the clouds at full throttle for 5 minutes until the battery died.   My GoPro camera was on board and recorded the entire thing, including the free fall from about 5000 feet up!

I guess it was my lucky day.  When I saw the quad disappear into the clouds I thought to myself, "oh crap, there goes $600 down the drain, I’ll never recover it".  But thanks to the very clear and still weather, and the fact that it kept its GPS lock all the way up, it didn’t wander too far from home.  It ended up about 500 meters from my house.  I could not see it coming down, but I could hear it because the motors spun up real fast just before landing so I could hear it and knew it was in the neighborhood.  It's as if the poor quadrocopter was screaming out for help before crashing into the trees.  Another neighbor also heard it which confirmed its general location.  I have some great neighbors because they came out and helped with the search-and-rescue through the woods in the vacant lot next to our neighborhood.  We did a grid search, and found it in about 45 minutes.

The camera survived, and not only that, it fell to the ground so we could find it, then the video helped me locate which tree the quad was caught in.  The even more amazing thing is the quad survived almost perfectly in-tact.  The only damage was one broken propeller and one of the thin carbon fiber rods on the landing gear snapped.  Simply amazing.  I’ve gotten more damage from a hard landing from 5 feet above my lawn!  This is the quadrocopter immediately after the crash:

Tags:

Quadrocopters rule!

by clovett22. January 2013 01:36

I've been playing with Quadrocopters for over a year now and they are so much fun. I love the Arduino based APM board from DIY Drones so you can get in there and hack around with the firmware.

I built the quad you see below using a carbon fiber frame I found on HobbyKing, and control it using the APM 2.5 board which is packed with all sorts of goodies including gyro, accelerometer, barometetric & compass sensors, as well as pins to plug in my external Sonar and GPS sensors.  I then use a nice C# app called the APM Mission Planner to create GPS missions for it so it flies autonomously. Not to mention the additional 3DR Radio for gathering real-time telemetry while it's flying so you can plot out what it's doing and debug it.

Combine that with a GoPro Hero3 and you get a very cool flying robot.  See the video of my neighborhood that it made all by itself on a lovely clear day.

Multi-rotor drones are taking off all over the world and people are finding some pretty amazing uses for them.

To find out how to program a mission like this see my little tutorial video below:

Tags:

C# | Quadrocopters

My First Windows 8 App

by clovett2. January 2013 22:51

I just published my first windows 8 app, called Fun Chores.  This is based on my Windows Phone app of the same name and my ASP.NET Web Site www.funchores.net.  I have to say, the Win8 app is more of a re-write than a port.  The main reasons for the re-write are:

  1. The C# Async pattern is very cool, but also very disruptive.
  2. The greater screen area required a lot of thought.
  3. A lot of stuff is missing in Win 8 C# XAML framework (like DatePicker!)
  4. Harder to find stuff on MSDN
  5. Unexpected bugs in win 8 itself that I had to work around.
  6. More devices to test on, with more diversity in performance than expected.
  7. Publishing is more involved.
  8. Feature creep...

So I found that it took a lot longer to build than the windows 7 phone app.  I was familiar with Silverlight and the Windows Phone app felt like a natural extension of that.  Windows 8 has a steeper learning curve and a more involved publishing process with lots of artwork and so on.

1. Async

Async in C# is a fantastic new feature of the language, and I’m now addicted to it and am using it heavily.  The downside is it makes it harder to port old code that doesn’t use that pattern.  It also tends to “bleed” like a C++ “const” – so once you make one method async, the caller has to be async, and it’s caller and so on until you reach an entry point.  You can stop that chain, but you get a green squiggly warning that you shouldn’t, so it’s always tempting you to keep on going ?  Which I did of course.  In my app I use async the way it was supposed to be used, so I switched from WebRequest to HttpClient which supports it, and all the way up through my web service wrappers, and my Model up to my MainWindow is now async, which is awesome for responsiveness, but very disruptive to code.  It also forced me to think more about my UI states, and what needs to be disabled during async operations so the user can’t get themselves in a weird state doing multiple async things at once and so on.  So careful thought is required here, and that takes time, it does for me anyway ?

2. Screen Size

The increase in screen size from the phone to the Win 8 app was also disruptive to my code.  But I could borrow more from my ASP.NET version which runs in the web browser.  But web pages tend to have a vertical document oriented model, whereas win 8 has chosen more horizontal scrolling in things like the GridView and so on.  

3. Missing Stuff

It was annoying that this was missing from win8 .net framework all together.  I spent a day or two cooking up my own Date Spinner control which I published to the code gallery.

I also found out that the VariableSizedWrapGrid is not so variable sized. It basically takes the size of your first time and sticks with that.  So I had to implement a recommendation I found on the forums which was to put my own WrapPanel inside the GridView and this handles my variable sized chore icons more gracefully.

To be fair, there was stuff missing in Windows Phone also, but there was a Windows Phone Toolkit that I could grab which had everything I wanted.  The problem is that the Windows 8 Toolkit is not available yet…

4. MSDN

I found it hard to find information about developing win 8 apps in C#.  My searches kept returning information about Jscript/HTML5 or C++ and in general I found the documentation rather sparse.  For example CreateFileAsync.  Does it throw an exception if the file is missing?  Turns out it does.  So is there a way to see if a file exists before-hand?  Turns out there’s a bunch of forum threads on this, but no updates to the documentation yet.

5. Bugs

I found two bad bugs.  First of all the Stream wrappers for IRandomAccessStream didn’t work as advertised when I wanted to convert base64 encoded images to bitmaps and back again for storing on at web service.  I spent days trying to find a work around.  See my forum thread on the topic – it could save you some time!

The second bug I found is that if you close a flyout for a settings page and navigate to a different main page in response to that then you hit a crash in Windows.UI.Xaml.dll.  So I spend another bunch of time creating a repro and filing a bug and looking for a work around.   Turns out if I wait until the animation of the closing flyout is finished before I navigate to the new page then the crash is averted.

6. Testing

Windows 8 runs a lot more devices than windows phone apps.  And when I thought I was done I tried it on my new Surface device.  I found that some of my controls performed very badly on this device and I had to code up some optimizations to get them to run better.  They are still not as good as I would like, but they are better.  So there’s not much head room for .net developers on those devices.  This could mean that the .NET/XAML stack is slower than it should be.

7. Publishing

Publishing my app on the win 8 app store turned out to be a bit of a nightmare because I hit the infamous live.com problem which happens if you don’t have a “trusted PC” associated with your account.  Turns out when you re-install windows (like upgrade from win7 to win8) live.com loses its previous trust, and if you have no other registered ways for live.com to contact you then you are screwed.  You either have to create a new live id, or wait 30 days for your id to be reset.  

Once I got past this and published my app there was nice progress feedback as my app progressed through the various stages, and when my first submission failed it is easy to figure out how to resubmit my fix.  The succeeding submission only took a few days to get approved and from there it was pretty much immediately available on the store.  Some of the test feedback was a bit cryptic but I was able to figure out what they were talking about.

8. Feature Creep

I decided to add full "admin" capability since I had the screen real estate to do it, and of course this took more time.  I guess it is a good sign that I still had the energy to do this after all of the above.  I guess it proves that once you get up the learning curve win 8 app development can be fun and rather addictive.

I that found the win8 tools in VS 2012 were very good.  For example, being able to run the remote debugger on my Surface device was fantastic.  I also used the windows 8 simulator so I could test out my handling of landscape versus portrait and so on.  F5 on local machine was great. I highly recommend multiple monitors so you can run the metro app on one screen while debugging on the other.

So in general the tooling was as good or better than windows 7 phone.  My only complaint is that *.resw files do not provide accompanying C# wrappers generated for me like you get with *.resx.  

Conclusion

Windows 7 Phone app, 865 lines of code

Windows 8 app, 2,010 lines of code

ASP.NET Web Site, 2,752 lines of code (including all the back end database stuff)

To be fair the windows 8 project template generated a lot of code for me, but I still had to read and understand and maintain all that code.   The win8 app also has more features.

It took me 2 days to build the ASP.NET web site, 3 to build the windows 7 phone app and about 3 weeks to build the windows 8 app.  I know that’s probably not what you want to hear, but if you are planning an app this might be useful to know.  The end result is worth it.  The win 8 app is definitely the best client for my fun chores service.

Tags:

C# | dev11 | Visual Studio | Windows 8 Apps

Model Based UI Testing using DGML

by clovett2. December 2012 15:33

Video: (WMV 26mb) (MP4 24mb).

Transcript: In this video  I'm going to explain how to use the DgmlTestMonitor  that I published to the Visual Studio Gallery. 

I've got a test project loaded here with some test methods and I have a DGML document.  The Directed Graph Markup Language is supported by Visual Studio 2012 and it allows me to edit these diagrams.  I created this diagram to model the user interface of an application that I'm testing.  Not only that, I have a way of executing this model and even setting breakpoints using the DgmlTestMonitor which you will find under View/Other Windows.

Now I can come in here and set a breakpoint on this node. So whenever it gets into editing transactions and editing categories  the model will stop executing.  Alright, so let’s run the test.  Notice I don’t actually have to debug the test to get these breakpoints.

1:00: This is the application that I’m testing, it’s a simple financial management tool that allows me to create bank accounts and transactions and so forth, and right there it has edited a category and so the model has paused.  So let’s go back to the application and we see it sitting on that node and if we look at the DgmlTestMonitor output we can see everything that it has done up until that point. I can even select through here and watch what it did up until that point. 

1:30: If you put Visual Studio on another monitor you will be able to watch the model executing while it’s actually testing the application.  Now that’s it’s reached this point I’m going to move the application to another monitor.  Now we can watch the model executing.  Maybe I want to stop it when it adds a transfer.  I’ll clear the previous breakpoint, tell it to resume, and off it goes. So the UI test is continuing, it’s doing various different steps.  Since I can watch the model executing I can get a feel for whether this is accurately modeling and end-user or not and I can edit the model, I can add links, I can add nodes and I can make the test do different things. 

2:15: Now in this case we hit a test bug because if we switch to the Test Explorer we see the test terminated when it was trying to set the date on a transaction.  Now it turns out this is a product bug, sometimes setting the date is a little flakey.  This is the kind of thing that model based testing is really good at, finding edge conditions that are really hard to find any other way, which is why I’m a big fan of model based testing.  I’m not saying this is going to replace all static testing, but I think this is a very interesting thing to add to your suite of test tools. 

2:45: Alright, a lot of magic to explain here.  How does this actually work?  Let’s take a look at the test code.  The way you do this is with in normal MSTest [TestMethod] you create a new DgmlTestModel , you give it the filename of the DGML document that you want to load and then you tell the model to run until you’re happy.  I’ve provided a Predicate that says run until 500 states have been executed.

All right, now you pass in an object to the DgmlTestModel constructor which is the target object that implements the states – so the test model is connected to my code.  If I double click this node it will take me to that method and this is the implementation of editing a category.  Notice that there’s lots of random number generation going on, some keystrokes to tab through various fields until it finds the category and so on.

3:43: This framework doesn’t change how you build test wrappers, you could use the CodedUI, I created these wrappers so I could automate, using System.Automation, the user interface of my application.   What’s new here is the way you can actually visually define the states that you want to travel through and the links that connect the states.  So when I edit a category I can go over here or I can go down there and the test model execution decides randomly which of these paths to take that have multiple choices. 

4:15: When you create a group you can encapsulate the complexity of editing transactions and you can define an entrypoint, so a node that has the EntryPoint Category which you can define using the property window, you can add the entry points.  You can also have entry points that are mutually exclusive, so if you choose this one, don’t choose that one and so that has a category on it saying it is a singleton.  You can define predicates on a link, so you can add link labels “IsSecuritySelected” we see that’s just a implemented as a Boolean  property.  You can implement all kinds of state machinery in your Boolean properties which will then guide the test in the right way.  For example, IsAccountSelected, or IsEditable and so forth.  So all of these things can encode various state about what your application is doing to make sure your test model doesn’t get stuck in a weird state and try to do something that’s invalid. 

5:16: Now since it is random, you’ll want to also be able to reproduce a failure and if you scroll to the bottom of the test output and you will see the “Model Seed = 3122235”.  This number is the seed to the random number generator.  If I plug this number in I will get the exactly the same test sequence that I got before.  So basically you can create a static test by just copying down these seeds, which is a pretty low maintenance way of creating static regression tests.

5:43: Ok, let’s edit this model and see if we can make some changes here.  So when we edit the payee we can go to SalesTax, or Deposit or Payment.  Well maybe I always want to have salestax, so I just delete those links and maybe I want to have the memo field be optional, so instead of always going from EditCategory to EditMemo, I’ll give it one more route from there to SelectTransaction and now I will only have EditMemo sometimes.  That’s it, that’s all you have to do, save the document and re-run the test.  Ok, we see it now adding a new transaction, that time it added a memo, this time it didn’t, this time it did, and now it added a transfer and it stopped, so let’s see what happened.

6:43: Ah, that’s because we have a breakpoint on AddTransfer, remember?  So I can go to the DgmlTestMonitor and we can see that it hit a breakpoint on that node and I can tell it to resume running the test.  You can also pause the test at any time, using the pause button, which is handy for a UI test because often interrupting a UI test is kind of tricky but when you tell the model to pause it’s usually in a pretty good place. 

7:18: Alright, so, one more thing and that is you can modify the thickness of these links.  If you go to the Property Window this link has a priority of 10 which means it’s going to pick that link more often than this link with Priority 1.  This way you can weight the model so that it’s doing various things in a way that’s similar to the way a real user would use your product. 

7:40: Lastly, when the model enters a group, like View Categories, notice there’s no exit links from SelectCategory.  Just to make it simple to create these models what happens when it hits a leaf node that has no exit, it will jump back up to the parent group and it will use any exit links that the parent has, and if the parent doesn’t have any exit links then it will go back to the root and look for an entry point there, and re-entry the model at the root and keep executing from there.  So that makes it easy, you don’t have to specify all possible exit routes and it makes your model a little bit simpler. 

8:18: That’s it!  I hope you like it.  I’ve posted a blog on my website, and in there you’ll see a readme that points to the source code that you can download so you can play with it, including sample code on how to wire it up and also documentation on this class, DgmlTestModel.  You will find DgmlTestModel and also the NamedPipeReader that listens to the model while it’s executing in case you want to build your own user interface. 

Thanks for watching!

 

 

 

Tags:

C# | dev11 | DGML | Visual Studio

Software Trails

by clovett7. September 2012 16:11

Video: (WMV 33mb)

 

Download Tool: SoftwareTrails.application

Hi,

Have you ever wanted to see what is going on in your program in real time?  I built a light weight profiling tool that shows you exactly that. Let me show you how it works.  To get started you click the Launch button and find the application you want to profile and immediately we see a bunch of activity showing what’s happening as the app launches.  This dashboard is showing a roll up across the top level Namespaces and the counters are showing the total number of function calls in each area.

You can scroll down and see everything that is going on.  We can see that there are some Microsoft namespaces, but there’s also a Walkabout namespace which happens to be the code for this application.  So I can take a look at startup again without the .NET frameworks by drilling down into the Walkabout namespace – this shows me the drill down path and the green blocks represent classes and namespaces and the white blocks represent method calls. 

So we can now see that we’re getting down to the method call level – maybe we’re interested in seeing what happens during LoadConfig and we can click that method and it will search the call information and build a conglomerate call stack graph basically showing a combination of all call stacks that it has seen involving that method Loadconfig.  And from this you can actually explore down into WPF or you can explore upwards and see what those function calls are doing.  You can also try and expand the entire graph but in this case it’s not recommended because the graph is huge.  When you’re done exploring the call stack you can close that and go back to the dashboard.

You can also Filter this information – so maybe I care about anything involving the word “Transaction” and I can see that during the launching of the application that there’s some Transactions Namespaces and I can also see that it shows up over here in the View namespace.  Let’s drill down into some controls and we see a class named TransactionView. 

At any point you can clear the buffer and start over and go back to <Home> and remember to clear the filter.  When I do that I can see that there’s actually a heart-beat going on with this application and the reason for that if I drill into the MainWindow is that there’s an OnTick method that is being fired by a DispatcherTimer. 

Ok, let’s go back over here and let me show you more about this application that I’m profiling.  Let me just disconnect for a second (you can disconnect at any time if you want to just pause the profiling).  I’m going to load some sample data into this application.  It’s basically a personal finance application like Quicken and I’m going to show you the real scenario that I’d like to debug, so now I have some sample data, the scenario that I want to debug is drag/drop – I want to find out why drag drop isn’t working in this window here.  Alright, so I can now reconnect back to that application, the profiler is ready for work, probably want to go back home, and let’s run the drag/drop scenario.  We see a lot of activity, 16 million function calls already, even after clearing the buffer, I see some stuff in the Walkabout namespace, and the System namespace, so let me apply a filter again and just search for anything containing the word “Drag”.

Now I could look at the Walkabout code, but first I want to take a look at the .NET frameworks.  So in the history of calls that we have in the buffer, this is everything that WPF does to implement Drag/Drop.  This gives me a really great place to start looking up documentation on MSDN to find out what these methods do, OnDragOver, OnDragEnter, OnDragLeave and so on, and also I see a method called DoDragDrop which sound like an interesting place to start, let’s take a look at that method, and if I look at the caller, well it’s this class called MoneyDataGrid.  So now we know the class over here in the UI is called MoneyDataGrid.

So now if we go back to the Walkabout namespace, there I can see the Controls namespace and the Views namespace and I can see the classed called MoneyDataGrid.  This is the guy that starts the drag/drop operation – so let’s remove the filter and take a closer look at everything he’s doing.  He’s getting the mouse move which begins the drag/drop operation and here we see a flag called “get_SupportDragDrop” and we see the OnQueryContinueDrag method which the MSDN documentation tells us is used by WPF to figure out whether the drag/drop operation should continue.  So let’s select OnQueryContinueDrag and see what it calls.  Now I have an excellent place to go and look in the code  - it’s in MoneyDataGrid, to find out why when I drag this window nothing happens.

Now if I clear the buffer and do it again, now I can see in real time a lot of these mouse moves come in, I see the call to get_SupportDragDrop and so on.  So being able to see in real time what’s happening, for example, when I do mouse moves, it’s is a really useful way to see what’s going on in my code.

I can also see that when I wiggle the mouse there are lots and lots of function calls, so if I’m worried about performance at all, that counter down there, plus the rollup for each Namespace is going to be a great way to figure out where I need to optimize and see if I can remove some of those calls and get better performance. 


Profiling Very Large Applications

Ok, enough of that application.  Now what about a bigger application?  I’m going to launch Visual Studio - Visual Studio consists of a huge amount of code.  I want to see what the overhead then is of the profiler in launching Visual Studio.  Already we can see all the .NET framework stuff that’s happening, and Visual Studio is already up and running in about 10 seconds.  So we can also see here that in order to launch Visual Studio it takes about 46 million function calls, but there’s only about 15 thousand unique functions.  Again, here’s the top level view, I can see the Microsoft.VisualStudio namespace, which is where all the Visual Studio code lives so I can drill down and re-run the launch scenario and I can see everything that happens in the Microsoft.VisualStudio namespace when it launches.  I can see stuff in the Platform, in the Shell, there’s some Packages, let’s see what package is doing, and drill down, and so on.

The breadcrumb toolbar allows you to go jump back up to any level that I want.  Let’s go back to VisualStudio. 
Now the same thing can be done – let’s say I want to explore a drag/drop scenario inside Visual Studio with the “Drag” filter.  First let’s clear the 50 million function call history, that was quick, and let’s load a DGML graph so we can explore drag/drop in DGML.  First off when I load the graph I can already see a bunch of classes, “GraphDragDrop”, and “DragSourceGesture” and these are getting fired up inside Visual Studio, so that’s promising.  Those things will probably add drag/drop support.  Let’s zoom in and grab one of these nodes, and do a drag/drop.  Just as before I’m able to see in real-time everything that’s happening – the DragSourceGesture did a lot of work, let’s drill into that and take a look at that again, ok, I’ve got too much stuff in my buffer, let’s clear that and do the drag/drop again, and instantly we see exactly what’s happening deep down inside Visual Studio to support that Drag/Drop operation and now I know where to find the code and set breakpoints and continue my debugging.
So I think you’ll find that this profiler scales up to some pretty big software. 

Testing

I think this tool will also make a great testing companion.  For example, if you own a document editor then there’s a special call deep down in the VS shell called “ReleaseDocument” that needs to happen when your document window is closed.  Bang, all right, so deep inside this Visual Studio namespace,  inside WindowManagement, there should be a call to ReleaseDocument.  If that doesn’t happen you have a pretty bad memory leak in your editor.  So this tool is a great way to do an ad hoc test that certain things are happening when they are supposed to.

That’s it, I hope you enjoy it, please send your comments to my blog.

Download Tool: SoftwareTrails.application

Tags:

C# | DGML | Utilities | Visual Studio | WPF | Debugging

WPF DataGrid Scrolling Performance

by clovett15. July 2012 12:07

I was investigating some sluggish scrolling performance in one of my apps that was using WPF DataGrid.  I used the Visual Studio Profiler to find some hot spots and found that “PrepareContainerForItemOverride” was a good place to instrument where most of the work was happening.  So I overrode this method, and wrapped it in an ETW event.  ETW events are awesome, they have very low overhead and there's lots of tools out there to process the logs they create.  I have a tool that produced the following graph while scrolling my grid:

Notice there is not much gap between each green block, which means scrolling is maxed out, which is why it seems sluggish.  If we can shrink the green blocks scrolling will get smoother.   I’ve noticed considerable degradation when my rows were smaller – probably because it has to “prepare” a lot more items while scrolling which means a lot more data binding overhead. 

I made one change and the time dropped considerably, the average time around PrepareContainerForItemOverride dropped from 2.1ms to 1.4ms (about 33% improvement) and scrolling does seem smoother.  Before the fix it took 20 seconds to page scroll to the bottom of my grid with 4121 rows; after the fix it took about 13 seconds (which is also about 33% improvement as expected).

 

The fix was to replace the following expensive data binding DataGridCell Template:

 

<Border x:Name="CellBorder">

    <ContentPresenter SnapsToDevicePixels="True"/>

</Border>

<ControlTemplate.Triggers>

    <DataTrigger Binding="{Binding Path=IsReconciling}" Value="True">                               

        <Setter Property="Background" Value="{DynamicResource ReconciledRowBackgroundBrush}" TargetName="CellBorder"/>

    </DataTrigger>

    <Trigger Property="IsSelected" Value="true">

         <Setter Property="Background" Value="{DynamicResource RowSelectedBrush}" TargetName="CellBorder"/>

         <Setter Property="Foreground" Value="{DynamicResource RowSelectedTextBrush}"/>

     </Trigger>

     <DataTrigger Binding="{Binding Path=Unaccepted}" Value="True">

         <Setter Property="FontWeight" Value="Bold"/>              

     </DataTrigger>

     <DataTrigger Binding="{Binding Path=Unaccepted}" Value="False">

         <Setter Property="FontWeight" Value="Normal"/>

     </DataTrigger>

     <DataTrigger Binding="{Binding Path=IsDown}" Value="True">

         <Setter Property="Foreground" Value="Red"/>

     </DataTrigger>

     <DataTrigger Binding="{Binding Path=IsReadOnly}" Value="True">

         <Setter Property="Foreground" Value="Gray"/>

     </DataTrigger>

</ControlTemplate.Triggers>

 

With this custom control:

 

<views:TransactionCell >

    <ContentPresenter SnapsToDevicePixels="True"/>

</views:TransactionCell>

 

Where all the binding above is done in code instead without using BindingExpressions.  The code unfortunately is quite verbose and not much fun to write, but it is pretty mechanical: 

 

    ///<summary>

    /// This class is here for performance reasons only.  The DataGridCell Template was too slow otherwise.

    ///</summary>   

    publicclassTransactionCell : Border

    {

        public TransactionCell()

        {

            this.DataContextChanged += newDependencyPropertyChangedEventHandler(OnDataContextChanged);

        }

 

        Transaction context;

        DataGridCell cell;

 

        protectedoverridevoid OnVisualParentChanged(DependencyObject oldParent)

        {

            DataGridCell cell = this.GetParentObject() asDataGridCell;

            SetCell(cell);

            base.OnVisualParentChanged(oldParent);

        }

 

        void SetCell(DataGridCell newCell)

        {

            if (this.cell != null)

            {

                cell.Selected -= newRoutedEventHandler(OnCellSelectionChanged);

                cell.Unselected -= newRoutedEventHandler(OnCellSelectionChanged);

            }

            this.cell = newCell;

            if (cell != null)

            {

                cell.Selected += newRoutedEventHandler(OnCellSelectionChanged);

                cell.Unselected += newRoutedEventHandler(OnCellSelectionChanged);

            }

        }

 

        void OnCellSelectionChanged(object sender, RoutedEventArgs e)

        {

            UpdateBackground();

            UpdateForeground();

        }

 

        void OnDataContextChanged(object sender, DependencyPropertyChangedEventArgs e)

        {

            ClearContext();

            Transaction t = e.NewValue asTransaction;

            if (t != null)

            {

                SetContext(t);

            }

           

            UpdateBackground();

            UpdateForeground();

            UpdateFontWeight();

        }

 

        void ClearContext()

        {

            if (this.context != null)

            {

                this.context.PropertyChanged -= newPropertyChangedEventHandler(OnPropertyChanged);

            }

        }

 

        void SetContext(Transaction transaction)

        {

            this.context = transaction;

            if (this.context != null)

            {

                this.context.PropertyChanged += newPropertyChangedEventHandler(OnPropertyChanged);

            }

        }

 

        void OnPropertyChanged(object sender, PropertyChangedEventArgs e)

        {

            switch (e.PropertyName)

            {

                case"IsReconciling":

                    UpdateBackground();

                    break;

                case"Unaccepted":

                    UpdateFontWeight();

                    break;

                case"IsDown":

                    UpdateForeground();

                    break;

                case"IsReadOnly":

                    UpdateForeground();

                    break;

            }

        }

 

        void UpdateBackground()

        {

            /*

               <DataTrigger Binding="{Binding Path=IsReconciling}" Value="True">

                    <Setter Property="Background" Value="{DynamicResource ReconciledRowBackgroundBrush}" TargetName="CellBorder"/>

                </DataTrigger

                <Trigger Property="IsSelected" Value="true">

                    <Setter Property="Background" Value="{DynamicResource RowSelectedBrush}" TargetName="CellBorder"/>

                </Trigger>

             */

            bool isReconciling = false;

            bool isSelected = IsSelected;

 

            if (this.context != null)

            {

                isReconciling = this.context.IsReconciling;

            }

            if (isSelected)

            {

                this.SetResourceReference(Border.BackgroundProperty, "RowSelectedBrush");

            }

            elseif (isReconciling)

            {

                this.SetResourceReference(Border.BackgroundProperty, "ReconciledRowBackgroundBrush");

            }

            else

            {

                this.ClearValue(Border.BackgroundProperty);               

            }

        }

 

        bool IsSelected

        {

            get

            {

                return (cell != null) ? cell.IsSelected : false;

            }

        }

 

        void UpdateForeground()

        {

            /*

               <Trigger Property="IsSelected" Value="true">

                    <Setter Property="Foreground" Value="{DynamicResource RowSelectedTextBrush}"/>

                </Trigger>

                <DataTrigger Binding="{Binding Path=IsDown}" Value="True">

                    <Setter Property="Foreground" Value="Red"/>

                </DataTrigger>

 

                <DataTrigger Binding="{Binding Path=IsReadOnly}" Value="True">

                    <Setter Property="Foreground" Value="Gray"/>

                </DataTrigger>

             *

            bool isSelected = IsSelected;

            bool isDown = false;

            bool isReadOnly = false;

 

            if (this.context != null)

            {

                isDown = this.context.IsDown;

                isReadOnly = this.context.IsReadOnly;

            }

            if (isReadOnly)

            {

                if (cell != null)

                {

                    cell.SetValue(DataGridCell.ForegroundProperty, Brushes.Gray);

                }

            }

            elseif (isDown)

            {               

                if (cell != null)

                {

                    cell.SetValue(DataGridCell.ForegroundProperty, Brushes.Red);

                }

            }

            elseif (isSelected)

            {               

                if (cell != null)

                {

                    cell.SetResourceReference(DataGridCell.ForegroundProperty, "RowSelectedTextBrush");

                }

            }

            else

            {               

                if (cell != null)

                {

                    cell.ClearValue(DataGridCell.ForegroundProperty);

                }

            }

        }

 

        void UpdateFontWeight()

        {

            /*

                <DataTrigger Binding="{Binding Path=Unaccepted}" Value="True">

                    <Setter Property="TextBlock.FontWeight" Value="Bold"/>

                </DataTrigger>

                <DataTrigger Binding="{Binding Path=Unaccepted}" Value="False">

                    <Setter Property="TextBlock.FontWeight" Value="Normal"/>

                </DataTrigger>

             */

 

            bool unaccepted = false;

 

            if (this.context != null)

            {

                unaccepted = this.context.Unaccepted;

            }

            if (unaccepted)

            {

                this.SetValue(TextBlock.FontWeightProperty, FontWeights.Bold);

            }

            else

            {

                ClearValue(TextBlock.FontWeightProperty);

            }

        }

    }

 

So what can we conclude from this little experiement? This was a relatively small change to my app.  I still have mountains of Styles in my Themes, I have 10-15 columns, each with complex DataTemplates, and complex controls in each cell from DatePickers to intellisense ComboBoxes.  So for this small change to make a 33% improvement in scrolling means that WPF DataBinding is a pig. Anything you can do to remove XAML binding will improve performance overall even though that is somewhat contrary to the XAML UI separation design point. 

 

Tags:

C# | Visual Studio | WPF

PasswordVault

by clovett8. April 2012 08:47

Video (wmv 15mb, (mp4 13mb)

Password Vault is my latest little ClickOnce app, built on the .NET Framework 4.0.

Managing online accounts is a real hassle these days. I have 180 separate userids and passwords to manage. Who can remember all that? And it’s not just userids and passwords any more, it’s also secret questions, password images, secure credit card numbers, links to specific URLs, license keys, and all sorts of other miscellaneous information associated with my online accounts.

The temptation of course is to get sloppy and use the same user id and password on every site and use real “secret” questions that I can remember but a wise person once said “don’t put all your eggs in one basket”. If a hacker breaks into one site I don’t want them to be able to steal my entire identity across all my other accounts, so I’ve diligently kept separate unique userids and passwords on every site and fake random unique secret questions. I also use strong passwords, combinations of digits, letters, and other special characters. But managing all this unique secure information for every account is a big hassle.

Up till now I’ve used a  password encrypted Microsoft Word file. Being a free-form text editor Word lets me paste anything I want, including all those annoying “secret questions”, even password image keys that some sites are now using, like Bank of America. But I want these passwords to be readily available on every machine that I use. Since the Word document is itself password encrypted, I can store it on my Windows Live Mesh folder which is automatically copied to all my machines.

But then if I edit the password file on one machine while it is offline then edit the file on another machine before it syncs then I have a conflict and will have to manually merge the two slightly different Word documents which is a real hassle.

So I wanted something that is a bit more structured that can handle automatic merging for me, but also without losing the flexibility of free form text editing for all that other random information besides the core web site, userid and password fields.

So I built Password Vault

The screen shot below shows a typical password file. Highlighted in various shades of green are the most recent changes. This green highlight fades over time, disappearing after one month.

When you select a row the 5th field appears containing a full rich text editor where you can paste any other details like secret questions or even an image key. You can cut & paste anything you want into this field and and it can be as big as you want. This field is also searchable.

When the program first starts the focus will be on the search field since you typically have some place in mind already before you launch the app. After you type in your search term (which is not case sensitive) the view will automatically filter down to show any matching rows:

From there you can select the row you want and press F12 to open that URL, and then you can copy the user name and password into the web site login page.

Implementation

I realize there’s a million similar apps out there, I just wanted to see how easy it would be to build one. The following is the solution I came up with.

Design.dgml (276.06 kb)

PasswordVault is about 1000 lines of code and shows the power of .NET Framework. It uses the .NET Cryptography api to encrypt/decrypt your password files using 256 bit encryption, and it uses WPF for the User Interface including a RichTextBox for the free form text editing part. PasswordVault also uses the Microsoft Credential Manager  to safely store your master password so you don’t have to keep entering the master password every time you open your password file.

Tags:

C# | Security | Visual Studio

Whats new in Visual Studio 11 Dependency Graph Features

by clovett10. March 2012 13:46

 

See video (WMV 17mb, MP4 14mb).

Visual Studio 11 has radically improved the Dependency Graph Features.   To take it for a spin I’ve loaded a real project that I’m working on, it contains about 25000 lines of code.  I can use the simplified top down dependency graph feature we see here under the Architecture menu and I see something new right off the bat called “Indexing the code”. 

What’s happening here is that Visual Studio 11 (I’ll call it dev11 from now on) is building a SQL database of all the dependencies in my code which in this case takes about 34 seconds.  Then it will use this database for all subsequent queries on the graph making the overall experience much better and it didn’t take much longer than dev10.  On dev10 the same operation on this solution takes 30 seconds.  So you’re paying a little overhead for the SQL database but you’ll see in a minute how much better the experience is as a result. 

For example, you’ll see that this top level dependency graph is very small, it only contains 20 nodes and 49 links.  The same top level graph in dev10 has about 12000 nodes and 60000 links, all hidden inside these collapsed groups, but that made any interaction on the canvas sluggish at best.

Dev11 instead is able to query the code index every time I expand something and get the detail.  Let’s try that here.  Oooh, nice smooth animation there too!  It found 4 new nodes.  You can see that there is an animation happening in the background here because it’s also fetching all the new dependencies for the nodes it’s just fetched.  Ok great.  Now we see the cross-group links showing all the dependencies for these new nodes. 

The look of the graph is also greatly improved; lighter and cleaner.  The problem with Generics is also fixed.  Dev10 put Generics in their own group which didn’t make much sense.  Generics are now integrated in with the rest of your code where they belong.

Let’s expand more of this graph.  The meat of the application is in this main dll here.

Bottom Up Dependency Graphs (2:02)

Now top down dependency graphs like this are interesting, for a large project they can show you the big picture of what’s happening in your code which you may have never seen before.  But sometimes you want a much smaller slice through your code.  I’m super excited about the new Solution Explorer Integration.

You may have already noticed that Solution Explorer can now drill down inside classes and methods.  You can also search.  I’m interested in a feature of this software called “portfolio”, and it comes back with all the classes and methods that contain that string.  Now if I select all of that you’ll see a new button that shows up that allows me to create a new dependency graph, with ancestors, of just the items I selected here.  It’ll use the code index again, this time to add all the dependencies between these things. 

That’s simply amazing.  This is called a “bottom up dependency graph” as opposed to top down dependency graph you get from the Architecture menu.

Now you also can build bottom up graphs in an incremental way.  For example, when I zoom in I see this method “GetInvestmentPortfolio” -  I want to know more about it.  That’s this method here (in the solution explorer) and I can right click and “find me everything that method Calls”, and see that information is now available in the solution explorer.  I can select all of that and add that to the dependency graph, with ancestors.  It uses the code index again and adds all those nodes, plus all the new dependencies between those nodes.   Great! 

Now I can look at the result, zoom in, and make some changes.  I don’t need mscorlib dependencies so I can delete that.  This class I will collapse.  Now I’m starting to build up a story about this particular feature that makes sense to me as a developer. 

Editing (3:55)

What’s more, the new DGML document editor supports full editing.  First off I can move nodes anywhere I want and all this is persisted, so if I save this graph, reload it, I see my layout the way I left it.  In fact, I’ve already done that let me open it up.  Here’s the same code and I’ve added some comments and a new group for an idea that could be added to the code in the future.

You can insert a new node using this hovering toolbar which allows you to add a node of the same type, you can also add a comment and that’s how I edited this graph very quickly and easily.  You can also add new links.

You can also insert nodes using the keyboard.  If you type the INSERT key a new node will show up where the mouse is.  You can type ALT-INSERT to add an outgoing dependency here, if you type ALT+SHIFT+INSERT you get an incoming dependency.

When you’re done editing you can press the layout button the toolbar again to get a full graph layout.  And if you didn’t mean to lose your custom layout you can hit undo to get back to the layout you had.

You can also edit the label of any node by just clicking on it twice and you can also add labels to links by pressing F2.  And of course, now that you can select links, you can delete them.  You can also select these nodes and put them in a new group.

You can also drag/drop a node across groups which will reparent that node in the target group.  When you do that you will see that cross-group arteries are computed automatically and we’ve actually created a circular dependency between these groups.  The analyzers by the way have been move to the legend add button – let’s add the circular analyzer and it shows us the problem that we’ve just created by highlighting these things in red.  I can move the node back and that will fix the problem.  This gives you the ability to do some interesting “what-if” analysis on your code.

By the way if you hold down the shift key when you move a node it actually bulldozes things out of the way instead of doing a drag/drop.  That can be handy if you need to make a little more room to put stuff.

Quick Find (6:27)

The quick find dialog that you see in the text editor has been integrated.  If I search for “investment” it takes me to the first hit and a really cool feature has been added here in the quick find command menu called “select all”.  This is handy if you want to do an operation like “delete” to delete all the matching nodes.  Of course, if I hit undo they all come back. 

Fast Incremental Layout (6:55)

What’s happening here is a fast incremental layout – it’s not doing a full clean layout like you get from the toolbar layout buttons.   That’s designed to give you good interactive performance when you’re editing these graphs.

Inline Toolbar (7:03)

You will also notice toolbar has been moved inside the graph document window– that’s really nice for multi-monitor situations where you want the toolbar to be available close to the graph you’re editing.

Zoom Widget (7:09)

There’s also a new zoom widget with a zoom to fit button.  You can also zoom to fit any group by double clicking it which will center and zoom in on it.  If you double click the graph background it will zoom to fit the entire graph.

Removed Some Unbaked Features (7:24)

You might have noticed that some features are missing.  There used to be a “dependency matrix”, and “neighborhood” and “butterfly” modes on the toolbar.  There was also a thing called “advanced selector” which showed up under the legend.  These features weren’t being used very much and it’s nice to see that Microsoft can actually remove stuff instead of bloating the code with stuff that nobody uses.

Help Pages (7:47)

If you create a new directed graph document you’ll see some new information about what you can do including some help on how the mouse works, and there were some slight tweaks here from dev10, like I said you can actually pan and zoom without holding the control key  down.  And there’s some help for various keyboard shortcuts.

Lastly, more good news, when you create a graph like this and you want to share it with people, dev11 Professional SKU will be able to open this graph and look at it, and even edit it.  The only feature missing from Pro SKU is the ability to reverse engineer new and create graphs from code using the code index.

So that’s about it!   I think you’ll agree that the dependency graph feature just got a whole lot cooler.

 

The following DGML document maps out where the new features fit WhatsNew.dgml (26.80 kb)

Tags:

dev11 | DGML | Visual Studio

Ruby Class Hierarchy to DGML

by clovett27. January 2012 08:49

I am having some fun learning Ruby. I'm using the Visual Studio 2010 Ruby IDE from www.sapphiresteel.com which makes things a lot more fun than using some silly interactive console window. They did a nice job.

Ruby is a strange little language. I'm quickly coming to think Ruby was deliberately designed to borrow as much stuff from as many different languages as possible. The ultimate mutt of programming languages. The following diagram illustrates what Ruby borrowed from where:

The thing that shocks me the most about Ruby is that all the strings are not Unicode. They say they are fixing that in Ruby 2.0, but that's not out till 2013. Amazing that a popular language this day and age would even dare to exist without native Unicode support.

I love DGML as you know, so I wrote the following code using the Ruby Symbol table to find all classes and generate a DGML graph of the Ruby class hierarchy:

#===============================================================================================

# This function generates a DGML graph from the superclass information in all known classes.

#===============================================================================================

class ClassHierarchy

      

       # This method finds all classes and calls the given handler passing the class.

       # (The "&" prefix here means the argument is a procedure)

       def AllClasses(&handler)

              # use the "each" method which takes a procedure (the stuff inside curly braces)

              # the |s| is the closure variable that binds to the "each" iterator.

              Symbol.all_symbols.each{

              |s|

                name = s.id2name

 

                # classes begin with uppercase letters

                # (notice the native support for regex in the language)

                if (name =~/[A-Z].*/) then

                            begin

                             # use eval to execute the code, which is just a symbol

                             # reference, so it returns the Class instance.

                             c = eval(name)

                             # if it's really a class then it should have a superclass

                             # method

                             sc = c.superclass

                             if (sc != nil) then

                                    # now pass this to our handler.

                                    handler.call(c)

                             end                 

                           rescue Exception

                             # exceptions here mean we found something that was not

                             # a class, so ignore it.

                           end

                 end

              }

       end

      

       # Generate a DGML graph linking every class with it's superclass

       def GenerateDgml()

         puts("<DirectedGraph xmlns='http://schemas.microsoft.com/vs/2009/dgml'>")

         puts("  <Links>")

 

         # use our AllClasses method, passing a procedure that prints out a

         # DGML <Link> for each class found, including a link to it's super class.

         AllClasses{|c| puts("    <Link Source='#{c.name}' Target='#{c.superclass.name}'/>") }

 

         puts("  </Links>")

         puts("</DirectedGraph>")

       end

end

 

browser = ClassHierarchy.new

browser.GenerateDgml();

 

 

This produces the following DGML graph

See the following DGML file: RubyClasses.dgml (9.71 kb)

Tags:

DGML | Ruby

About the author

Chris Lovett is a Principal Software Engineer at  Microsoft working on Visual Studio.

See my resume in SVG Smile.

Month List

Careers