Upgrading to the new MongoDB WiredTiger storage engine

For one of our customers we have a property search API backed by a mongo database using the original storage engine (MMAPv1) with almost 400k documents.  As the document count rose over time I kept horizontally scaling the Linux VM to the point where it was getting a little expensive (4GB RAM and 160GB disk) to host.  I did this scaling because general searching is difficult to keep fast without indexing every combination of fields you may search against and so I was constantly adding indexes to keep the performance reasonable.

Under load the mongod process would consistently have 28.9 GB Virtual Memory allocated and 940MB Resident while using about 20GB on disk.

Without the 4GB RAM searching performance would really suffer when a new (non-uniq) query would be executed.  Usually the second and subsequent times the same query was performant due I think to the fact that it was now in the working set.  Being RAM constrained keep all popular combinations from being available in the working set at the same time.

Besides just being slow many times the query would pass the timeout and just fail, I think we can all agree that an API that fails the 1st time you call it but works on subsequent calls is not ideal.

What to do?  I could just click the slider in the hosting control panel over one more click to the 8GB VM but now the cost is about comparable to some of the Mongo as a Service offerings and I’m wondering if I should have just used a SQL db…

Before we move on to the obvious solution mentioned in the title I should mention another constraint that a general purpose search has that makes the mongo queries difficult to keep fast; paging.  We have to support paging and provide the count of matching documents plus the predicate can be almost any combination of fields in the documents.  The only implementation I could come up with was to use limit and skip and do it in 2 steps:

  1. $cursor = $collection->find($query)->limit($pageSize)->skip($skip);
  2. $count = $collection->count($query, $max);

Note that tuning batchSize also helped but we were at the edge of acceptable performance and something needed to be done.

 

Along comes the Mongo press release for the optional new WiredTiger storage engine, what’s this?  It supports compression, that’s what we need!

Please understand this is not a direct apples to apples comparison because the Linux distro and kernel are not exactly the same but after provisioning a new VM and restoring from a mongo dump the numbers are staggering.   The on disk size (/var/lib/mongo) is now down to 2.1GB!  Under load the mongod process is using around 2GB Virtual and 1GB Resident.

 

Win, win.  Performance is comparable and it costs less to host with just the pain of converting the config file to the new format.  Nice job WiredTiger developers.

ASP.NET MVC DropDownListFor exception

If you create a basic view model with some data you wish to present in an HTML select list you might proceed as follows by using the C# 3.0 feature of automatic properties:

public class Agent {
   public int SelectedOfficeId { get; set; }
   public IEnumerable<SelectListItem> OfficeItems { get; set; }
}

Then in your Razor view markup:

@model ViewModels.Agent
@Html.DropDownListFor(m => m.SelectedOfficeId, Model.OfficeItems)

Now you move on to your controller just to check if all this is working:

   var model = new ViewModels.Agent();
   return View(model);

Well you will get this exception:
The ViewData item that has the key ‘SelectedOfficeId’ is of type ‘System.Int32’ but must be of type ‘IEnumerable<SelectListItem>’.

DropDownListForException

The problem is your collection of list items is null

In a real app you would have some mapping code that populated your view model anyway but with the exception not being intuitive debugging is difficult.

Another solution is to hand craft your properties to protect against this:

public class Agent
    {
        private IEnumerable<SelectListItem> _offices;
    
        public int SelectedOfficeId { get; set; }
        public IEnumerable<SelectListItem> OfficeItems
        {
            get
            {
                return _offices ?? new List<SelectListItem>();
            }
            set
            {
                _offices = value;
            }
        }

What do you think?

Good related references:

  1. DropDownListFor with ASP.NET MVC on Ode To code by K. Scott Allen
  2. C# 3.0 Auto Properties discussion

Dependency Injection in ASP.NET MVC 5

When beginning a new .NET project I find myself reviewing my preferences for Solution and Project organization as well as additional frameworks to include.

The state of Dependency Injection (DI) / Inversion of Control (IoC) in .NET can be overwhelming initially because of all the choices and the friction of adding yet another framework to your code base.

I have always been very keen on using multiple projects in my ASP.NET solutions to separate concerns so revisiting the choice of N-Layer vs Onion got me to reading this blog post. It explains clearly with simple C# code and diagrams which projects should reference each other and why and where the interface files should reside vs the concrete implementations.

I then started with Ninject via NuGet because that is what the Pluralsight author (Shawn Wildermuth) used in this course on building Web API and it looked approachable. My first problem came when I was binding Data Access Layer (DAL) interfaces in the Web project which doesn’t have an assembly reference to the DAL… this SO post covers it.

I couldn’t get Ninject to work by scanning assemblies, sure I could just add a reference to the DAL project from the UI but I’m very stubborn so I ventured out to the Google/Bing.

I found (sorry can’t remember how) this benchmark and comparison on .NET DI frameworks which in the conclusion pointed me at Simple Injector and LightInject mostly because they are fast.

After trying both of those and still not having any luck with the assembly scanning It finally dawned on me. With no reference to the DAL from Web the build system wasn’t copying DAL.dll to the bin directory! So the final solution is to setup a post build event to copy DAL.dll into the Web deployment directory (bin).

I settled on LightInject for this project so when I found this awesome DI testing project it reinforced my decision as LightInject scores very highly.

PhoneGap and Sencha Cmd debugging techniques

If your wrapping your Sencha Touch application in PhoneGap/Cordova using Sencha Cmd to automate the tasks here are some tips for when the process does not go smoothly.

First off since your starting with an existing Sencha Touch application you initialize the project for PhoneGap with this Sencha Cmd syntax:
sencha phonegap init App-Id App-Name
App-Id is usually in reverse domain notation (com.company.AppName) but this is also used as the package in the resulting Java (at least for Android) source file.  Be sure you get it right or if you want to change it I suggest doing a sencha phonegap remove and starting over as the builds tend to fail silently in cases like this.

Also note that App-Name should be the same as the value in the “name” property in the Sencha Touch app.json config file.

 

If all you get is a white screen in the emulator there is probably a JavaScript exception or bad asset reference happening before PhoneGap/Cordova initializes.  Remember this is just an html5 app so try loading it directly in your browser (file://) and then refer to the browsers developer tools console for hints.

In Chrome F12 followed by Ctrl-O then navigate to your Sencha project directory and select the index.html in the phonegap/www directory.

 

If your using Adobe’s phonegap build service they do have built in debugging but if your app is not launching correctly or hangs before the weirne debugger sees your target try some of these techniques:

  • Download the .apk file from build.phonegap.com and load it into a running Android emulator with: adb install path/to/app.apk
  • In the case where a local build works but pushing to the phonegap build service is failing you can zip up the contents of the phonegap/platforms/android/ directory for example and manually upload it via their web interface.  This was useful for me when my local phonegap install was 3.5.0 but the Adobe service only supported up to v3.4.0

 

In the case that your sencha app build -run native command is failing silently and depending on your remote setting (phonegap.local.properties) either is not spawning the emulator or not uploading your app to the phonegap build service try executing phonegap directly:

From your Sencha project directory cd phonegap then for example phonegap build android This will give you much more verbose output which hopefully will point you to your problem.

References:

  • http://phonegap.com/blog/2013/11/20/SenchaPhoneGap/

 

Javascript, PHP, and sorting

Does your daily development have you jumping between client and server code like mine often does?

I recently needed to sort an array of “documents” from a NoSQL DB in PHP. So off to PHP.net to review the docs… there are 13 array sort functions in the table of choices. I decided on usort because my JSON was an array of objects – which is represented in PHP by an array of associative arrays – where I want to sort on a field of the child array. Note the description for usort explains it sorts by value using a user defined function, in my case the value is each child array and I want to sort on the Object-ID field.

Here is the JSON:

photos: [
   {
      Object-ID: 1,
      Location: "http://foo/x.jpeg"
   }, {
      Object-ID: 2,
      Location: "http://foo/y.png"
   }
]

The Javascript idiom for this would be:

photos.sort(function (p1, p2) {
   return p1['Object-ID'] - p2['Object-ID']
});

Please note I know the Object-ID field is an int but you should consider datatypes in your scenario.

The point of this story is my initial PHP solution just didn’t feel right. First of all you write your 1 line sort function and then supply a string of the name as the 2nd argument to usort. While not a big deal having a string match a function name and hoping over the lifetime of the code they never get out of sync nags at me.

The next irritant was my code was in a class so this syntax did not work (as an aside a static method does not work either) and as the docs point out (in example #3) you can pass an array with the class name AND the sort function name. Darn now we have two magic strings.

   usort($photos, array('MyClass', 'sortby_object-id'));

Then I ran into this great “Functional Programming in PHP” reference:
http://www.phptherightway.com/pages/Functional-Programming.html

Nice!

   usort($photos, function ($p1, $p2) {
      return $p1['Object-ID'] - $p2['Object-ID'];
   });

I will admit if you read far enough down on the PHP doc page you will see example #4 using a closure which might also get you to this syntax but never the less PHP: The Right Way is worth a look.