Running out of memory resizing images in dot net

Question

I'm having a lot of issues running out of memory resizing images. I was at first convinced that I was simply leaking, so I switched my custom code for something based around https://github.com/JimBobSquarePants/ImageProcessor and the same issues occur.

So a source image causing problems might be 7137x10096 Uncompressed that should be 7137x10096x4 = 288220608 bytes or 274Mb. OK, thats "big" but why is it troublingly big? Running it IISExpress (32bit) its throws out-of-memory. Running on Azure in a 64bit server with 3.5GB it simply unreliable.

Error is "An exception of type 'System.OutOfMemoryException' occurred in System.Drawing.dll but was not handled in user code"

Code based on ImageProcessor throws on the imageFactory.Load statement

   public bool ProcessStream(Size size, Func<MemoryStream, bool> resultCallback, string gravity, Stream inStream, int quality=80)
    {
        inStream.Position = 0;
        bool res = false;
        using (var outStream = new MemoryStream())
        {
            using (var imageFactory = new ImageFactory())
            {
                var anchor = AnchorPosition.Center;
                if (gravity == "top") anchor = AnchorPosition.Top;
                if (gravity == "bottom") anchor = AnchorPosition.Bottom;
                var layer = new ResizeLayer(size, ResizeMode.Pad, anchor);

                imageFactory.Load(inStream)
                    .Resize(layer)
                    .BackgroundColor(System.Drawing.ColorTranslator.FromHtml("#FFFFFF"))
                    .Quality(quality)
                    .Save(outStream);
                res = resultCallback(outStream);
            }
        }
        return res;
    }

In its simplest test case I'm calling the above with

     public bool nop(MemoryStream s)
     {
         return true;
     }

     public string TestResize()
     {
         var resizer = new ResizeImage();
         var size = new System.Drawing.Size(300, 400);
         using (
             var filestream =
                 new FileStream(
                     @"C:\Users\Andrew\problem issues\NastyLargeImageThatBreaksThings.jpg",
                     FileMode.Open, FileAccess.Read))
         {
             var res = resizer.ProcessStream(size, nop, "top", filestream, 75);
             return res.ToString();
         }
     }

I can:

1) Try a different technique

2) Upgrade my Azure servers to have more memory (not sure if that will help)

3) Try resizing via a 3rd party service such as http://www.blitline.com/ which has some Azure specific integration.

Any understanding of why images this size can't be handled in Systen.Drawing would be much appreciated.