Using reflection to create a factory class for dynamic method invocation using templates.

•March 6, 2010 • 1 Comment

Using reflection to create a factory class for dynamic method invocation using templates.

My group is using entity framework and we are trying to decouple ourselves from our legacy framework which returns tightly bound data contracts to our customers.   We use extension methods to convert entities into data contracts.  This is actually more complicated than it sounds because we also have a concept of a business entity which is embedded into our data contracts.  Neither of them implements interfaces so converting from one to the other and back requires a series of loops, and logic switches.

Additionally we use classes called repositories in our data access layer which call entity framework and then map from entity framework entities to data contracts and business entities via extension methods.  

Anyway here is the problem.

  • Our entity repositories are tightly coupled to data contracts and entities and only returns data contracts.
  • Our data contracts and business entities are tightly coupled and don’t return interfaces.
  • We already have a great deal of extension methods that we want to reuse.

Here is the proposal.

  •  I want to create a factory in between entity repositories and the mapper classes so we can decouple our entity repositories from our data contracts and business entities.
  • I want to reuse the existing extension methods.
  • I want to write new mapper classes targeted at specific contracts which are not extension methods.
  • I want to account for overloaded methods.

 I also want the factory method to be as generic and flexible as possible, I don’t want the factory to know about mappers and entities and return types.   I decided to use Reflection but in a very optimized way so that I can maintain good performance.   If you want to know more about performance pitfalls of applications and reflection, there is a very good article by Joel Pobar in msdn magazine, you can read it here:  http://msdn.microsoft.com/en-us/magazine/cc163759.aspx

Approach A.

First the factory method to reuse the data extensions.  Since the legacy EF mapper class is a class of extension methods with an overloaded TODC() method, there is no way of passing in the method to invoke so I have to check the input and output type of the method to see if it matches the input and output type of the contracts.  Let’s create the method signature of my factory class.

public class EntityFactory

{  

public OutputType Create<OutputType, InputType>(InputType entity, Type mapperTemplate) where OutputType : class

{}   

}

OK fair enough, I have a generic output type, and input type, a mapper template type and some constraints.  NOTE:  mapperTemplate is the static extension class with the extension methods that I want to reuse.

Using reflection I will find the methods in the mapper template class and then check the return type and the input parameters.   I will need to use reflection’s  MemberInfo to do this. As you can see below,  once I find the method I invoke it and return the result.   This works and is optimized but not as optimized as it could be due to the fact that I am reusing an existing extension method class. 

NOTE:  Stay with me if this seems messy; it gets cleaner later on in the post.

public OutputType Create<OutputType, InputType>(InputType entity, Type mapperTemplate) where OutputType : class

      {

         Type[] inputTypes = new Type[1] { typeof(InputType) };

         foreach (MemberInfo mi in mapperTemplate.GetMethods())

         {

            if (((MethodInfo)mi).ReturnType == typeof(OutputType))

            {

               MethodInfo method = mapperTemplate.GetMethod(((MethodInfo)mi).Name, inputTypes);

               if (method != null)

               {

                  return method.Invoke(null, new object[] { entity }) as OutputType;

               }

            }

         }

         return null;

      }

You’ll notice that after I get the matching return type I attempt to retrieve the method by name and by passing in the method parameter types.  If the method is returned then I invoke the method, otherwise I try again.   If you don’t pass in the method parameter types you may get back an overloaded method by the same name and the attempt to invoke will cause reflection to throw an Ambiguous Match Exception

Here is how I call the method above from my unit tests.

DAL is the namespace for Data Access Layer.

DC is the namespace for my Data Contracts.

List<MyItem> myItems = new List<MyItem>();

  var results = this.repository.SearchResults(criteria);

  foreach (var item in results)

 {

   myItems.Add(entityFactory.Create<DC::Item, DAL::ItemTypeTemplate>(results, typeof(DAL.EFExtensionMappers))

    );

   }

Approach B.

Next I want my factory method to use new mapper template classes.  Since this class will have unique method names, I can just pass the method name and the template class and call invoke.  Pretty simple and very optimized.

public OutputType Create<OutputType, InputType>(InputType entity, Type mapperTemplate, string methodName) where OutputType : class

      {

         Type[] inputTypes = new Type[1] { typeof(InputType) };

         MethodInfo mi = mapperTemplate.GetMethod(methodName, inputTypes);

         return mi.Invoke(null, new object[] { entity }) as OutputType;

      }

Notice I pass in the input parameter types again to avoid the Ambiguous Match Exception caused by overloaded methods.

Here is how I call the method above from my unit tests.

DAL is the namespace for Data Access Layer.

DC is the namespace for my Data Contracts.

List<MyItem> myItems = new List<MyItem>();

var results = this.repository.SearchResults(criteria);

  foreach (var item in results)

  {

    myItems.Add(entityFactory.Create<DC::Item, DAL::ItemTypeTemplate>(results, typeof(DAL.NewtemMappers), “GetItems”)

         );

   }

Up to this point I have only been able to pass to methods with one argument.  Now I want to pass to methods with multiple arguments.   Here is how I handle this one.

public OutputType Create<OutputType>(Type[] inputTypes, object[] arguments, Type mapperTemplate, string methodName) where OutputType : class

      {

         MethodInfo mi = mapperTemplate.GetMethod(methodName, inputTypes);

         return mi.Invoke(null, arguments) as OutputType;

      }

Finally, here is how I call this method.

List<MyItem> myItems = new List<MyItem>();

var args = new object[] { results, “test” };

Type[] parms = new Type[2] { typeof(DAL::AgreementTypeTemplate), typeof(String) };

foreach (var item in results)

{

   myItems.Add(entityFactory.Create<DC:: Item >(parms, args, typeof(DAL. ItemTypeTemplate), “GetItems”)

);

 }

I hope you have found this type of dynamic factory helpful.

Cheers!

Where are my stable isotopes and is computer science really a science?

•February 17, 2010 • 1 Comment

 If you ask someone with a “hard science” degree about the discipline of Computer Science, they’ll be the first to tell you that Computer Science “it’s not a real science”.  I guess I kind of see their point because while  I don’t have a CS degree, (I’m probably too old to have one anyway ), I did graduate from UW with a degree in Interdisciplinary arts and sciences which means that I got to dabble in many different scientific disciplines.

Whenever I hear hard scientists say something like Computer Science “it’s not a real science” I think are just trying to convey that the hard sciences, like Chemistry for example, demand that you have a kind of decision tree where each subsequent decision is made from the conclusive  output of the previous decision which was also based upon hard scientific research.   It is also scientifically sound if you base new research upon research that the scientific community has either already done or has concluded is an acceptable starting point.   Even a “radical departure” in the scientific community is still based upon some solid scientific foundations.

In the field of computer science however, you can go completely mad.  You can ignore most or all of the facts, and you can even choose to ignore the entire CS community for that matter and design an entire solution however you want based upon your own conclusions about what is the right thing to do.  And guess what?  Your design will actually work well too, at least for a while anyway.  

I understand this completely, I have been there myself and I’ve even had to forgive myself over the years for these mistakes.  Perhaps it’s because Computer Science has a bit of Art mixed in with real science that we tend to move forward with enterprise level decisions on our own before asking what the rest of our community is doing or even testing our ideas first. 

Forging ahead when there is no prior foundation for what you are doing is gutsy and pioneering, but forging ahead on your own while ignoring the entire community who went before you is wholly offensive to scientists, and is tantamount to accepting a “flat earth” model.  

Nowadays, I’m starting to attend more and more meetings where questions are being asked like “what is the reality of that situation?” and “what is the likelihood that this will ever happen?”.   Is it good science to go in a direction based upon what  may happen in the unpredictable future?  Should we base our entire platform design on a few legacy clients when it would be cheaper and more efficient just to redesign the legacy clients to use the new platform?  What does this thing in the middle here do?   

I believe questions like these are the foundation for  the good critical, scientific examination that our profession requires in order to evolve.  But what is the answer?  Where are our stable isotopes?

To the extent that we argue against well established patterns, algorithms, and design principals and just “do our own thing” in the face of proven methodologies and best practices of our community, we fail to elevate our vocation to that of a “real” science.

At the core of Computer Science is the computer, and at our most basic level we are merely dealing with electrical circuits, switches if you will, and we’re trying to determine the most efficient way of turning them on and off.  Many well established patterns are out there, many of them have already been discovered and there are many more are yet to be discovered.  I promise you that stable isotopes do exist in our profession, we just need to remember to use them.

Cheers!

-c

My thoughts on the View Data Model pattern for ASP.NET MVC

•January 14, 2010 • Leave a Comment

 In almost every scenario only a small number of UI elements can be directly bound to the Model, especially if the developer has very little or no control at all over the Model itself.   In many cases, I have observed that the Model will have data types that are incompatible with the View and the Model data must be synthesized or “massaged” before it can be consumed directly by the View.  In order to compensate for this, the UI may have to perform complex operations that are in the purest terms, inappropriate actions for a UI layer. These operations may be too “View Specific” for the Model and at the same time too “Model Specific” for the View.

 The View Model pattern provides a bridge between the Model and the View, it is actually intended to provide a “Model of the View”.   The View Model provides a “View specific” Model that can be used directly by the View, it can perform data-transformations which enable direct data binding by the View’s controls and it can provide commands that the View can use to interact directly with the Model. I always think of a View Model as a View-Centric interface to the Model which provides View-Centric interaction with the Model.

For instance:

In my last project for example, we wanted to bind directly to dropdown boxes. Our architecture required that our models return entities that were much too complex for direct binding to the Views and we had some business specific policies (inappropriate logic for the UI layer) that had to be enforced for specific user types and program types which was much too complex for the Controller Actions.  If this wasn’t complex enough already for the UI, , our architecture also required that our policies be enforced on the Web Server. 

 To solve this problem, we created a View Data Model in the Web Application Layer which contained our policy logic and which “massaged” the model data in a way that allowed our controller code to be very small and enabled direct binding to our data types by our Views.  This also provided a bit of re-use, but keep in mind that at the time this was a new feature in ASP.NET MVC and there wasn’t a whole lot of information about the pattern.

Evolution in our world is very rapid, like fruit flies and Today, just a month or two later there are plenty of examples and best practices that we can reference as we continue to implement this great pattern moving forward and if we think about it up front, will provide a lot of good re-use.  Our first implementation was not perfect and if you want to learn more, I would suggest that you choose from among the many examples that now exist to evolve the ideas even further.

 Cheers! -c

Using ASP.NET MVC and OpenXML api to stream Excel files.

•September 25, 2009 • 2 Comments

I’ve been heads down for the last several weeks Scrumming and it has been a while since I’ve updated my blog so I figured if I was going to keep your interest  I should give you something really useful.  We’ve been transitioning from web forms to ASP.NET MVC and we had a requirement to export the contents of a grid to an Excel spreadsheet from one of our ASP.NET MVC pages and prompt the user to open or save.

After some digging I wasn’t happy with the idea of porting the legacy solution over so I did a little more digging.  Opening an Excel spreadsheet is not anything new but I had a few requirements of my own.

  • I wanted to call the function from my ASP.NET MVC controller.
  • I wanted to send in a collection of data from my model.
  • I wanted to minimize hard disk I/O.
  • I wanted to add custom headers.
  • I wanted to give my Excel file a meaningful name.
  • I wanted to give my spreadsheet a meaningful name, say matching the page the user was exporting from to the sheet name.
  • I wanted to be able to extend the class to add custom styles.
  • I wanted to format the columns for a specific type of data
  • I wanted Excel to recognize the file right away.  We can’t have any “unrecognized file type” nag screens confusing our customers.
  • Finally, I wanted to use this with web forms.

It seemed pretty obvious that I could minimize I/O by saving the file to a memory stream, but what about the rest of my requirements? 

I’ll take you through the steps and provide source code in the following sections. 

Source code should be available soon at The Code Project.

Step 1 Extending the controllers.    Steve Sanderson’s blog   has an excellent article on extending the controllers and returning an Excel file as an action result.   Phil Haack also has an excellent post on the subject which you can find here:  Steve’s example uses a DataContext and XML to stream the file however Excel doesn’t recognize the file and prompts you with a nag screen.  Excel will eventually open the file and it looks perfect but I really wanted to eliminate this prompt.  I also wanted to be able to pass in my own data from the model and localize the Headers and the data.  I borrowed extensively from Steve’s tutorial with a little modification.  So how do I get Excel to play nice? 

I had to learn a little bit about Excel compatibility to get things to work so I starting with the Office Developer’s kit going back to Excel 97.  If you have a few hours or an open weekend I would suggest you read up on BIFF8.  While BIFF8 is well documented in the Office Developer’s SDK, and I did eventually get the file to stream in BIFF8 I found it almost incomprehensible when it comes to adding additional features and frankly I didn’t want to be the keeper of knowledge Excel spreadsheets within my group which is ultimately what would have happened.  This solution needs to scale out because I work with a lot of other developers and they will have more complex requirements as the project grows.  I came across the OpenXML api api from Microsoft the OOXML extensions from James Westgate and I was off and running.   More information on OpenXml and file formats here.

 Get the Open XML SDK 

Get the OOXML api 

So to recap I’m going to extend my ASP.NET MVC controllers so all of my controllers will have access to the functionality, I will send in data from my model, next I will create a custom spreadsheet and workbook with styled headers, and finally I will assign a meaningful name to the the Excel file and sheet.

 

Step1.  Extending the controller. 

 If you have read Steve or Phil’s blog, and I encourage you to read these blogs first, this will look like pretty standard stuff, no magic going on here.

 

namespace YourNamespaceGoesHere.Controllers.Extensions

{

   using System.Linq;

   using System.Web.Mvc;

   using YourNamespaceGoesHere.Controllers.Results;

 

   /// <summary>

   /// Excel controller extensions class.

   /// </summary>

   public static class ExcelControllerExtensions

   {

      /// <summary>

      /// Controller Extension: Returns an Excel result constructor for returning values from rows.

      /// </summary>

      /// <param>This controller.</param>

      /// <param>Excel file name.</param>

      /// <param>Excel worksheet name: default: sheet1.</param>

      /// <param>Excel row values.</param>

      /// <returns>Action result.</returns>

      public static ActionResult Excel(this Controller controller, string fileName, string excelWorkSheetName, IQueryable rows)

      {

         return new ExcelResult(fileName, excelWorkSheetName, rows);

      }

 

      /// <summary>

      ///  Controller Extension: Excel result constructor for returning values from rows and headers.

      /// </summary>

      /// <param>This controller.</param>

      /// <param>Excel file name.</param>

      /// <param>Excel worksheet name: default: sheet1.</param>

      /// <param>Excel row values.</param>

      /// <param>Excel header values.</param>

      /// <returns>Action result.</returns>

      public static ActionResult Excel(this Controller controller, string fileName, string excelWorkSheetName, IQueryable rows, string[] headers)

      {

         return new ExcelResult(fileName, excelWorkSheetName, rows, headers);

      }

 

      /// <summary>

      ///  Controller Extension: Excel result constructor for returning values from rows and headers and row keys.

      /// </summary>

      /// <param>This controller.</param>

      /// <param>Excel file name.</param>

      /// <param>Excel worksheet name: default: sheet1.</param>

      /// <param>Excel row values.</param>

      /// <param>Excel header values.</param>

      /// <param>Key values for the rows collection.</param>

      /// <returns>Action result.</returns>

      public static ActionResult Excel(this Controller controller, string fileName, string excelWorkSheetName, IQueryable rows, string[] headers, string[] rowKeys)

      {

         return new ExcelResult(fileName, excelWorkSheetName, rows, headers, rowKeys);

      }

   }

}

 

I have 3 overloads, here. 

  • Overload 1 takes the Excel File name and an IQueryable collection of rows. .
    • Overload 2 takes the Excel File Name, a worksheet name, and IQueryable collection of rows and an array of headers.
    • This method allows for passing in headers separately but they must match the keys for the rows. 
    • Overload 3 takes the Excel File Name, a worksheet name, and IQueryable collection of rows and an array of headers and a collection of row keys.
      • This method allows you to have different header names for your rows.

 

Step 3 extending the action result.  This class stores the values passed in when constructed.  If you examine this class you’ll see that it overrides the ExcecuteResult function and adds the functionality to create the Excel file and save it to a memory stream.  This is a pretty clever technique and I have deliberately separated the functionality of streaming the file from the functionality of creating the file so my Excel class can be reused used in non MVC applications.

namespace YourNamespaceGoesHere.Controllers.Results

{

   using System;

   using System.IO;

   using System.Linq;

   using System.Web;

   using System.Web.Mvc;

   using YourNamespaceGoesHere.Controllers.ControllerExtensions;

 

   /// <summary>

   /// Excel result class

   /// </summary>

   public class ExcelResult : ActionResult

   {

      /// <summary>

      /// File Name.

      /// </summary>

      private string excelFileName;

 

      /// <summary>

      /// Sheet Name.

      /// </summary>

      private string excelWorkSheetName;

 

      /// <summary>

      /// Excel Row data.

      /// </summary>

      private IQueryable rowData;

 

      /// <summary>

      /// Excel Header Data.

      /// </summary>

      private string[] headerData = null;

 

      /// <summary>

      /// Row Data Keys.

      /// </summary>

      private string[] rowPointers = null;

 

      /// <summary>

      /// Action Result: Excel result constructor for returning values from rows.

      /// </summary>

      /// <param>Excel file name.</param>

      /// <param>Excel worksheet name: default: sheet1.</param>

      /// <param>Excel row values.</param>

      public ExcelResult(string fileName, string workSheetName, IQueryable rows)

         : this(fileName, workSheetName, rows, null, null)

      {

      }

 

      /// <summary>

 

namespace YourNamespaceGoesHere.Controllers.Results

{

   using System;

   using System.IO;

   using System.Linq;

   using System.Web;

   using System.Web.Mvc;

   using YourNamespaceGoesHere.Controllers.ControllerExtensions;

 

   /// <summary>

   /// Excel result class

   /// </summary>

   public class ExcelResult : ActionResult

   {

      /// <summary>

      /// File Name.

      /// </summary>

      private string excelFileName;

 

      /// <summary>

      /// Sheet Name.

      /// </summary>

      private string excelWorkSheetName;

 

      /// <summary>

      /// Excel Row data.

      /// </summary>

      private IQueryable rowData;

 

      /// <summary>

      /// Excel Header Data.

      /// </summary>

      private string[] headerData = null;

 

      /// <summary>

      /// Row Data Keys.

      /// </summary>

      private string[] rowPointers = null;

 

      /// <summary>

      /// Action Result: Excel result constructor for returning values from rows.

      /// </summary>

      /// <param>Excel file name.</param>

      /// <param>Excel worksheet name: default: sheet1.</param>

      /// <param>Excel row values.</param>

      public ExcelResult(string fileName, string workSheetName, IQueryable rows)

         : this(fileName, workSheetName, rows, null, null)

      {

      }

 

      /// <summary>

      /// Action Result: Excel result constructor for returning values from rows and headers.

      /// </summary>

      /// <param>Excel file name.</param>

      /// <param>Excel worksheet name: default: sheet1.</param>

      /// <param>Excel row values.</param>

      /// <param>Excel header values.</param>

      public ExcelResult(string fileName, string workSheetName, IQueryable rows, string[] headers)

         : this(fileName, workSheetName, rows, headers, null)

      {

      }

 

      /// <summary>

      /// Action Result: Excel result constructor for returning values from rows and headers and row keys.

      /// </summary>

      /// <param>Excel file name.</param>

      /// <param>Excel worksheet name: default: sheet1.</param>

      /// <param>Excel row values.</param>

      /// <param>Excel header values.</param>

      /// <param>Key values for the rows collection.</param>

      public ExcelResult(string fileName, string workSheetName, IQueryable rows, string[] headers, string[] rowKeys)

      {

         this.rowData = rows;

         this.excelFileName = fileName;

         this.excelWorkSheetName = workSheetName;

         this.headerData = headers;

         this.rowPointers = rowKeys;

      }

 

      /// <summary>

      ///  Gets a value for file name.

      /// </summary>

      public string ExcelFileName

      {

         get { return this.excelFileName; }

      }

 

      /// <summary>

      ///  Gets a value for file name.

      /// </summary>

      public string ExcelWorkSheetName

      {

         get { return this.excelWorkSheetName; }

      }

 

      /// <summary>

      /// Gets a value for rows.

      /// </summary>

      public IQueryable ExcelRowData

      {

         get { return this.rowData; }

      }

 

      /// <summary>

      /// Execute the Excel Result.

      /// </summary>

      /// <param>Controller context.</param>

      public override void ExecuteResult(ControllerContext context)

      {

         MemoryStream stream = ExcelDocument.Create(this.excelFileName, this.excelWorkSheetName, this.rowData, this.headerData, this.rowPointers);

         WriteStream(stream, this.excelFileName);

      }

 

      /// <summary>

      /// Writes the memory stream to the browser.

      /// </summary>

      /// <param>Memory stream.</param>

      /// <param>Excel file name.</param>

      private static void WriteStream(MemoryStream memoryStream, string excelFileName)

      {

         HttpContext context = HttpContext.Current;

         context.Response.Clear();

         context.Response.AddHeader(“content-disposition”, String.Format(“attachment;filename={0}”, excelFileName));

         memoryStream.WriteTo(context.Response.OutputStream);

 

         memoryStream.Close();

         context.Response.End();

      }

   }

}

 

 

Finally creating the Excel file. Just a note about static methods.  I wanted to be sure that my Excel Document was thread safe so I created static methods at the expense of making this easily mockable.  You’ll need to weight that decision on your own.

 

Any public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe.

 

 

 

namespace YourNamespaceGoesHere.Controllers.ControllerExtensions

{

   using System;

   using System.IO;

   using System.Linq;

   using DocumentFormat.OpenXml;

   using DocumentFormat.OpenXml.Extensions;

   using DocumentFormat.OpenXml.Packaging;

   using DocumentFormat.OpenXml.Spreadsheet;

 

   /// <summary>

   /// Excel document.

   /// </summary>

   public static class ExcelDocument

   {

      /// <summary>

      /// Default spread sheet name.

      /// </summary>

      private const string DefaultSheetName = “Sheet1″;

 

      /// <summary>

      /// Create the exel document for streaming.

      /// </summary>

      /// <param>Excel file name.</param>

      /// <param>Excel worksheet name: default: sheet1.</param>

      /// <param>Row data to write.</param>

      /// <param>Header data.</param>

      /// <param>Row pointers.</param>

      /// <returns>Memory stream.</returns>

      public static MemoryStream Create(string documentName, string excelWorkSheetName, IQueryable rowData, string[] headerData, string[] rowPointers)

      {

         return CreateSpreadSheet(documentName, excelWorkSheetName, rowData, headerData, rowPointers, null);

      }

 

      /// <summary>

      /// Create the spreadsheet.

      /// </summary>

      /// <param>Excel file name.</param>

      /// <param>Excel worksheet name: default: sheet1.</param>

      /// <param>Row data to write.</param>

      /// <param>Header data.</param>

      /// <param>Row pointers.</param>

      /// <param>Style sheet.</param>

      /// <returns>Memory stream.</returns>

      private static MemoryStream CreateSpreadSheet(string documentName, string excelWorkSheetName, IQueryable rowData, string[] headerData, string[] rowPointers, Stylesheet styleSheet)

      {

         int rowNum = 0;

         int colNum = 0;

         int maxWidth = 0;

         int minCol = 1;

         int maxCol = rowPointers == null ? minCol : rowPointers.Length;

         maxCol = maxCol == 1 && headerData == null ? 1 : headerData.Length;

 

         MemoryStream xmlStream = SpreadsheetReader.Create();

         SpreadsheetDocument spreadSheet = SpreadsheetDocument.Open(xmlStream, true);

 

         SetSheetName(excelWorkSheetName, spreadSheet);

 

         if (styleSheet == null)

         {

            SetStyleSheet(spreadSheet);

         }

         else

         {

            spreadSheet.WorkbookPart.WorkbookStylesPart.Stylesheet = styleSheet;

            spreadSheet.WorkbookPart.WorkbookStylesPart.Stylesheet.Save();

         }

 

         WorksheetPart worksheetPart = SpreadsheetReader.GetWorksheetPartByName(spreadSheet, excelWorkSheetName);

 

         WriteHeaders(headerData, out rowNum, out colNum, out maxWidth, spreadSheet, worksheetPart);

         AddCellWidthStyles(Convert.ToUInt32(minCol), Convert.ToUInt32(maxCol), maxWidth, spreadSheet, worksheetPart);

 

         if (rowPointers == null || rowPointers.Length == 0)

         {

            WriteRowsFromHeaders(rowData, headerData, rowNum, out maxWidth, spreadSheet, worksheetPart);

         }

         else

         {

            WriteRowsFromKeys(rowData, rowPointers, rowNum, out maxWidth, spreadSheet, worksheetPart);

         }

 

          // Save to the memory stream

         SpreadsheetWriter.Save(spreadSheet);

         spreadSheet.Close();

         spreadSheet.Dispose();

         return xmlStream;

      }

 

      /// <summary>

      /// Set the name of the spreadsheet.

      /// </summary>

      /// <param>Spread sheet name.</param>

      /// <param>Spread sheet.</param>

      private static void SetSheetName(string excelSpreadSheetName, SpreadsheetDocument spreadSheet)

      {

         excelSpreadSheetName = excelSpreadSheetName ?? DefaultSheetName;

 

         Sheet ss = spreadSheet.WorkbookPart.Workbook.Descendants<Sheet>().Where(s => s.Name == DefaultSheetName).SingleOrDefault<Sheet>();

         ss.Name = excelSpreadSheetName;

      }

 

      /// <summary>

      /// Add cell width styles.

      /// </summary>

      /// <param>Minimum column index.</param>

      /// <param>Maximum column index.</param>

      /// <param>Maximum column width.</param>

      /// <param>Spread sheet.</param>

      /// <param>Work sheet.</param>

      private static void AddCellWidthStyles(uint minCol, uint maxCol, int maxWidth, SpreadsheetDocument spreadSheet, WorksheetPart workSheetPart)

      {

         Columns cols = new Columns(new Column() { CustomWidth = true, Min = minCol, Max = maxCol, Width = maxWidth, BestFit = false });

 

         workSheetPart.Worksheet.InsertBefore<Columns>(cols, workSheetPart.Worksheet.GetFirstChild<SheetData>());

      }

 

      /// <summary>

      /// Set the style sheet.

      // Note: Setting the style here rather than passing it in ensures that all worksheets will have a common user interface design.

      /// </summary>

      /// <param>Spread sheet to change.</param>

      private static void SetStyleSheet(SpreadsheetDocument spreadSheet)

      {

         // Note: Setting the style here rather than passing it in ensures that all worksheets will have a common user interface design.

         Stylesheet styleSheet = spreadSheet.WorkbookPart.WorkbookStylesPart.Stylesheet;

 

         styleSheet.Fonts.AppendChild(

             new Font(new FontSize() { Val = 11 }, new Color() { Rgb = “FFFFFF” }, new FontName() { Val = “Arial” }));

 

         styleSheet.Fills.AppendChild(new Fill()

         {

            PatternFill = new PatternFill()

            {

               PatternType = PatternValues.Solid,

               BackgroundColor = new BackgroundColor() { Rgb = “D8D8D8″ }

            }

         });

 

         spreadSheet.WorkbookPart.WorkbookStylesPart.Stylesheet.Save();

      }

 

      /// <summary>

      /// Save the styl for worksheet headers.

      /// </summary>

      /// <param>Cell location.</param>

      /// <param>Spreadsheet to change.</param>

      /// <param>Worksheet to change.</param>

      private static void SeatHeaderStyle(string cellLocation, SpreadsheetDocument spreadSheet, WorksheetPart workSheetPart)

      {

         Stylesheet styleSheet = spreadSheet.WorkbookPart.WorkbookStylesPart.Stylesheet;

         Cell cell = workSheetPart.Worksheet.Descendants<Cell>().Where(c => c.CellReference == cellLocation).FirstOrDefault();

 

         if (cell == null)

         {

            throw new ArgumentNullException(“Cell not found”);

         }

 

         cell.SetAttribute(new OpenXmlAttribute(“”, “s”, “”, “1”));

         OpenXmlAttribute cellStyleAttribute = cell.GetAttribute(“s”, “”);

         CellFormats cellFormats = spreadSheet.WorkbookPart.WorkbookStylesPart.Stylesheet.CellFormats;

 

         // pick tthe first cell format.

         CellFormat cellFormat = (CellFormat)cellFormats.ElementAt(0);

 

         CellFormat cf = new CellFormat(cellFormat.OuterXml);

         cf.FontId = styleSheet.Fonts.Count;

         cf.FillId = styleSheet.Fills.Count;

 

         cellFormats.AppendChild(cf);

 

         int a = (int)styleSheet.CellFormats.Count.Value;

 

         cell.SetAttribute(cellStyleAttribute);

 

         cell.StyleIndex = styleSheet.CellFormats.Count;

 

         workSheetPart.Worksheet.Save();

      }

 

      /// <summary>

      /// Replace special characters.

      /// </summary>

      /// <param>Value to input.</param>

      /// <returns>Value with special characters replaced.</returns>

      private static string ReplaceSpecialCharacters(string value)

      {

         value = value.Replace(“’”, “‘”);

         value = value.Replace(““”, “\””);

         value = value.Replace(“””, “\””);

         value = value.Replace(“–”, “-“);

         value = value.Replace(“…”, “…”);

         return value;

      }

 

      /// <summary>

      /// Write values to the spreadsheet.

      /// </summary>

      /// <param>Row Column Value.</param>

      /// <param>Value to write.</param>

      /// <param>Spreadsheet to write to. </param>

      /// <param>Worksheet to write to. </param>

      private static void WriteValues(string cellLocation, string strValue, SpreadsheetDocument spreadSheet, WorksheetPart workSheet)

      {

         WorksheetWriter workSheetWriter = new WorksheetWriter(spreadSheet, workSheet);

 

         int intValue = 0;

         if (strValue.Contains(“$”))

         {

            strValue = strValue.Replace(“$”, “”);

            strValue = strValue.Replace(“,”, “”);

 

            workSheetWriter.PasteValue(cellLocation, strValue, CellValues.Number);

         }

         else if (int.TryParse(strValue, out intValue))

         {

            workSheetWriter.PasteValue(cellLocation, strValue, CellValues.Number);

         }

         else if (string.IsNullOrEmpty(strValue))

         {

            workSheetWriter.PasteText(cellLocation, strValue);

         }

         else

         {

            workSheetWriter.PasteText(cellLocation, strValue);

         }

      }

 

      /// <summary>

      /// Write the excel rows for the spreadsheet.

      /// </summary>

      /// <param>Excel row values.</param>

      /// <param>Excel row-key values.</param>

      /// <param>Row number.</param>

      /// <param>Max width.</param>

      /// <param>Spreadsheet to write to. </param>

      /// <param>Worksheet to write to. </param>

      private static void WriteRowsFromKeys(IQueryable rowData, string[] rowDataKeys, int rowNum, out int maxWidth, SpreadsheetDocument spreadSheet, WorksheetPart workSheet)

      {

         maxWidth = 0;

 

         foreach (object row in rowData)

         {

            int colNum = 0;

            foreach (string rowKey in rowDataKeys)

            {

               string strValue = row.GetType().GetProperty(rowKey).GetValue(row, null).ToString();

               strValue = ReplaceSpecialCharacters(strValue);

               maxWidth = strValue.Length > maxWidth ? strValue.Length : maxWidth;

 

               string cellLocation = string.Format(“{0}{1}”, GetColumnLetter(colNum.ToString()), rowNum);

               ExcelDocument.WriteValues(cellLocation, strValue, spreadSheet, workSheet);

 

               colNum++;

            }

 

            rowNum++;

         }

      }

 

      /// <summary>

      /// Convert column number to alpha numeric value.

      /// </summary>

      /// <param>Column number.</param>

      /// <returns>ASCII value for number.</returns>

      private static string GetColumnLetter(string colNumber)

      {

         if (string.IsNullOrEmpty(colNumber))

         {

            throw new ArgumentNullException(colNumber);

         }

 

         string colName = null;

 

         try

         {

            for (int i = 0; i < colNumber.Length; i++)

            {

               string colValue = colNumber.Substring(i, 1);

 

               int asc = Convert.ToInt16(colValue) + 65;

 

               colName += Convert.ToChar(asc);

            }

         }

         finally

         {

            colName = colName ?? “A”;

         }

 

         return colName;

      }

 

      /// <summary>

      /// Write the values for the rows from headers.

      /// </summary>

      /// <param>Excel row values.</param>

      /// <param>Excel header values.</param>

      /// <param>Row number.</param>

      /// <param>Max width.</param>

      /// <param>Spreadsheet to write to. </param>

      /// <param>Worksheet to write to. </param>

      private static void WriteRowsFromHeaders(IQueryable rowData, string[] headerData, int rowNum, out int maxWidth, SpreadsheetDocument spreadSheet, WorksheetPart workSheet)

      {

         WorksheetWriter workSheetWriter = new WorksheetWriter(spreadSheet, workSheet);

         maxWidth = 0;

 

         foreach (object row in rowData)

         {

            int colNum = 0;

            foreach (string header in headerData)

            {

               string strValue = row.GetType().GetProperty(header).GetValue(row, null).ToString();

               strValue = ReplaceSpecialCharacters(strValue);

               maxWidth = strValue.Length > maxWidth ? strValue.Length : maxWidth;

 

               string cellLocation = string.Format(“{0}{1}”, GetColumnLetter(colNum.ToString()), rowNum);

 

               ExcelDocument.WriteValues(cellLocation, strValue, spreadSheet, workSheet);

               colNum++;

            }

 

            rowNum++;

         }

      }

 

      /// <summary>

      /// Write the excel headers for the spreadsheet.

      /// </summary>

      /// <param>Excel header values.</param>

      /// <param>Row number.</param>

      /// <param>Column Number.</param>

      /// <param>Max column width</param>

      /// <param>Maximum Column Width to write to. </param>

      /// <param>Worksheet to write to. </param>

      private static void WriteHeaders(string[] headerData, out int rowNum, out int colNum, out int maxWidth, SpreadsheetDocument spreadSheet, WorksheetPart workSheet)

      {

         rowNum = 1;

         colNum = 0;

         maxWidth = 0;

 

         foreach (string header in headerData)

         {

            string strValue = ReplaceSpecialCharacters(header);

 

            string cellLocation = string.Format(“{0}{1}”, GetColumnLetter(colNum.ToString()), rowNum);

            maxWidth = strValue.Length > maxWidth ? strValue.Length : maxWidth;

            ExcelDocument.WriteValues(cellLocation, strValue, spreadSheet, workSheet);

            SeatHeaderStyle(cellLocation, spreadSheet, workSheet);

            colNum++;

         }

 

         rowNum++;

      }

   }

}

 

 

I hope you found this useful.  I’m not going to go into great detail here about the OpenXml api.  This is all working code so I would actually recommend you step through the code yourself and get acquainted with Microsoft’s OpenXml developer web site.  http://msdn.microsoft.com/en-us/office/bb738430.aspx

 

 

Regards,

 

-c

Using a Unity Container with Service Locator for Test Driven Development

•February 4, 2009 • Leave a Comment

If you read my page on Unity Container and Service Locator, the page  on Using Unity and Service Locator for Unit Testing brings them together.

So how do I use a unity container for Test Driven Development?  

This is the really cool part of DI containers, since you’re using Dependency Injection via Interfaces, you can change behavior at runtime.

Since I’m currently working on an ASP.Net MVC prototype I’m going to demonstrate Unity Testing a Controller which uses Dependency Injection.

  In my case I’m using ASP.Net MVC controllers that have a dependency on certain actions.  I wanted to keep the controller as clean as possible so I created an Iaction interface.

Let’s call it HomeController for now.

 

/// <summary>

   /// HomeController class.

   /// </summary>

   public class HomeController : SecuredBaseController

   {

      /// <summary>

      /// Search Action Interface.

      /// </summary>

      private ISearchAction searchAction;

 

      /// <summary>

      /// Constructor for Home Controller.

      /// </summary>

/// <param name=”searchAction”>Home page search action.</param>

      public HomeController(ISearchAction searchAction)

      {

         this.searchAction = searchAction;

      }

 

/// <summary>

      /// Load view Index.

      /// </summary>

      /// <returns>Action Result.</returns>

      public ActionResult Home()

      {

         ViewData["Title"] = “Summary”;

         ViewData["Message"] = “Welcome to ASP.NET MVC!”;

 

         //// get valid review status

         SubmissionStatus[] submissionStatusList = this.searchAction.GetSubmissionStatusForAuditor();

       

 SelectList reviewStatusList = new SelectList(submissionStatusList, “Id”, “InternalDisplayName”);

         ViewData["ReviewStatusId"] = reviewStatusList;

 

         //// primary countries

         CountryRegion[] countryList = this.searchAction.GetCountries();

         SelectList countrySelectList = new SelectList(countryList, “Id”, “Name”);

         ViewData["CountryId"] = countrySelectList;

 

         //// get programs

         Program[] programs = this.searchAction.GetFilteredProgramList();

         SelectList programSelectList = new SelectList(programs, “Id”, “Name”);

 

         ViewData["ProgramId"] = programSelectList;

 

         //// get activity types

         Item[] activityTypes = this.searchAction.GetActivityTypesForAuditor();

        

SelectList activityTypesSelectList = new SelectList(activityTypes, “Id”, “Name”);

        

ViewData["ActivityTypeId"] = activityTypesSelectList;

 

         return View(“ViewPoeSubmissions”);

      }

 

Ok so you can see that we have a search page that calls an Search Action interface with all the code contained in a SearchAction class.  The SearchAction class is in a, you guessed it..  Unity Container.  I won’t go into the factory class that creates the unity container right now, I’ll write another article on that item but for now, the Unity container is actually created on Application_Start() and looks something like:       ControllerBuilder.Current.SetControllerFactory(UnityContainers.Instance.Resolve<IControllerFactory>());

So how do we test this?  As I mentioned above, the untiy container is created on Application_Start()  for our web site but since the controller takes an Interface, I can mock that up using a Repository pattern in the Unit Tests.

Here is what the SearchAction class looks like:

/// <summary>

   /// Search action class for the controller.

   /// </summary>

   public class SearchAction : IPoeSearchAction

   {

      /// <summary>

 /// Taxonomy model interface.

 /// </summary>

 private ITaxonomyModel taxonomyModel;

 

 /// <summary>

 /// Activity request model interface.

 /// </summary>

 private IRequestModel requestModel;

 

/// <summary>

/// Search Action.

/// </summary>

/// <param name=”taxonomyModel”>Taxonomy Model Interface.</param>

/// <param name=”requestModel”>Request Model Interface.</param>public SearchAction(ITaxonomyModel taxonomyModel, IRequestModel requestModel)

      {

         this.taxonomyModel = taxonomyModel;

         this.requestModel = requestModel;

      }

 

 

Notice the constructor for Search Action takes 2 interfaces, ItaxonomyModel and IrequestModel.

The GetCountries() method of the SearchAction class looks like this:

    /// <summary>

      /// Get list of primary countries.

      /// </summary>

      /// <returns>Country region collection.</returns>

      public CountryRegion[] GetCountries()

      {

         // Grab the Instance from Activity Request Process.

         return this.taxonomyModel.GetCountries();

      }

 

It’s actually the TaxonomyModel class that calls the web service and takes care of the GetCountries() code so I can create a class called TestTaxonomyModel that implements the ItaxonomyModel interface and in that test class I can put mock code in the GetCountries method that will return a subset of Test Data. 

 

 

 

 

 

The RequestModel class works the same way.

/// <summary>

/// Search for a Claim.

/// </summary>

/// <param name=”form”>Form Collection.</param>

/// <returns>Activity Based Claim Search Result.</returns>

 public ClaimSearchResult Search(FormCollection form)

 {

 

/// Create some search criteria from the form data.

ClaimSearchCriteria criteria = this.CreateSearchCriteria(form);

 

 return this.requestModel.SearchClaims(searchCriteria);

 

}

I can create a class called TestRequestModel that implements the IRequestModel interface and put logic to return test data based upon the search criterion passed in.

Here is what my unit test looks like.  First I initialize a unity container with test data:

/// <summary>

/// Internal chip service locator.

/// </summary>

private IServiceLocator serviceLocator;

 

/// <summary>

/// Unity container.

/// </summary>

private IUnityContainer unityContainer;

 

 /// Initialize tests.

 /// </summary>

 TestInitialize]

public void InitializeTests()

{

 

this.unityContainer = new UnityContainer()

          .RegisterType<IController, HomeController>()

          .RegisterType<ISearchAction, SearchAction>();

 

this.unityContainer.Configure<InjectedMembers>()

           .ConfigureInjectionFor<SearchAction>(new InjectionConstructor(new object[] { new TestTaxonomyModel(), new TestRequestModel() }))

          

 .ConfigureInjectionFor<HomeController>(new InjectionConstructor(this.unityContainer.Resolve<ISearchAction>()));

 

unityContainer.Resolve<IController>();

 

this.serviceLocator = new MyServiceLocator(this.unityContainer);

 

}

 /// <summary>

 /// Search test method.

 /// </summary>

 [TestMethod]

 public void SearchTest()

 {

   var homeController = new HomeController(this.serviceLocator.GetInstance<SearchAction>());

 

FormCollection form = new FormCollection();

 

         form["ProgramId"] = “3”;

         form["ActivityId"] = “8”;

         form["CountryId"] = “33”;

         form["Filter"] = “3”;

         form["Region"] = “Foo”;

        

searchController.Search(form);

    

}

There – you can see it is above, we call the Search method on the search controller psssing it the form we created and the code will be applied against our test controllers. 

 

Regards,

-c

Are Properties really a bad idea?

•February 2, 2009 • Leave a Comment

Like many experienced programmers over the years, I have been exposed to hundreds of different programming styles,

read long philosophical threads about programming methodologies new and old and have been fortunate enough to work with some really expert programmers.

 

 

For the most part, I have enjoyed the dialogue and somewhere along the way have learned to let go of of the small stuff,

that is unless of course the small stuff which I would otherwise overlook in the right context, is at risk of becoming

detrimental in the wrong context.

 

 

I can remember when I was learning to sail I kept calling those things that hung off the side of the boat bumpers

when in fact I should have been calling them fenders. My instructor immediately corrected me and said “its ok, that really

used to bother me but I’ve learned to let that one go.” What he had learned through unfortunate experience, was that students

needed to understand immediately how to tie fenders on correctly well before the vessel reached the the slip so as not to

damage hull, less important to him was what we called them at the moment of impact. Besides, I would learn the correct

terminology with more practice soon enough.

 

 

So back to the subject at hand, properties. Properties are one of the small things I’ve learned to let go of over the years,

because there is just so much widespread confusion over them that they have become ubiquitous. It would require a huge

 investment of resources and time to get emotionally tied to their “proper” use one way or the other, unless, as I mentioned

earlier, I feel they are endangering the codebase.

 

 

If you search for the effective use of properties, you’ll find literally thousands of conflicting statements about why properties exist,

 when to favor properties over member variables and even whether they are evil and should be exercised from all properly

object oriented code completely. Yet in object oriented programming classes, we’re told that people have certain properties

like a name or sex or origin, and we should immediately start adding them to the person base class (another thing I’m learning to

let go of..) so that whatever extends that person class will pick up those common properties. OK fair enough, this is exactly how I learned about properties too.

 

 

So are properties really a bad idea?

My thesis is that as long as we don’t violate some important design principals, properties are absolutely fine under most

circumstances. What are properties exactly? Well, if you wiki property, you’ll find that a property is defined as a kind of hybrid

sort of class member somewhere between a data member and a method. In the spirit of keeping it simple, I have highlighted some rules about properties and how to use them without causing too much damage.

 

 

Maintainability:

Do the properties you are about to add offer more or less maintainability? In other words, are you writing a class to DoSomething() or are you writing a class that requires the dependant class to get all the information from you and then doSomething(). Good object oriented classes DoSomething(), not the other way around. It is generally acceptable to return an object or interface using property accessors. For instance, if our person class, is used by our personnel system, we expect the personnel system to ask the person for their address and to have the person return the address to the personnel system, what we don’t want is to have the personnel system ask the person for the street, city, state, and zip code and then have the personnel system build the address, that would be counterintuitive.

 

 

Object Oriented or Procedural Programming:

Are you really writing OO code if you have properties everywhere? Consider for instance that procedural programming relies on local knowledge of objects to perform tasks while OO programming is based on use cases and conversations. Do getters and setters come up during conversations? I will reference the experts Kent Beck and Ward Cunningham for more detailed information but I encourage you to think about the classes you need, and the responsibilities they must perform as well as how they will collaborate. Beck and Cunningham developed some really good practices using CRC cards to model “conversations between classes”.

 

 

Encapsulation:

While properties offer better encapsulation than fields do, you shouldn’t necessarily start blindly replacing fields with properties in every case. The more accessors you create, the more dependency you create and the more at risk you are to exposing implementation details of your class. Consider your DoSomething() methods, do they take reference or value parameters. Also consider the use of internal and private properties carefully, do you really need them?

 

State:

Getters and setters which “report” get or “change” set the state of some private member variable shouldn’t change the state of the object itself, or any other unrelated object for that matter. Remember when I said that a property is somewhere between a method and a field or member? This is one of the potential hazards of treating your properties as methods. Code in getters or setters which change the state of the object itself will cause unbelievable headaches from adopters. Properties should be reliable, if you call any property repeatedly, you should be able to depend on the value returned from the getter.

 

 

Performance:

Impact on performance is another potential pitfall of treating your properties as methods. Performing lengthy or costly operations in getter or setter methods could significantly impact performance. Having for instance a large custom collection or web control object which is dynamically built or retrieved in your getter or setter methods could slow performance as objects are added and increase performance as objects are removed. This can lead to lots of head scratching from adopters, not to mention the loss of hours of productivity troubleshooting mysterious performance issues, which appear and disappear for no reason.

 

 

So should you use accessors or not? In my experience, when you start working for large corporations, refactoring decisions sometimes become more about ROI than style and quality is re-defined as the number lines of code over defects or the number of unit tests over uncovered blocks rather than the elegance of design or the implementation of industry best practices. Style and elegance often gives way to brute force. Human beings are certainty freaks, and we have to balance planning and risk with our “drop dead” delivery deadlines. There is never enough time up front but then again, nothing lives forever anyway right? To sum it up, if you are going to use accessors you need understand what they are for and for goodness sakes use them properly because, just to tie the metaphors together, a damaged hull is expensive to fix and if the hole is big enough you may never fully trust the vessel again.

 

 

Beck, K. & Cunningham, W. (1989) A laboratory for teaching Object-Oriented Thinking. Apple computer whitepaper, http://c2.com/doc/oopsla89/paper.html#references

 

Wagner, B. (2008) Choose between Methods and Properties. Visual Studio Magazine p.40 http://visualstudiomagazine.com/columns/article.aspx?editorialsid=2719

Haloub, A. (2003) Why getter and setter methods are evil. http://www.JavaWorld.com

 

 

 

 

Improve Performance with Asynchronus processing, Anonymous methods and Thread Pools.

•February 1, 2009 • 1 Comment

The very first week in my new group, I was asked if I could improve the performance of a loop in a method that was being called from the business logic façade of an application. The purpose of the loop was to synthesize data back and forth between custom objects fetched from the data layer into similar but incompatible custom objects in the business layer. I know what you’re thinking… pretty standard OOP stuff from around 2001, but you probably also know that this stuff requires good code familiarity due to the symbiotic relationship that exists between all of the moving parts. To make things even more difficult, some of the properties in those objects were actually returning collections of other incompatible objects that needed to be synthesized back and forth as well.

I decided that the best course of action would probably be to focus on the loop itself. I knew that the loop iterated through a collection of objects and converted them to a collection of other objects, and I knew that the collection could be potentially large so I figured that I could get some immediate performance improvement by simply populating the objects in the loop asynchronously versus in-line.

The first thing I needed was a baseline performance count. I need to get a count In milliseconds of how long the loop alone took to complete without the fetch operation, which could skew the statistics by as much as 50%. In the old days of 1.1 I used to have to import the QueryPerformanceCounter class in the Kernel32 library and run start, stop and clear operations and calculate the totals. http://msdn2.microsoft.com/en-us/library/ms979201.aspx

Today however we can do the same thing with the Stopwatch class in the System.Diagnostics library with 2 commands, Start and Stop. Don’t forget to write the output from the stopwatch using a formatted string and ToString() the ElapsedMiliseconds property to avoid boxing operations.

Let’s look at a mock up of the code:

public sealed class SolutionEntityMapper

{

public static FooDataCollection MapObject(FooEntityCollection fooEntityCollection)

{

 

FooDataCollection fooDataCollection = new FooDataCollection();

 

#if DEBUG

System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();

 

//Start the timer

sw.Start();

#endif

 

 

//Run the loop inline

foreach (FooEntityItem fooEntityItem in fooEntityCollection)

{

FooDataItem fooDataItem = new FooDataItem ();

fooDataItem.SomeProperty = fooEntityItem.SomeProperty;

 

//Do more long running tasks

fooDataCollection.Add(fooDataItem);

}

 

#if DEBUG

//Stop the timer

sw.Stop();

 

//Print out the elapsed time in miliseconds without boxing

System.Diagnostics.Debug.WriteLine(String.Format(“Operation Executed in {0} miliseconds.”, sw.ElapsedMilliseconds.ToString()));

#endif

 

//Return the collection

return fooDataCollection;

}

}

Looks pretty straight forward, start the watch run the loop, make the new object, fill in the properties (remember some properties return collections so they have to call other helper functions to populate the collection) and stop the watch. To prepare the collection, I created a unit test that would pre populate the collection with 1000 Entity objects. The loop ran in just under 6000 miliseconds, for conversion sake we’ll call it 6 seconds. That’s barely tolerable for a small company, but for a large company serving 30 thousand plus users it just doesn’t scale well at all.

OK so let’s try it asynchronously. The first think I wanted to accomplish was to have the loop set data item in motion, move on to the next, set it in motion and then wait for everything to finish and return the data. The code to synthesize data was pretty long and I tend to favor chunky methods that isolate behavior so I moved the code to a method named ProcessEntity and focused on the asynchronus operations in the MapObject method. Next I had to create a delegate for ProcessEntity method so I could call it Asynchrounsly. Lock the shared method.

public sealed class SolutionEntityMapper

{

Static object myLock = new object();

 

delegate FooDataItem ProcessEntityDelegate (FooEntityItem fooEntityItem);

 

private static FooDataItem processEntity(FooEntityItem fooEntityItem)

{

lock (myLock);

{

FooDataItem fooDataItem = new FooDataItem();

fooDataItem.SomeProperty = fooEntityItem.SomeProperty;

//Do more long running tasks

return fooDataItem;}

}

 

}

}

 

Now that I’ve got the methods separated let’s think about how to call the loop asynchronously. We know that if we call BeginInvoke and then call EndInvoke during our loop, then we may as well just run the loop in line because EndInvoke blocks until the thread is completed. The only way to run a loop asynchronously and achieve the type of performance we need is to use a callback method. This presents it’s own problems because we also need the callback method to return a filled in fooDataItem and then add the item to a collection and return the collection back to the calling method.

We could use out parameters, but then how do we get the output into a new collection when they’re returning asynchronously? We could declare a Member collection and populate that but keeping track of member collections that we fill in asynchronous callback methods is not something I want to do in future generations of the code. Here is where the beauty of C# Anonymous methods come in. If we create an anonymous delegate for the callback method, we can access the dataItem in one single method. One consideration if we use an anonymous method for the callback is to ensure that we don’t return control to the calling method before the collection is filled up. For this I created a while loop that ensures the collection returned is the same size as the collection processed. I check the count property on each collection because each time a member is added to the collection, the Count property is updated and the member is stored for later use. This isn’t the best choice, but we’ll fix it later in the post.

 

while (fooDataCollection.Count != fooEntityCollection.Count)

{ /*sit and spin until the thread finishes*/ }

 

 

 

 

 

Now let’s write the code.

public sealed class SolutionEntityMapper

{

Static object myLock = new object();

delegate FooDataItem ProcessEntityDelegate (FooEntityItem fooEntityItem);

delegate void callBackDelegate(IAsyncResult ar);

 

 

public static FooDataCollection MapObject(fooEntityCollection fooEntityCollection)

{

 

ProcessEntityDelegate processEntityDelegate = new ProcessEntityDelegate SolutionEntityMapper.processEntity);

FooDataCollection fooDataCollection = new FooDataCollection();

 

#if DEBUG

System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();

 

//Start the timer

sw.Start();

#endif

 

//Create the Anonymous delegate method

callBackDelegate del = delegate(IAsyncResult ar)

{

ProcessEntityDelegate processSolution = ar.AsyncState as ProcessEntityDelegate;

 

//Pull the item out of the IAsynchResult

FooDataItem fooDataItem = processSolution.EndInvoke(ar);

 

//Add the item to the collection

fooDataCollection.Add(fooDataItem);

 

};

 

 

//Start the loop

foreach (fooEntityItem fooEntityItem in fooEntityCollection)

{

//Start Asynchronous operation

IAsyncResult result = processEntityDelegate.BeginInvoke(fooEntityItem, new AsyncCallback(del), processEntityDelegate);

 

}

 

while (fooDataCollection.Count != fooEntityCollection.Count)

{ /*sit and spin until the thread finishes*/ }

 

#if DEBUG

/Stop the timer

sw.Stop();

 

//Print out the elapsed time in miliseconds without boxing

System.Diagnostics.Debug.WriteLine(String.Format(“Operation Executed in {0} miliseconds.”, sw.ElapsedMilliseconds.ToString()));

#endif

 

//Return the collection

return fooDataCollection;

 

}

 

private static FooDataItem processEntity(FooEntityItem fooEntityItem)

{

lock (myLock);

{

FooDataItem fooDataItem = new FooDataItem();

fooDataItem.SomeProperty = fooEntityItem.SomeProperty;

 

//Do more long running tasks

return fooDataItem;

}

}

}

 

So what’s the output time for processing 1000 records synchronously vs asynchronusly?

Original loop: 6000 milliseconds or 6 seconds.

Asynchronous loop: 194 milliseconds or 0.194 seconds.

Not a bad improvement for a little refactoring but, can do better? Waiting for the asynchronous operations to finish with a while loop is Not something we want to do if we can avoid it.

Asynchronous calls were really designed more for running single threads for processing like I/O operations while Thread Pools were designed for processing multiple tasks called background threads, and they take care of the thread management overhead for us. The web pool will hold a maximum settable number of worker threads up to a predefined limit (default is 25) and keep them in a suspended state so they are ready for the next asynchronous operation. For now we’ll use the default, more notes on tuning at the end of this posting. For more information please refer to http://msdn2.microsoft.com/en-us/library/0ka9477y.aspx

 

We will need to ensure that control is not returned to the calling method until the thread pool is finished and we have several options available to us. My choice is to use the AutoResetEvent in combination with a WaitHandle. More information on the topic can be found here http://msdn2.microsoft.com/en-us/library/system.threading.waithandle.aspx. Ideally we would replace the while loop with an array of Auto Reset Events and add one AutoResetEvent per thread, then call the WaitHandle.WaitAll passing in the AutoResetEvent array. To make things more in line with the example above, I will replace the while loop with a single AutoResetEvent and signal the WaitHandle using a similar syntax to the original while loop.

 

 

Let’s replace the asynchronous callback delegate with a thread pool waitcallback delegate and see what happens.

public sealed class SolutionEntityMapper

{

Static object myLock = new object();

 

delegate FooDataItem ProcessEntityDelegate (FooEntityItem fooEntityItem);

 

We turn the anonymous method to a thread pool wait callback method except this time we pull out the FooDataItem from the threadContext. Create an array of auto reset events and add a single AutoResetEvent to the array, initializing it to false.

 

 

public static FooDataCollection MapObject(fooEntityCollection fooEntityCollection)

{

 

ProcessEntityDelegate processEntityDelegate = new ProcessEntityDelegate (SolutionEntityMapper.processEntity);

FooDataCollection fooDataCollection = new FooDataCollection();

AutoResetEvent[] threadCompleted = new AutoResetEvent[] { new AutoResetEvent(false) };

 

 

 

#if DEBUG

System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();

//Start the timer

sw.Start();

#endif

 

WaitCallback del = delegate(Object threadContext)

{

FooDataItem fooDataItem = threadContext as FooDataItem;

//Add the item to the collection

fooDataCollection.Add(fooDataItem);

 

//Signal the end of processing

if (fooDataCollection.Count == fooEntityCollection.Count)

threadCompleted[0].Set();

 

}

 

Now we can move the Thread Pool code into the loop:

//Start the loop

foreach (fooEntityItem fooEntityItem in fooEntityCollection)

{

//Start Asynchronus operation

 

ThreadPool.QueueUserWorkItem(new WaitCallback(del), processEntityDelegate (fooEntityItem));

 

}

#if DEBUG

/Stop the timer

sw.Stop();

 

//Print out the elapsed time in miliseconds without boxing

System.Diagnostics.Debug.WriteLine(String.Format(“Operation Executed in {0} miliseconds.”, sw.ElapsedMilliseconds.ToString()));

#endif

 

//Wait for all events to complete

WaitHandle.WaitAny(threadCompleted);

//Return the collection

return fooDataCollection;

}

 

So what’s the output time for processing 1000 records synchronously vs asynchronously vs Thread Pooling?

Original loop: 6000 milliseconds or 6 seconds.

Asynchronous loop: 194 milliseconds or 0.194 seconds.

Thread Pool: 101 milliseconds or 0.101 seconds.

 

Tuning: The number of threads in the pool will control how many tasks you can complete in tandem. A thread pool with a limit of 25 will allow 25 worker threads per processor core. That is actually plenty, when dealing with threads less is more. A dual processor or dual-core machine with a thread pool set to 25 max threads will allow 50 thread pool threads. You may use logic to deduce that if your application receives 100 simultaneous requests on a dual processor system with a thread pool limit of 25, 50 will be immediately processed and the other 50 will wait in the queue. As the initial 50 are completed the other will move up in the queue. In fact, this is not the case. The thread pool will fire up a single thread, which is very expensive, and then use that thread to process an asynchronus request.  When control is returned, the thread is returned to the pool to be used for processing another request. The thread pool will reuse the thread because it is much cheaper to reuse an existing thread than to create a brand new one for each request. It is the thread pools job to manage threads so don’t touch it! Just leave it to the default settings.   

Also, remember that the number of users in your system or simultaneous connections is not the same as the number of simultaneous requests.

Best Regards,

-c

 

 
Follow

Get every new post delivered to your Inbox.