Moving C1 to Azure (File I/O)

by ingvar 17. november 2010 12:33

This blog post focus on file I/O when moving a complex solution to the Azure platform.

Composite C1 is a state of the art ASP.NET 4.0 CMS and is currently not 100% Azure ready. There is a lot of things to consider when moving a really complex product like Composite C1 to the Azure platform. For a full description on all the problems we havae moving C1 to Azure and all the people that attended a three day workshop, see this very descriptive blog post by Marcus Wendt.

Brainstorm and one-to-one implementation

First we sat down a listed all the classes we knew did file I/O. Some of the classes that we found were StreamReader, StreamWriter, TextReader and XmlReader. We also found some methods:XElement.Load, XElement.Save, XDocument.Load and XDocument.Save.

After we completed this list, we created one-to-one implementations of the found classes. These classes had the all same methods as the original classes and contained one private field of the original class type. The methods mapped one-to-one to the original class. Here is a simple example:


[Serializable]
public class C1StreamWriter : TextWriter, IDisposable
{
    private StreamWriter streamWriter;

    public C1StreamWriter(string path)
    {
        streamWriter = new StreamWriter(path);
    }


    public override void Write(char value)
    {
        streamWriter.Write(value);
    }

    /* And all the other constructors and methods */
}

Then we did a search'n'replace on the whole solution to insert our classes instead of the originals. We also created extension methods to replace the found methods. Next task was to see if we had found ALL the classes/methods that did file I/O. Which proved to be a rather had task.

IntelliTrace

To see if we had found all the classes, we used the new feature in Visual Studio 2010, IntelliTrace. We enabled IntelliTrace with only the File events marked and then started C1 in debug mode. This quickly showed that we had missed several classes and methods. One-to-one implementations of these newly found classes were made and IntelliTrace was fired up again.

After some time doing this, it became rather time consuming and error prone looking through several thousands stack traces of file I/O events. So I started looking at the IntelliTrace API and developed a tool, that we could use to filter the events in an intelligent way. See my blog post on how to get started with the IntelliTrace API.

FxCop

After doing all this work on finding all file I/O’s it would be nice to have a way of ensuring that no C1 developer by mistake used one of the, now forbidden, classes or methods. Also, it would be nice to have something to give all developer using C1, so they could do their developing knowing that their code also would be Azure ready. FxCop and custom rules to the rescue! See my blog post on how to make custom FxCop rules.

Findings

So which classes and methods did we find? And how hard was it to exchange them with our own classes and methods. To answer these questions I will group them into 4 groups: Static classes, non-static classes, methods and configuration classes. In the following I'll describe in more detail about the findings in each group.

Static classes

The found classes in this group were:

  • System.IO.File
  • System.IO.Directory

These were really easy to create our own implementation off. Just create a static class and do a one-to-one mapping of all methods.

Non.static classes

The found classes in this group were:

  • System.IO.StreamReader (Disposable)
  • System.IO.StreamWriter (Disposable)
  • System.IO.FileStream (Disposable)
  • System.IO.FileSystemWatcher
  • System.IO.FileInfo
  • System.IO.DirectoryInfo

These were also really easy to create our own implementation off. One important thing to remember here is to implement the dispose method the right way. Otherwise there will be problems with writing to unclosed files etc. Here is an example of doing it the right way:

~FileStream()
{
    Dispose(false);
}

protected overwrite void Dispose(bool disposing)
{
    if (disposing)
    {
        originalClass.Dispose();
    }
}

Methods

The found methods in this group were:

  • System.Xml.Linq.XDocument.Load (Local and remote)
  • System.Xml.Linq.XDocument.Save
  • System.Xml.Linq.XElement.Load (Local and remote)
  • System.Xml.Linq.XElement.Save
  • System.Xml.XmlReader.Create
  • System.Xml.XmlWriter.Create
  • System.Xml.XmlSchemaSet.Add
  • System.Xml.Xsl.XslCompiledTransform.Load

Here we simply created new static methods, some of them, extension methods and used a stream approach rather than using a path/uri approach. So instead of passing a path-string to XmlReader.Create we passed our own implementation of System.IO.FileStream to the XmlReader.Create method.


Special care had to be taken when making new methods for XDocument and XElement because their string version of the Load method can also fetch a file over the network. Here we had to look at the inputUri string and see if it was a local file (Use our own FileStream implementation) or remote (Use WebRequest to fetch the file).

Configuration classes

Found classes in this group were:

  • System.Configuration.Configuration
  • System.Configuration.ExeConfigurationFileMap (The Load method)

These classes/methods was the hardest ones. There was no way of replacing their file I/O functionality in a nice way, like we did with the other classes. So in this case we had to accept some local file I/O. But what about the Azure platform and shared configuration file across multiple instances etc? One way of solving this is to have some hooks on when a configuration file is loaded and saved. In this way we could 'fetch' a configuration file on load and 'store' it on save. Here is the full implementation of a new Configuration class that solves this:

public class C1Configuration
{
    Configuration _configuration;

    public static C1Configuration Load(string path)
    {            
        ExeConfigurationFileMap map = new ExeConfigurationFileMap();
        map.ExeConfigFilename = path;
        Configuration configuration =
            ConfigurationManager.
            OpenMappedExeConfiguration(map, ConfigurationUserLevel.None);            

        return new C1Configuration(configuration);
    }

    protected Configuration(Configuration configuration)
    {
        _configuration = configuration;
    }

    public ConfigurationSectionCollection Sections
    {
        get
        {
            return _configuration.Sections;
        }
    }

    public ConfigurationSection GetSection(string sectionName)
    {
        return _configuration.GetSection(sectionName);
    }

    public void Save()
    {
        _configuration.Save();
    }
}

API and plugin architecture

Next step was to create an API for C1 developer to use when doing file I/O. And a plugin architecture so that we could make C1 run on a local IIS, on the Azure platform or possible other platforms. The API is for most parts the same as the API for the original classes and methods, so this was just simple make-it-so work. C1 uses Microsoft Enterprise library (Object build) as plugin architecture. So this work was also pretty straight forward. At the moment we are not done with this work but when we are done, I'll post at link to the API.

Final thoughts

At the moment we have not created a Azure implementation of our file I/O plugin. This implementation will use the blob storage for keeping the files. So in the near future I'll post how it went with the Azure implementation.

Also still missing is the ASP.NET/Webserver file I/O part. This can be resolved by using Virtualizing Access to Content. Another solution could be: Having the website files locally and do some kind of synchronization if files are added/updated/deleted. This synchronization is possible through our new file I/O abstraction layer and can be implemented in the Azure implementation. This synchronization could also be used to solve the System.Configuration.Configuration problem.

Stay tuned for details regarding the Azure implementation and other cool stuff!

Tags:

.NET | C# | C1

Custom FxCop rules

by ingvar 3. november 2010 09:41

FxCop is a good way of ensuring the health and quality of your code. FxCop ships with a bunch predefined rules. But sometimes these rules are not enough. Fortunately there is a SDK for creating you own rules for FxCop. It's pretty simple to create your own rules and in the following I will describe a step-by-step guide to create custom FxCop rules.

Step 1: Creating a Visual Studio project

You start by creating a new class library project (call it MyFxCopRules) and then add the three following references:

  • FxCopSdk.dll
  • FxCopCommon.dll
  • Microsoft.Cci.dll

These can be found in the FxCop install directory (c:\Program Files\Microsoft FxCop 1.36).

 

Step 3: Adding a new cs class file

Add a new cs class file and here I will name this class DoNotCallXDocumentLoadWithPath.

 

Step 4: Implementing the rule

Here is the implementation of the rule. There is one important detail to the code (See below).

namespace MyFxCopRules
{
    public class DoNotUseMethodWithStringBaseRule : BaseIntrospectionRule
    {
        protected string ClassName { get; set; }

        public DoNotUseMethodWithStringBaseRule(string name, string className)
            : base(name,
                   "MyFxCopRules.FxCopRules",
                   typeof(DotNotUseStreamWriterClass).Assembly)
        {
            this.ClassName = className;
            this.MethodToExclude = new List<string>();
        }


        public List<string> MethodToExclude { get; set; }


        public override ProblemCollection Check(Member member)
        {
            Method method = member as Method;
            if (method == null) return this.Problems;

            for (int i = 0; i < method.Instructions.Count; i++)
            {
                Instruction instruction = method.Instructions[i];

                if (instruction.Value == null) continue;

                string value = instruction.Value.ToString();

                if (!value.Contains(this.ClassName)) continue;

                if (this.MethodToExclude.Where(f => value.Contains(f)).Any()) continue;

                Method calledMethod = instruction.Value as Method;
                if (calledMethod == null) continue;

                if (calledMethod.Parameters.Count == 0) continue;

                Parameter parameter = calledMethod.Parameters[0];

                if (parameter.Type.FullName != typeof(string).FullName) continue;

                Resolution resolution = GetResolution(new string[] { method.ToString() });
                this.Problems.Add(new Problem(resolution));
            }

            return this.Problems;
        }
    }
}


DoNotCallXDocumentLoadWithPath inherits the class BaseIntrospectionRule. BaseIntrospectionRule constructor takes three arguments. Getting one of these wrong and the rule will not show up in the FxCop UI or FxCop will refuse to load the assembly.

  • name: This must be unique among all your rules and it's used in the XML rule description file. See step 6.
  • resourceName: This string is used by the SDK to locate an embedded xml resource file, by calling GetManifestResourceStream on the assembly given as the third argument. The string should have the following composition: (default)namespace.filename. In this example the string is “MyNamespace.FxCopRules”. It's optional to append the string with ".xml" like this "MyNamespace.FxCopRules.xml".
  • resourceAssembly: This should point to the assembly where the embedded xml resource file is located (named in the second argument - resourceName).


Step 5: Adding a XML rule description file

Each rule needs to have a XML description in order to work. If a rule is missing its description or there is an error in the description, the rule wont turn up in the FxCop UI. So for step 5 you need to do the following:

  • Add a new empty XML file to your project. Here I name this file FxCopRules.xml
  • Change the build action to: Embedded Resource. Important!


Step 6: Writing the rule description

Here is an example of how to write the rule description:


<?xml version="1.0" encoding="utf-8" ?>
<Rules FriendlyName="My Custom IO Rules">
  <Rule TypeName="DoNotCallXDocumentLoadWithPath"
        Category="MyCategory"
        CheckId="MyCategory.DoNotCallXDocumentLoadWithPath">
    <Name>Dot not call System.Linq.Xml.XDocument.Load(string uri, ...)</Name>
    <Description>Dot not call System.Linq.Xml.XDocument.Load(string uri, ...)</Description>
    <Owner></Owner>
    <Url></Url>
    <Resolution>Use the stream version of System.Linq.Xml.XDocument.Load</Resolution>
    <Email></Email>
    <MessageLevel Certainty="100">Warning</MessageLevel>
    <FixCategories>Breaking</FixCategories>
  </Rule>
</Rules>

A few notes on some of the values in this description:

  • TypeName: This should be the same as the first argument to the BaseIntrospectionRule constructor. See step 4.
  • CheckId: This should be unique among all your rules in the same category

 

Step 7: All done!

Now you can compile your assembly and load it in the FxCop UI. Enjoy!

Tags:

.NET | C#

Reading IntelliTrace files (.iTrace)

by ingvar 2. november 2010 12:07

IntelliTrace is a really cool new feature in Visual Studio 2010. This feature gives you a lot of information that was not available before, like file I/O. But if you enable IntelliTrace and the file event, then you pretty quickly end up with a huge amount of events to look through in the IDE. But there is a pretty nice API for reading .iTrace files, so data mining these huge files is pretty easy.

Here is a rather simple method that reads the .iTrace file and goes through all diagnostic events, prints its category id and the stack trace including module name.

IntelliTraceFile file = new IntelliTraceFile("Sample.iTrace");

foreach (IntelliTraceProcess process in file.Processes)
{
    DiagnosticStreamChain chain = process.CreateProcessChain<DiagnosticStreamChain>();

    EventToken eventToken = chain.FirstValidToken;
    while (eventToken != chain.AfterLastToken)
    {
        IntelliTraceEvent intelliTraceEvent = chain.GetEvent(eventToken);

        DiagnosticEvent traceEvent = intelliTraceEvent as DiagnosticEvent;
        if (traceEvent != null)
        {
            ResolvedDiagnosticEvent diagnosticEvent =
                new ResolvedDiagnosticEvent(process, traceEvent);                       

            Console.WriteLine("Event: " + diagnosticEvent.CategoryId);

            var stackFrames = diagnosticEvent.DiagnosticEvent.StackFrames.Reverse();
            foreach (IntelliTraceStackFrame traceStackFrame in stackFrames)
            {
                ResolvedStackFrame stackFrame =
                    new ResolvedStackFrame(process, traceStackFrame);

                IntelliTraceModule module =
                    process.Modules.
                    Where(f => f.Mvid == stackFrame.Module.Mvid).
                    SingleOrDefault();

                string parameterNames = "";
                foreach (string parameterName in stackFrame.Method.ParameterNames)
                {
                    if (parameterNames.Length > 0)
                    {
                        parameterNames += ", ";
                    }

                    parameterNames += parameterName;
                }

                string stackFramePrint =
                    stackFrame.Method.ContainingTypeName + "." +
                    stackFrame.Method.MethodName + "(" + parameterNames + ")";

                if (module != null)
                {
                    stackFramePrint += " @ " + Path.GetFileName(module.FileName);
                }

                Console.WriteLine(stackFramePrint);
            }                        
        }

        eventToken = chain.GetNextToken(eventToken);
    }
}

I have not looked through all different data related to a diagnostic event (DiagnosticEvent.DataBytes). But here is an example of how to get the file name of the file that has been touched in a “file.access” diagnostic event.

if (diagnosticEvent.CategoryId == "file.access")
{
    UnicodeEncoding enc = new UnicodeEncoding();
    string filename = enc.GetString(diagnosticEvent.DiagnosticEvent.DataBytes.Skip(5).ToArray());
}

Tags:

.NET | C#

About the author

Martin Ingvar Kofoed Jensen

Architect and Senior Developer at Composite on the open source project Composite C1 - C#/4.0, LINQ, Azure, Parallel and much more!

Follow me on Twitter

Read more about me here.

Read press and buzz about my work and me here.

Stack Overflow

Month List