Why is MD5 not part of ListBlobs result?

by ingvar 26. June 2011 20:13

Edit 6th july:

It turns out that this is a bug in the Windows Azure Client Library. Read more here.


Original version:

For some strange reason the ContentMD5 blob property is only populated if you call FetchAttributes on the blob. This means that you can not obtain the value of the ContentMD5 blob property by calling ListBlobs, with any parameter settings. If you have to call FetchAttributes to obtain the value it is not feasible to use on a large set of blobs because of the request time for calling FetchAttributes is going to take a long time. This means that it can not be used in a optimal way when doing something like fast folder synchronization I wrote about in this blog post.

The ContentMD5 blob property is never set by the blob storage it self. And if you try to put any value in it that is not a correct 128 MD5 hash of the file, you will get an exception. So it has limited usage.

I have made some code to illustrate the missing population of the ContentMD5 blob property when using the listBlobs method. 

Here is the output of running the code:

GetBlobReference without FetchAttributes: Not populated
GetBlobReference with FetchAttributes: zhFORQHS9OLc6j4XtUbzOQ==
ListBlobs with BlobListingDetails.None: Not populated
ListBlobs with BlobListingDetails.Metadata: Not populated
ListBlobs with BlobListingDetails.All: Not populated

And here is the code:

CloudStorageAccount storageAccount = /* Initialize this */;
CloudBlobClient client = storageAccount.CreateCloudBlobClient();

CloudBlobContainer container = client.GetContainerReference("mytest");

const string blobContent = "This is a test";

/* Compute 128 MD5 hash of content */
MD5 md5 = new MD5CryptoServiceProvider();
byte[] hashBytes = md5.ComputeHash(Encoding.UTF8.GetBytes(blobContent));

/* Get the blob reference and upload it with MD5 hash */
CloudBlob blob = container.GetBlobReference("MyBlob1.txt");
blob.Properties.ContentMD5 = Convert.ToBase64String(hashBytes);

CloudBlob blob1 = container.GetBlobReference("MyBlob1.txt");

/* Not populated - this is expected */
Console.WriteLine("GetBlobReference without FetchAttributes: " + 
    (blob1.Properties.ContentMD5 ?? "Not populated"));


/* Populated - this is expected */
Console.WriteLine("GetBlobReference with FetchAttributes: " + 
    (blob1.Properties.ContentMD5 ?? "Not populated"));

CloudBlob blob2 = container.ListBlobs().OfType().Single();
/* Not populated - this is NOT expected */
Console.WriteLine("ListBlobs with BlobListingDetails.None: " + 
    (blob2.Properties.ContentMD5 ?? "Not populated"));

CloudBlob blob3 = container.ListBlobs(new BlobRequestOptions
    { BlobListingDetails = BlobListingDetails.Metadata }).

/* Not populated - this is NOT expected */
Console.WriteLine("ListBlobs with BlobListingDetails.Metadata: " + 
    (blob3.Properties.ContentMD5 ?? "Not populated"));

CloudBlob blob4 = container.ListBlobs(new BlobRequestOptions
        UseFlatBlobListing = true,
        BlobListingDetails = BlobListingDetails.All

/* Not populated - this is NOT expected */
Console.WriteLine("ListBlobs with BlobListingDetails.All: " + 
    (blob4.Properties.ContentMD5 ?? "Not populated"));


.NET | Azure | Blob | C#

Comments (3) -

Gaurav Mantri
Gaurav Mantri India
6/28/2011 3:43:14 AM #

It is really interesting. I almost didn't believe you Smile so I ran a test and found out that it is indeed not being returned. What is interesting is that you get all other properties back (at least I tested content type property). What is more interesting is that the storage service does return you blob's content MD5 in the response back (I checked it in Fiddler) but it seems Storage Client library just decides not to populate ContentMD5 member in blob.Properties.



ingvar United States
6/28/2011 5:56:58 AM #

@Gaurav Mantri. Nice one using Fiddler! Should have thought of that my self ;) It seems more like a bug in the Storage Client library now.

ingvar Denmark
6/28/2011 4:52:23 PM #

@Gaurav Mantri. Did some more invistegation and confirmed your findings and a little more. toolheaven.net/.../...-Storage-Client-library.aspx

Pingbacks and trackbacks (1)+

About the author

Martin Ingvar Kofoed Jensen

Architect and Senior Developer at Composite on the open source project Composite C1 - C#/4.0, LINQ, Azure, Parallel and much more!

Follow me on Twitter

Read more about me here.

Read press and buzz about my work and me here.

Stack Overflow

Month List