ContentMD5 not working in the Storage Client library

by ingvar 28. juni 2011 16:33

Edit: 16th february

This bug was fixed in the Windows Azure SDK v. 1.5 or v. 1.6. So go download the newest version!

 

Edit: 6th july

This is a bug and it has been filed withint the team thats developing the Windows Azure Client Library. More information here.

 

Edit: 1st july - new version of this blog post:

I think my old version (see below) was not totally clear on how to reproduce what I was trying to say. So I have made a new code snip that will show that the ContentMD5 blob property is not populated by the Storage Client library event if the MD5 value is in the REST result. Below is the code that show it. To try the code, just copy/past it to a console application and change the account, storageKey and containerName to match your settings.

I should also add, that I'm not doing this to show of or talk down on Microsofts work with Windows Azure. I think all the Azure stuff is awesome!! It is just that, if this is a bug, and if it is fixed, it will make working with MD5 hash values on blobs a lot easier and faster. If I'm wrong I will take it all back :)

Here is the output of the code:

/md5test/MyBlob.txt MD5 =
MyBlob.txt MD5 = zhFORQHS9OLc6j4XtUbzOQ==

Here is the output of the code:

string account = "ACCOUNT NAME";
string storageKey = "STORAGE KEY";
string containerName = "md5test";

/* ListBlobs done by storage library */
StorageCredentialsAccountAndKey credentials =
    new StorageCredentialsAccountAndKey(
        account, 
        storageKey);
CloudStorageAccount storageAccount = new CloudStorageAccount(credentials, false);
CloudBlobClient client = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = client.GetContainerReference(containerName);
container.CreateIfNotExist();

string blobContent = "This is a test";
MD5 md5 = new MD5CryptoServiceProvider();
byte[] hashBytes = md5.ComputeHash(Encoding.UTF8.GetBytes(blobContent));
CloudBlob cloudBlob = container.GetBlobReference("MyBlob.txt");
cloudBlob.Properties.ContentMD5 = Convert.ToBase64String(hashBytes);
cloudBlob.UploadText(blobContent);

foreach (var blob in container.ListBlobs().OfType())
{
    Console.WriteLine(blob.Uri.LocalPath + " MD5 = " +
        blob.Properties.ContentMD5 ?? "Not populated");
}

/* ListBlobs done by REST */
DateTime now = DateTime.UtcNow;

string nowString = now.ToString("R", CultureInfo.InvariantCulture);

string canonicalizedResource = 
    string.Format("/{0}/{1}\ncomp:list\nrestype:container", 
    account, containerName);
string canonicalizedHeaders = 
    string.Format("x-ms-date:{0}\nx-ms-version:2009-09-19", 
    nowString);

string messageSignature =
    string.Format("GET\n\n\n\n\n\n\n\n\n\n\n\n{0}\n{1}",
    canonicalizedHeaders,
    canonicalizedResource

);

byte[] SignatureBytes = Encoding.UTF8.GetBytes(messageSignature);
HMACSHA256 SHA256 = new HMACSHA256(Convert.FromBase64String(storageKey));
String authorizationHeader = 
    string.Format("SharedKey {0}:{1}", 
    account, 
    Convert.ToBase64String(SHA256.ComputeHash(SignatureBytes)));


Uri url = new Uri(
   string.Format("http://{0}.blob.core.windows.net/{1}?restype=container&comp=list",
   account, containerName));
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
request.ContentLength = 0;
request.Headers.Add("x-ms-date", nowString);
request.Headers.Add("x-ms-version", "2009-09-19");
request.Headers.Add("Authorization", authorizationHeader);

HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);

string responseString = reader.ReadToEnd();

reader.Close();
responseStream.Close();
response.Close();

XElement responseElement = XElement.Parse(responseString);

foreach (XElement element in responseElement.Descendants("Content-MD5"))
{
    string name = element.Parent.Parent.Element("Name").Value;
    string md5value = element.Value;

    if (string.IsNullOrEmpty(md5value))
    {
        md5value = "Not populated";
    }

    Console.WriteLine(name + " MD5 = " + md5value);
}

 

 

 

Original version of this blog post

This is a follow up on my other blob post about the ContentMD5 blob property (Why is MD5 not part of ListBlobs result?). And it is motivated by the comment from Gaurav Mantri stating that the MD5 is actually in the REST response, it is just not populated by the Storage Client library.

So I also did some more testing and I found out that the MD5 is calculated by the and set blob stroge when you upload the blob. Because when I did a simple test of creating a new container and uploading a simple 'Hello World' txt file and tried doing a ListBlobs through the REST API, I got the XML back below. Note that I did not populate the ContentMD5 when I uploaded the file! If I try doing a ListBlobs method call on a CloudBlobContainer instance the ContentMD5 is NOT populated (Read more about this in my other blog post). So my best guess is that this is a error in the API of some sort.

I would be very nice if this was fixed, so we did not have to compuste the MD5 hash by hand and adding it to the blobs metadata. I would make code like I did for the local folder to/from blob storage synchronization much simpler (Read it here). 


<EnumerationResults ContainerName="http://myblobtest.blob.core.windows.net/md5test">
 <Blobs>
  <Blob>
   <Name>MyTestBlob.xml</Name>
   <Url>http://myblobtest.blob.core.windows.net/md5test/MyTestBlob.xml</Url>
   <Properties>
   <Last-Modified>Tue, 28 Jun 2011 16:20:30 GMT</Last-Modified>
   <Etag>0x8CE038BC293943D</Etag>
   <Content-Length>12</Content-Length>
   <Content-Type>text/xml</Content-Type>
   <Content-Encoding/>
   <Content-Language/>
   <Content-MD5>LL0183lu9Ms2aq/eo4TesA==</Content-MD5>
   <Cache-Control/>
   <BlobType>BlockBlob</BlobType>
   <LeaseStatus>unlocked</LeaseStatus>
  </Properties>
  </Blob>
 </Blobs>
 <NextMarker/>
</EnumerationResults>

 

Here is the code I did for doing the REST call:

string requestUrl = 
"http://myblobtest.blob.core.windows.net/md5test?restype=container&comp=list"; string certificatePath = /* Path to certificate */ HttpWebRequest httpWebRequest =
(HttpWebRequest)HttpWebRequest.Create(new Uri(requestUrl, true)); httpWebRequest.ClientCertificates.Add(new X509Certificate2(certificatePath)); httpWebRequest.Headers.Add("x-ms-version", "2009-09-19"); httpWebRequest.ContentType = "application/xml"; httpWebRequest.ContentLength = 0; httpWebRequest.Method = "GET"; using (HttpWebResponse httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse()) { using (Stream responseStream = httpWebResponse.GetResponseStream()) { using (StreamReader reader = new StreamReader(responseStream)) { Console.WriteLine(reader.ReadToEnd()); } } }

Tags:

.NET | Azure | Blob

Comments (7) -

Neil Mackenzie
Neil Mackenzie United States
29-06-2011 01:29:54 #

This Windows Azure Storage Team post has an explanation of when the MD5 attribute is set:

blogs.msdn.com/.../...azure-blob-md5-overview.aspx

ingvar
ingvar Denmark
01-07-2011 09:39:53 #

Interesting. I Will redo my tests and show the code for how I upload the blob.

But, I still find it strange that if I do a "ListBlobs" through the REST API, then I get the MD5 value. But if I use the ListsBlobs on a container I do not get the MD5 value.

Also, Gaurav Mantri tried using Fiddler when calling ListBlobs on a container and the REST response had a MD5 value, but the ContentMD5 was still not populated.

Gaurav Mantri
Gaurav Mantri India
01-07-2011 18:18:31 #

@Neil My guess is that it is because of some mistake in Storage Client library. If I call list blobs using Storage Client library and see the response returned by the storage service in Fiddler, I see content-md5 property is being set on the blob and I can see that value in response XML. However when I look up ContentMD5 property of the blob it is not set. Also other properties (like ContentType) is set properly for the blob object.

I even tried to find the code in StorageClient library which is actually parsing the XML response and setting up the properties but the code is so convoluted, I was not able to figure that out.

ingvar
ingvar Denmark
01-07-2011 23:59:33 #

@Neil Mackenzie: To further illustrate it I have added the full code for both upload through the storage API and the REST ListBlobs code. Running this code in a console application shows that the ContentMD5 is not populated.

@Gaurav Mantri: Yep, correct. Again, nice using Fiddler. I also tried Fiddler with the same result as you.

Neil Mackenzie
Neil Mackenzie United States
02-07-2011 00:10:17 #

It looks like a bug. You should post it on the MSDN Forum so it gets visibility inside Microsoft.

ingvar
ingvar Denmark
02-07-2011 00:41:14 #

@Neil. Thanks, done! http://bit.ly/jh6wMV

ingvar
ingvar Denmark
16-02-2012 06:56:19 #

The bug has been fixed!! Thank you all for the input! Smile

About the author

Martin Ingvar Kofoed Jensen

Architect and Senior Developer at Composite on the open source project Composite C1 - C#/4.0, LINQ, Azure, Parallel and much more!

Follow me on Twitter

Read more about me here.

Read press and buzz about my work and me here.

Stack Overflow

Month List