Azure Blob Storage: block blob uploads

Azure Blob Storage is a very efficient file stoker service that can be used as a content delivery network (CDN). If you store large files, it allows you to upload them by block. In this article we will see how to upload a file into multiple blocks.

Uploaded

Lupload of block file is done in 3 steps:

• Calculation of file md5
• Send blocks of data
• Send the block list to complete the upload

Calculation of file md5

The calculation of the md5 hash of the file is very simple: the entire file is stored as byte array Then we calculate the associated md5.

 var bytes = await File.ReadAllBytesAsync(path, cancellationToken).ConfigureAwait(false);

var fileMD5 = Md5Helper.GetMD5String(bytes);

Below is the code to calculate the MD5 hash:

    public static class Md5Helper
    {
        public static string GetMD5String(byte[] data)
        {
            byte[] result = null;
            using (MD5 md5 = new MD5CryptoServiceProvider())
            {
                result = md5.ComputeHash(data);
            }
            return Convert.ToBase64String(result);
        }
    }

Note: blob storage APIs only accept the basic form 64 of the Md5 hash

Send blocks of data

Lupload is quite simple. Just read the block file of a given size. In our example we'll read the file by block of 10 megas.

int blockSize = 10 * 1024 * 1024; //10 méga octets
var blockIds = new List<string>();
using (var reader = new BinaryReader(File.Open(path, FileMode.Open)))
{
  int blockId = 0;

  while (reader.BaseStream.Position != reader.BaseStream.Length)
  {
    var block = reader.ReadBytes(blockSize);
    blockIds.Add(await UploadBlockAsync(block, blobName, blockId).ConfigureAwait(false));
    blockId++;
  }

The UploadBlockAsync function sends the blocks of data to blob storage. For each block we will calculate the md5 hash of the block that we will send with the block, which will allow the blob storage to validate that the data is not corrupted during the upload.

public async Task<string> UploadBlockAsync(byte[] blockData, string blobName, int blockId, CancellationToken cancellationToken = default)
        {

            var options = new BlobRequestOptions
            {
                DisableContentMD5Validation = false,
                ServerTimeout = TimeSpan.FromMinutes(30),
                SingleBlobUploadThresholdInBytes = blockSize,
                ParallelOperationThreadCount = 1,
                RetryPolicy = new ExponentialRetry(TimeSpan.Zero, 3),
                UseTransactionalMD5 = true,
                StoreBlobContentMD5 = true,
            };
            // l'identifiant de chaque block doit être encodé en base64 sinon l'api blob storage refuse le block
            var blockIdString = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId.ToString("d7")));

            cancellationToken.ThrowIfCancellationRequested();
            var md5Hash = Md5Helper.GetMD5String(blockData);

            var blockBlob = await GetBlockBlobRefenceAsync(blobName);

            cancellationToken.ThrowIfCancellationRequested();
            await blockBlob.PutBlockAsync(
                                        blockIdString,
                                        new MemoryStream(blockData, true),
                                        md5Hash,
                                        null,
                                        options,
                                        null);

            cancellationToken.ThrowIfCancellationRequested();
            Console.WriteLine(blockId);
            return blockIdString;
        }

Send the block list to complete the upload

The last step to verify that the file is sent correctly is to send the list of identifiers for each uploaded block.

Note: If you don't take this step, your blob’is not considered complete, and after some time it will be automatically deleted.

var options = new BlobRequestOptions
            {
                DisableContentMD5Validation = false,
                ServerTimeout = TimeSpan.FromMinutes(30),
                ParallelOperationThreadCount = 1,
                RetryPolicy = new ExponentialRetry(TimeSpan.Zero, 3),
                UseTransactionalMD5 = true,
                StoreBlobContentMD5 = true,
            };

            cancellationToken.ThrowIfCancellationRequested();

            var blockBlob = await GetBlockBlobRefenceAsync(blobName);

            cancellationToken.ThrowIfCancellationRequested();
            blockBlob.Properties.ContentMD5 = md5;
            await blockBlob.PutBlockListAsync(blockIds, null, options, null);

And here you are, uploading a block by block file to Azure's blob storage!

Happy coding.