Jamie Maguire

Software Architect / Consultant / Developer / Microsoft AI MVP

Analytics and Big Data, C#, Cognitive Services, Instagram API, Machine Learning, Sentiment Analysis, Social Media, Tooling

Workshop Recap: Data Transformation with Azure Cognitive Services

Last week I was invited to help deliver a workshop alongside Grey Matter and Microsoft at the Microsoft Reading Campus.

The event was centred around data, how businesses can use AI to surface insights in data and how tools such as Power BI can be used by digital citizens to easily self-author rich reports and visualisations.

The day long workshop was split up into 4 sessions which consisted of:

  1. Introductions to the Azure Data Platform
  2. Introduction to Azure Cognitive Services
  3. Deep dives into Text Analytics API with C# demos
  4. Deep dives into Computer Vision API with C# demos
  5. Deep dives into Custom Vision API with C# demos
  6. Deep dives into Power BI

My sessions centred around how you can build interfaces in C# which connect to and extract data using the Instagram Graph API (points 2-5).

It followed by how you can then use Azure Cognitive Services Text Analytics and Computer Vision to extract further insights.

I extended an existing interface I had written which extracted over 10,000 images related to the search terms such as #microsoft and #xbox.

You can see Parts 2 and 3 of the Instagram mini-series I’ve been writing for further information about connecting to and extracting data from the Instagram Graph API.

***Spoiler Part 4 will be getting published soon! ***

Text Analytics API

I introduced Azure Text Analytics API, went through demos that highlighted the key features such as:

  • key phrase extraction
  • detecting the existence of entities
  • performing sentiment analysis

I also shared how I was able to swap out my custom API which was based on Bayesian Theorem and easily integrate the Cognitive Services API by leveraging the .NET SDK.

Computer Vision API

Computer Vision API was next and here I shared how you can provision this in Azure and use it to extract image related insights which included, but where not limited to:

  • Objects in an image
  • Auto generated human readable tags and descriptions for a given image
  • Existence of brands in an image

I showed how these insights can be extracted from images on your local file system or even online images by passing in a URL.  In the following screen shot, you can see the entire method code that’s needed to perform image classification:

public static async Task AnalyseImageLocalMachine(string localImage)
{
    ComputerVisionClient client = Authenticate("azure endpoint", "azure key");

    // the list of features we want to find in the image 
    List<VisualFeatureTypes> features = new List<VisualFeatureTypes>()
        {
            VisualFeatureTypes.Objects,
            VisualFeatureTypes.Categories, VisualFeatureTypes.Description,
            VisualFeatureTypes.Tags,
            VisualFeatureTypes.Color, VisualFeatureTypes.Brands
        };

    Console.WriteLine($"Analysing local image {Path.GetFileName(localImage)}...");
    
    using (Stream imageStream = File.OpenRead(localImage))
    {
        try
        {
            // Analyse the local image.
            ImageAnalysis results = results = await client.AnalyzeImageInStreamAsync(imageStream, features);
            
            // Summary of image content.
            Console.WriteLine("Summary:");
            foreach (var caption in results.Description.Captions)
            {
                Console.WriteLine($"{caption.Text} with confidence {caption.Confidence}");
            }
            Console.WriteLine();
            // Display categories the image is divided into.
            Console.WriteLine("Categories:");
            foreach (var category in results.Categories)
            {
                Console.WriteLine($"{category.Name} with confidence {category.Score}");
            }
            Console.WriteLine();
            // Image tags with confidence score.
            Console.WriteLine("Tags:");
            foreach (var tag in results.Tags)
            {
                Console.WriteLine($"{tag.Name} {tag.Confidence}");
            }
            Console.WriteLine();
            // Objects.
            Console.WriteLine("Objects:");
            foreach (var obj in results.Objects)
            {
                Console.WriteLine($"{obj.ObjectProperty} with confidence {obj.Confidence}");
            }
            Console.WriteLine();
            // detect Well-known brands
            Console.WriteLine("Brands:");
            foreach (var brand in results.Brands)
            {
                Console.WriteLine($"Logo of {brand.Name} with confidence {brand.Confidence}");
            }
            Console.WriteLine();

            // colours                 
            Console.WriteLine("Is black and white?: " + results.Color.IsBWImg);
            Console.WriteLine("Dominant colors: " + string.Join(",", results.Color.DominantColors));
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
    }
}

One example showed this image:

….the output of this when the following image is processed to the Computer Vision API:

Occasionally, there will be edge cases the Computer Vision API incorrectly classifies and that’s ok, it’s not a one size fits all.

I demonstrated this by giving a demo with a BMW car key and how the Computer Vision API classified it as a mobile phone!

That was OK though as it was a nice segue into the Custom Vision API!

Custom Vision API

In this final part of my session, I introduced the Custom Vision API. I ran through an overview of the site www.customvision.ai.

The Custom Vision AI dashboard lets you upload, label images and build an image classification model related to your unique edge case.

You can then use this to make predictions and publish them in the cloud for general consumption by your software or services.

I shared how we could leverage this to properly predict that a BMW car key was indeed a car key and how you can then consume the custom image classification model in C#.

This image below shows you the output of a Console Application that consumed the custom image model I created.  A car key being correctly predicted as a car key:

The Output

The end of my session resulted in a dataset which contained:

  • 10,029 images from Instagram
  • 167,322 key words and associated phrases
  • the sentiment of each caption for associated image
  • associated number of comments for each image
  • associated number of likes for each image
  • a discrete JSON file with Computer Vision API insights for each image

This data was then pre-processed and transformed using Azure Data Flow before being laid to rest in an Azure Data Lake for further analysis using Power BI.

Summary

The workshop went well with loads of great questions from attendees throughout the day. I also learned a fair bit myself as Power BI and some of the Azure data storage options and processing aren’t things I use regularly.

All sample code I ran through is on GitHub which you can find at the following URLS:

Text Analytics API – key phrase extraction, entity detection and sentiment analysis

Computer Vision API – image classification

Custom Vision API – used to consume custom trained image classification models

Finally, the event was recorded, and you can find my session here.

(It’s always hard listening back to yourself – not for vanity but to identify how you can do better next time!)

JOIN MY EXCLUSIVE EMAIL LIST
Get the latest content and code from the blog posts!
I respect your privacy. No spam. Ever.

Leave a Reply