In recent years, artificial intelligence algorithms have become very good at understanding what’s in pictures and videos. In some cases, they’re even better than humans at recognizing particular types of objects in images. Computer vision can be tremendously helpful when you need to do things like extract text from images and find interesting events in videos.

Microsoft makes it easy for you to add computer vision capabilities to your applications using the Azure AI Vision service, which doesn’t require any knowledge of machine learning. Azure AI Vision is actually a collection of services, including:

Image Analysis
Optical Character Recognition
Face, and
Spatial Analysis

I’ll show you all of them, starting with image analysis, but there’s something we need to do first. To use the AI Vision service, you can either create an AI Vision resource or a more general AI services resource. I prefer to create an AI services resource because you can also use it for other AI services besides AI Vision, such as AI Language.

In the Azure portal, search for “AI services”. Select the one that says, “multi-service account” so it can be used for multiple AI services. Then click “Create”. Create a new resource group called “ai-services”. The instance name has to be globally unique, so I’ll call mine “ca-ai”. If you’re doing this yourself, you’ll have to call yours something different. For the pricing tier, choose “Standard S0”. Then check the box saying that you’ve read the terms below.

Click “Review + create” and “Create”. Then click “Go to resource”.

You can see that there are many different categories of AI services, such as Decision, Language, and Speech. Choose “Vision”. This just has links to the documentation. That’s because the normal way to use AI services is to write code that calls one of its APIs (or Application Programming Interfaces).

If you go into the Computer Vision documentation and click on “Get started with Image Analysis”, you’ll see that there’s documentation for different programming languages, such as C# and Python.

Fortunately, there’s an easier way to try out the AI Vision service. You can use the Vision Studio, which is a web-based interface to the AI Vision API. It’s at this URL.

Let’s see what image analysis tasks we can perform. One of the most popular is “Detect common objects in images”, also known as object detection. First, you have to select your AI services resource, and then check the "Try it out" box.

If you choose one of the sample images, you can see what it does. It comes back with a list of things it detected in the image, such as footwear, person, and laptop. It also puts what are called “bounding boxes” around the objects. If you put your mouse pointer over a box, it’ll show you which object it thinks it is.

You’ll also notice that there’s a percentage next to each object in the list. This is the AI model’s confidence that it has identified the object correctly. For example, it’s only 52% confident about the footwear.

Sometimes its guesses are wrong, so you might only want to see objects that have a high confidence level.

You can remove low-confidence items from the list by increasing the threshold value. If you increase it to 55, then the list will only include objects that have at least a 55% confidence level. So, items such as footwear are no longer on the list.

If you have an application that needs to know the coordinates of each bounding box, then you can get those from the JSON response to the API call. The x and y coordinates are for the top left corner of the box. W is the width, and h is the height.

Microsoft’s prebuilt image analysis model is quite good at detecting a wide variety of common objects, but what if you need to detect very specific objects, such as different species of birds? Then you can train your own custom model.

First, you need to decide whether you want a model that will find various objects within an image or classify an entire image. The example we just saw is object detection because it came back with a list of objects in an image. With image classification, it would come back with a single category for an image, such as “people” or “meeting”, depending on what categories you define.

Then you need to find some examples of images that contain the objects or classifications that you want the model to detect. Next, you need to label all of the images. If you’re going to train an image classification model, then you just have to label each image with a description, but if you’re going to train an object detection model, then you’ll need to specify bounding boxes for each object in each image. To make this easier, you can create a labeling project in Azure Machine Learning.

Once you have some labeled images, you can train your custom model. Then you can evaluate the model by feeding it some unlabeled images and seeing how well it performs. If it isn’t as accurate as you’d like, then you can train the model with more labeled images than the first time.

Okay, going back to the types of image analysis tasks that AI Vision can perform, here are a couple more. You can add captions to images, which means it will come back with a description of each image, such as “A baseball player holding a bat”.

You can also extract common tags from images, which is similar to object detection, but it can generate multiple tags for the same object, such as “laptop” and “computer”, as well as give non-object descriptions, such as “indoor” and “sitting”. You can add these tags to the image as metadata, which would make it easier to search for specific types of images.

As I mentioned earlier, the Vision Studio is mostly meant to give you an easy way to try out the AI Vision API, but at some point, you’ll likely need to call the API from your applications, which means you (or someone on your team), will need to write code. When you do, be aware that there are different versions of the API, and they have different features. For example, version 3.2 has features that will detect landmarks, celebrities, or brands in images, but at the moment, those features are not available in version 4.0.

Okay, next, let’s try Optical Character Recognition. This is a pretty straightforward service. It lets you extract text from images. It is quite good at reading text, even if it’s handwritten or at an angle.

Here’s the JSON response from the API. It includes each piece of text and the associated bounding polygon, which is like a bounding box except that all four corners are specified since the box is at an angle.

Spatial analysis is different because it analyzes videos rather than images. You can get it to do things like count the number of people in an area at different points in the video. Video Retrieval and Summary is much more sophisticated. It will analyze a video and look for noteworthy events.

For example, the summary for this video says that someone spilled liquid on the floor, an employee was seen mopping the floor in response to the spill, and a woman was shown falling on the ground. You can jump to one of those events in the video by selecting it from this dropdown. You’ll notice that there are actually multiple events when liquid was spilled on the floor. One time, it was milk, and another time, it was coffee. The clumsy customers in this store must really keep this guy busy.

Note that Microsoft also has another service called Azure AI Video Indexer that performs some of the same functions. I’m assuming that these two services will eventually be merged.