You have to admit, Microsoft Cognitive Services has a lot of really nice offerings to bring to your applications. One of the newer ones is the Video Indexer. This a powerful service since when you upload your videos it analyzes them to extract several pieces of analytics, including finding people in the video and attempts to find out who they are, gives speech sentiment, and extracts keywords. These analytics are throughout the whole video so you can, for example, what parts of the video give a negative sentiment and which are positive.
There are actually two ways to use Video Indexer, through its own portal and through the APIs. In this post, we will examine how to use the portal to upload and examine a video to see what insights we can get from the video. For the video we’ll use the Introduction to ML Services, Part 1 of 4 video looks like a good one to upload to Video Indexer.
Uploading a Video
Once you’re at the main portal page click on “Get started”. If you have a Microsoft account go ahead and log in. It should take you to a page that looks similar to this:
To upload you can either select a file or enter a video by URL. Note that the URL would need to be a direct link to the video.
Give your video a name and feel free to keep the defaults. The last row of drop downs are some settings you can set to determine what language the video is in, what preset or whether to use audio only or just noise reduction, and whether the analysis is private to your account or available to the public.
Click upload, and you’ll be taken back to the main page and given a progress of analysis of your video. Keep in mind this can take several minutes depending on the length of the video. Once it’s completed analyzing, you can click on it to go into the details.
Examining Video Analysis
Now comes the fun part. Once we get to the video details we can learn all sorts of things. On the left side is the video and the right side has all of our analysis.
The first part is the people that Video Indexer notices throughout the video. It does its best to do facial analysis and detect who the person is. For instance, it found that the person on the right is Seth Juarez, who is a common host on Channel 9 videos. Video Index can tell what percentage of the video that person appears in. Also, take note of the timeline near the appearance percentage. The black portions of it are where Video Indexer determined that’s where in the video the person is in.
Below that are the keywords Video Indexer found during the video. You can click on the left or right arrows to go to each occurrence in the video where the keyword was mentioned.
Scrolling down on the right side, we’ll find more analysis. We find annotations, which seems really interesting. If you click on each occurrence of “person”, you will most likely catch a glimpse of a person in the background of the video. So Video Indexer annotations different elements of the video.
Below that, we have brands, which highlights mentions of different well-known brands, such as companies or software in our case. It also highlights in the video where these brands are mentioned.
And finally, it gives us sentiment analysis of the entire video. Most of this video was neutral, but 22% of it was positive, and there was 0.5% of the video that had a negative sentiment. Here we are given a color-coded timeline to see what type of sentiment happens during the video.
Other than analysis on the video, we are also given an automatic transcript of the audio.
In fact, you actually get a transcript of more than just the audio. Above where the transcript is displayed there’s a search. If we search for “ai platform” we’ll get parts where it was mentioned in the audio, but also it reads any text on the screen and puts them into the transcript.
One other thing we can do with transcripts is that we can very easily translate them into other languages. Next to the search you can see “English”, but it is a drop down. There are quite a few languages there and just look at how quickly it translates the transcript.
Editing Video Indexer Results
The analyzing that Video Indexer does, as good as it is, doesn’t get everything correct. So the portal offers us a way to edit transcripts and people.
For instance, going back to our person analysis, we can see that Seth was picked up twice and the second time Video Indexer couldn’t tell it was him. To edit, switch the “Edit” slider and click on the one that has him as “Unknown”. Then click on the pencil icon and add his name to it and hit “Enter”.
You may have to refresh to see the changes, but now you can see the edit took effect and Seth is now in 49% of the video.
We can do the same for any part of the transcript that may not have been transcribed correctly. Check the “Edit” switch to allow editing and click on the text you want to update and once finished hit the check mark in the corner of the text.
In this post we went over what Video Indexer gives you and a walk through different aspects of the portal. With one service you can get a lot of information about your videos, whether the sentiment, the people in the video, or even just to get a transcript, the Video Indexer service has a lot going for it.