Computer Vision API (v3.2-preview.3)

The Computer Vision API provides state-of-the-art algorithms to process images and return information. For example, it can be used to determine if an image contains mature content, or it can be used to find all the faces in an image. It also has other features like estimating dominant and accent colors, categorizing the content of images, and describing an image with complete English sentences. Additionally, it can also intelligently generate images thumbnails for displaying large images effectively.

This API is currently available in:

  • Australia East - australiaeast.api.cognitive.microsoft.com
  • Brazil South - brazilsouth.api.cognitive.microsoft.com
  • Canada Central - canadacentral.api.cognitive.microsoft.com
  • Central India - centralindia.api.cognitive.microsoft.com
  • Central US - centralus.api.cognitive.microsoft.com
  • East Asia - eastasia.api.cognitive.microsoft.com
  • East US - eastus.api.cognitive.microsoft.com
  • East US 2 - eastus2.api.cognitive.microsoft.com
  • France Central - francecentral.api.cognitive.microsoft.com
  • Japan East - japaneast.api.cognitive.microsoft.com
  • Japan West - japanwest.api.cognitive.microsoft.com
  • Korea Central - koreacentral.api.cognitive.microsoft.com
  • North Central US - northcentralus.api.cognitive.microsoft.com
  • North Europe - northeurope.api.cognitive.microsoft.com
  • South Africa North - southafricanorth.api.cognitive.microsoft.com
  • South Central US - southcentralus.api.cognitive.microsoft.com
  • Southeast Asia - southeastasia.api.cognitive.microsoft.com
  • UK South - uksouth.api.cognitive.microsoft.com
  • West Central US - westcentralus.api.cognitive.microsoft.com
  • West Europe - westeurope.api.cognitive.microsoft.com
  • West US - westus.api.cognitive.microsoft.com
  • West US 2 - westus2.api.cognitive.microsoft.com

Analyze Image

This operation extracts a rich set of visual features based on the image content.

Two input methods are supported -- (1) Uploading an image or (2) specifying an image URL. Within your request, there is an optional parameter to allow you to choose which features to return. By default, image categories are returned in the response.

A successful response will be returned in JSON. If the request failed, the response will contain an error code and a message to help understand what went wrong.

Http Method

POST

Select the testing console in the region where you created your resource:

West US West US 2 East US East US 2 West Central US South Central US West Europe North Europe Southeast Asia East Asia Australia East Brazil South Canada Central Central India UK South Japan East Central US France Central Korea Central Japan West North Central US South Africa North UAE North Norway East West US 3 Jio India West

Request URL

Request parameters

(optional)
string

A string indicating what visual feature types to return. Multiple values should be comma-separated.
Valid visual feature types include:

  • Adult - detects if the image is pornographic in nature (depicts nudity or a sex act), or is gory (depicts extreme violence or blood). Sexually suggestive content (aka racy content) is also detected.
  • Brands - detects various brands within an image, including the approximate location. The Brands argument is only available in English.
  • Categories - categorizes image content according to a taxonomy defined in documentation.
  • Color - determines the accent color, dominant color, and whether an image is black&white.
  • Description - describes the image content with a complete sentence in supported languages.
  • Faces - detects if faces are present. If present, generate coordinates, gender and age.
  • ImageType - detects if image is clipart or a line drawing.
  • Objects - detects various objects within an image, including the approximate location. The Objects argument is only available in English.
  • Tags - tags the image with a detailed list of words related to the image content.
(optional)
string

A string indicating which domain-specific details to return. Multiple values should be comma-separated.
Valid visual feature types include:

  • Celebrities - identifies celebrities if detected in the image.
  • Landmarks - identifies landmarks if detected in the image.
(optional)
string

A string indicating which language to return. The service will return recognition results in specified language. If this parameter is not specified, the default value is "en".
Supported languages:

  • en - English, Default.
  • es - Spanish.
  • ja - Japanese.
  • pt - Portuguese.
  • zh - Simplified Chinese.

Request headers

string
Media type of the body sent to the API.
string
Subscription key which provides access to this API. Found in your Cognitive Services accounts.

Request body

Input passed within the POST body. Supported input methods: raw image binary or image URL.

Input requirements:

  • Supported image formats: JPEG, PNG, GIF, BMP.
  • Image file size must be less than 4MB.
  • Image dimensions must be at least 50 x 50.

{"url":"http://example.com/images/test.jpg"}
[Binary image data]
[Binary image data]

Response 200

The response include the extracted features in JSON format.

Here is the definitions for enumeration types
ClipartType

  • Non-clipart = 0,
  • ambiguous = 1,
  • normal-clipart = 2,
  • good-clipart = 3.

LineDrawingType
  • Non-LineDrawing = 0,
  • LineDrawing = 1.

{
  "categories": [
    {
      "name": "abstract_",
      "score": 0.00390625
    },
    {
      "name": "people_",
      "score": 0.83984375,
      "detail": {
        "celebrities": [
          {
            "name": "Satya Nadella",
            "faceRectangle": {
              "left": 597,
              "top": 162,
              "width": 248,
              "height": 248
            },
            "confidence": 0.999028444
          }
        ],
        "landmarks":[
          {
            "name":"Forbidden City",
            "confidence": 0.9978346
          }
        ]
      }
    }
  ],
  "adult": {
    "isAdultContent": false,
    "isRacyContent": false,
    "isGoryContent": false,
    "adultScore": 0.0934349000453949,
    "racyScore": 0.068613491952419281,
    "goreScore": 0.08928389008070282
  },
  "tags": [
    {
      "name": "person",
      "confidence": 0.98979085683822632
    },
    {
      "name": "man",
      "confidence": 0.94493889808654785
    },
    {
      "name": "outdoor",
      "confidence": 0.938492476940155
    },
    {
      "name": "window",
      "confidence": 0.89513939619064331
    }
  ],
  "description": {
    "tags": [
      "person",
      "man",
      "outdoor",
      "window",
      "glasses"
    ],
    "captions": [
      {
        "text": "Satya Nadella sitting on a bench",
        "confidence": 0.48293603002174407
      }
    ]
  },
  "requestId": "0dbec5ad-a3d3-4f7e-96b4-dfd57efe967d",
  "metadata": {
    "width": 1500,
    "height": 1000,
    "format": "Jpeg"
  },
  "faces": [
    {
      "age": 44,
      "gender": "Male",
      "faceRectangle": {
        "left": 593,
        "top": 160,
        "width": 250,
        "height": 250
      }
    }
  ],
  "color": {
    "dominantColorForeground": "Brown",
    "dominantColorBackground": "Brown",
    "dominantColors": [
      "Brown",
      "Black"
    ],
    "accentColor": "873B59",
    "isBWImg": false
  },
  "imageType": {
    "clipArtType": 0,
    "lineDrawingType": 0
  },
  "objects": [
    {
      "rectangle": {
        "x": 25,
        "y": 43,
        "w": 172,
        "h": 140
      },
      "object": "person",
      "confidence": 0.931
    }
  ]
}

Response 400

Possible Errors:

  • InvalidImageUrl
    Image URL is badly formatted or not accessible.
  • InvalidImageFormat
    Input data is not a valid image.
  • InvalidImageSize
    Input image is too large.
  • NotSupportedVisualFeature
    Specified feature type is not valid.
  • NotSupportedImage
    Unsupported image, e.g. child pornography.
  • InvalidDetails
    Unsupported domain-specific model.
  • NotSupportedLanguage
    The requested operation is not supported in the language specified.
  • BadArgument
    Additional details are provided in the error message.

{
	"code":"InvalidImageFormat",
	"requestId":"B8D802CF-DD8F-4E61-B15C-9E6C5844CCBC",
	"message":"The input file is not in a valid image format that the service can support. "
}

Response 415

Unsupported media type error. Content-Type is not in the allowed types:

  • For an image URL: Content-Type should be application/json
  • For a binary image data: Content-Type should be application/octet-stream or multipart/form-data

{
        "code":"BadArgument",
        "message":"Invalid Media Type"
}

Response 500

Possible Errors:

  • FailedToProcess
    Failed to process the image.
  • Timeout
    Image processing time out.
  • InternalServerError
    Internal server error.

{
	"code":"FailedToProcess",
	"requestId":"B8D802CF-DD8F-4E61-B15C-9E6C5844CCBC",
	"message":"Could not extract image features"
}

Code samples

@ECHO OFF

curl -v -X POST "https://switzerlandwest.api.cognitive.microsoft.com/vision/v3.2-preview.3/analyze?visualFeatures=Categories&details={string}&language=en"
-H "Content-Type: application/json"
-H "Ocp-Apim-Subscription-Key: {subscription key}"

--data-ascii "{body}" 

using System;
using System.Net.Http.Headers;
using System.Text;
using System.Net.Http;
using System.Web;

namespace CSHttpClientSample
{
    static class Program
    {
        static void Main()
        {
            MakeRequest();
            Console.WriteLine("Hit ENTER to exit...");
            Console.ReadLine();
        }
        
        static async void MakeRequest()
        {
            var client = new HttpClient();
            var queryString = HttpUtility.ParseQueryString(string.Empty);

            // Request headers
            client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "{subscription key}");

            // Request parameters
            queryString["visualFeatures"] = "Categories";
            queryString["details"] = "{string}";
            queryString["language"] = "en";
            var uri = "https://switzerlandwest.api.cognitive.microsoft.com/vision/v3.2-preview.3/analyze?" + queryString;

            HttpResponseMessage response;

            // Request body
            byte[] byteData = Encoding.UTF8.GetBytes("{body}");

            using (var content = new ByteArrayContent(byteData))
            {
               content.Headers.ContentType = new MediaTypeHeaderValue("< your content type, i.e. application/json >");
               response = await client.PostAsync(uri, content);
            }

        }
    }
}	
// // This sample uses the Apache HTTP client from HTTP Components (http://hc.apache.org/httpcomponents-client-ga/)
import java.net.URI;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

public class JavaSample 
{
    public static void main(String[] args) 
    {
        HttpClient httpclient = HttpClients.createDefault();

        try
        {
            URIBuilder builder = new URIBuilder("https://switzerlandwest.api.cognitive.microsoft.com/vision/v3.2-preview.3/analyze");

            builder.setParameter("visualFeatures", "Categories");
            builder.setParameter("details", "{string}");
            builder.setParameter("language", "en");

            URI uri = builder.build();
            HttpPost request = new HttpPost(uri);
            request.setHeader("Content-Type", "application/json");
            request.setHeader("Ocp-Apim-Subscription-Key", "{subscription key}");


            // Request body
            StringEntity reqEntity = new StringEntity("{body}");
            request.setEntity(reqEntity);

            HttpResponse response = httpclient.execute(request);
            HttpEntity entity = response.getEntity();

            if (entity != null) 
            {
                System.out.println(EntityUtils.toString(entity));
            }
        }
        catch (Exception e)
        {
            System.out.println(e.getMessage());
        }
    }
}


<!DOCTYPE html>
<html>
<head>
    <title>JSSample</title>
    <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"></script>
</head>
<body>

<script type="text/javascript">
    $(function() {
        var params = {
            // Request parameters
            "visualFeatures": "Categories",
            "details": "{string}",
            "language": "en",
        };
      
        $.ajax({
            url: "https://switzerlandwest.api.cognitive.microsoft.com/vision/v3.2-preview.3/analyze?" + $.param(params),
            beforeSend: function(xhrObj){
                // Request headers
                xhrObj.setRequestHeader("Content-Type","application/json");
                xhrObj.setRequestHeader("Ocp-Apim-Subscription-Key","{subscription key}");
            },
            type: "POST",
            // Request body
            data: "{body}",
        })
        .done(function(data) {
            alert("success");
        })
        .fail(function() {
            alert("error");
        });
    });
</script>
</body>
</html>

#import <Foundation/Foundation.h>

int main(int argc, const char * argv[])
{
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
    
    NSString* path = @"https://switzerlandwest.api.cognitive.microsoft.com/vision/v3.2-preview.3/analyze";
    NSArray* array = @[
                         // Request parameters
                         @"entities=true",
                         @"visualFeatures=Categories",
                         @"details={string}",
                         @"language=en",
                      ];
    
    NSString* string = [array componentsJoinedByString:@"&"];
    path = [path stringByAppendingFormat:@"?%@", string];

    NSLog(@"%@", path);

    NSMutableURLRequest* _request = [NSMutableURLRequest requestWithURL:[NSURL URLWithString:path]];
    [_request setHTTPMethod:@"POST"];
    // Request headers
    [_request setValue:@"application/json" forHTTPHeaderField:@"Content-Type"];
    [_request setValue:@"{subscription key}" forHTTPHeaderField:@"Ocp-Apim-Subscription-Key"];
    // Request body
    [_request setHTTPBody:[@"{body}" dataUsingEncoding:NSUTF8StringEncoding]];
    
    NSURLResponse *response = nil;
    NSError *error = nil;
    NSData* _connectionData = [NSURLConnection sendSynchronousRequest:_request returningResponse:&response error:&error];

    if (nil != error)
    {
        NSLog(@"Error: %@", error);
    }
    else
    {
        NSError* error = nil;
        NSMutableDictionary* json = nil;
        NSString* dataString = [[NSString alloc] initWithData:_connectionData encoding:NSUTF8StringEncoding];
        NSLog(@"%@", dataString);
        
        if (nil != _connectionData)
        {
            json = [NSJSONSerialization JSONObjectWithData:_connectionData options:NSJSONReadingMutableContainers error:&error];
        }
        
        if (error || !json)
        {
            NSLog(@"Could not parse loaded json with error:%@", error);
        }
        
        NSLog(@"%@", json);
        _connectionData = nil;
    }
    
    [pool drain];

    return 0;
}

<?php
// This sample uses the Apache HTTP client from HTTP Components (http://hc.apache.org/httpcomponents-client-ga/)
require_once 'HTTP/Request2.php';

$request = new Http_Request2('https://switzerlandwest.api.cognitive.microsoft.com/vision/v3.2-preview.3/analyze');
$url = $request->getUrl();

$headers = array(
    // Request headers
    'Content-Type' => 'application/json',
    'Ocp-Apim-Subscription-Key' => '{subscription key}',
);

$request->setHeader($headers);

$parameters = array(
    // Request parameters
    'visualFeatures' => 'Categories',
    'details' => '{string}',
    'language' => 'en',
);

$url->setQueryVariables($parameters);

$request->setMethod(HTTP_Request2::METHOD_POST);

// Request body
$request->setBody("{body}");

try
{
    $response = $request->send();
    echo $response->getBody();
}
catch (HttpException $ex)
{
    echo $ex;
}

?>
########### Python 2.7 #############
import httplib, urllib, base64

headers = {
    # Request headers
    'Content-Type': 'application/json',
    'Ocp-Apim-Subscription-Key': '{subscription key}',
}

params = urllib.urlencode({
    # Request parameters
    'visualFeatures': 'Categories',
    'details': '{string}',
    'language': 'en',
})

try:
    conn = httplib.HTTPSConnection('switzerlandwest.api.cognitive.microsoft.com')
    conn.request("POST", "/vision/v3.2-preview.3/analyze?%s" % params, "{body}", headers)
    response = conn.getresponse()
    data = response.read()
    print(data)
    conn.close()
except Exception as e:
    print("[Errno {0}] {1}".format(e.errno, e.strerror))

####################################

########### Python 3.2 #############
import http.client, urllib.request, urllib.parse, urllib.error, base64

headers = {
    # Request headers
    'Content-Type': 'application/json',
    'Ocp-Apim-Subscription-Key': '{subscription key}',
}

params = urllib.parse.urlencode({
    # Request parameters
    'visualFeatures': 'Categories',
    'details': '{string}',
    'language': 'en',
})

try:
    conn = http.client.HTTPSConnection('switzerlandwest.api.cognitive.microsoft.com')
    conn.request("POST", "/vision/v3.2-preview.3/analyze?%s" % params, "{body}", headers)
    response = conn.getresponse()
    data = response.read()
    print(data)
    conn.close()
except Exception as e:
    print("[Errno {0}] {1}".format(e.errno, e.strerror))

####################################
require 'net/http'

uri = URI('https://switzerlandwest.api.cognitive.microsoft.com/vision/v3.2-preview.3/analyze')

query = URI.encode_www_form({
    # Request parameters
    'visualFeatures' => 'Categories',
    'details' => '{string}',
    'language' => 'en'
})
if query.length > 0
  if uri.query && uri.query.length > 0
    uri.query += '&' + query
  else
    uri.query = query
  end
end

request = Net::HTTP::Post.new(uri.request_uri)
# Request headers
request['Content-Type'] = 'application/json'
# Request headers
request['Ocp-Apim-Subscription-Key'] = '{subscription key}'
# Request body
request.body = "{body}"

response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
    http.request(request)
end

puts response.body