OpenAI's ChatGPT can now analyse images using GPT-4 Turbo with Vision

OpenAI on Wednesday announced an update for GPT-4 Turbo, its most advanced AI language model. The AI model has now received Vision capabilities, allowing it to analyse multimedia inputs. This means that it can now analyse images and offer insights. This ability will be available in API for developers as well as for the public through ChatGPT.

OpenAI’s GPT-4 Turbo with Vision can analyse images

OpenAI’s developer account took to X (formerly Twitter) to announce GPT-4 with Vision. The post said ” GPT-4 Turbo with Vision is now generally available in the API. Vision requests can now also use JSON mode and function calling.”

GPT-4 Turbo with Vision is now generally available in the API. Vision requests can now also use JSON mode and function calling.https://t.co/cbvJjij3uL

Below are some great ways developers are building with vision. Drop yours in a reply 🧵

— OpenAI Developers (@OpenAIDevs) April 9, 2024

With Vision capabilities, it can analyse images and decipher them. The company also shared some examples of how this feature will work. Several brands will be using the updated API that the Vision capabilities.

Bengaluru-based Healthfy Me is also using its updated API with Vision capabilities to make tracking macros easier for its customers. Users need to point the camera to the food and the AI model will tell the macros and suggest if you need to walk after having the meal.

As for ChatGPT, the feature will be available to Plus users.  For the unversed, ChatGPT Plus is a paid subscription that costs $20 per month. Plus users can simply send a photo to ChatGPT in the chat and ask it to offer insights into the image. For instance, if you send a picture of a tourist destination like Burj Khalifa, then it will tell you the details about the place such as the height, material used to build it, and other similar things.

OpenAI has used the training data that was available till December 2023 to train GPT-4 Turbo with Vision. This is the same as the previous version of the model. ChatGPT’s free version uses GPT-3.5 and isn’t trained with the latest data. It has been trained with data available till September 2021. That said, it cannot offer the latest information or analyse images like GPT-4 Turbo with Vision.

When we asked GPT-3.5-powered ChatGPT, it responded by saying “I’m afraid I can’t directly analyze images since I’m a text-based AI and don’t have access to visual input.”

In other news about OpenAI, the company’s Dall-E received an update. Dall-E now allows users to edit AI-generated images directly in ChatGPT. Read the linked article to know how.