Google allows users to turn on the camera feed while in a conversation with Gemini Live, so they can ask about things that are in view. Although a blind user can take advantage of this feature, it is not designed specifically to assist blind people. Google is now rolling out Guided vision, a mode that uses the camera in a live conversation to help blind people get an overview of their surroundings, read text, and more. This post covers turning the feature on and using it, as well as my first impressions from testing it.
Note that I haven’t done anything special to access Guided vision on my Honor 600 Pro, but I am running a three-month AI Pro trial that came as an offer with my device, so I don’t know if that affected the feature’s arrival on my device. The feature is not available for my other Accessible Android colleagues yet.
How to Enable Guided vision
To enable Guided vision:
- Open the Gemini app.
- Select the signed-in account, shown as your account name.
- Choose Settings and search for Guided vision in Live.
- Toggle it on.
A notification appears on the screen with the ability to start a Live session directly. When you use the button to start the Live session, it starts with the camera feed enabled, so you can start talking.
Note that the feature is still not available for everyone, so you should wait if you cannot find the Guided vision toggle in Settings.
According to the Google Accessibility help article, it’s possible to assign Guided vision to an accessibility shortcut, and you can launch it through TalkBack too. However, both are still not available on my device. In fact, I haven’t received any TalkBack update recently.
I haven’t tested to see if launching the camera in a Live conversation when Guided vision is active uses the mode without any user’s intervention, but during my tests, I always asked Gemini to start Guided vision to make sure that it works in that mode. When I tried to prompt the assistant to start the feature, it asked me to point the camera at what I wanted described, but that was one of its hallucinations, as it didn’t open the camera feed or start the Live conversation. Starting a Live conversation is still necessary.
At another time, I asked it to start a Live conversation with Guided vision, but at that instance, it told me that it couldn’t start it automatically and asked me to start Live manually.
Note that turning off Guided vision and then turning it back on from Settings shows the notification to start a Live conversation again, which can be used to start a conversation with the camera turned on.
Early Tests and Observations
When I had my first conversation with Live with Guided vision turned on, Gemini reminded me that it cannot detect all hazards or guarantee path clearance and that I should use my mobility aid before almost every response. I kept telling it that I understand the risks and that it shouldn’t repeat those warnings. During the later conversations, I didn’t hear any of the warnings. I am not sure, though, if my early prompting helped, or if you hear those warnings only during the first conversation to alert you of the risks or when Gemini thinks that it should do so for your safety.
During conversations with Guided vision active, I couldn’t find any indication that a special mode is active. All Live-related controls are the same, with no difference. However, the difference is in how Gemini acts. During my short tests, I found that Gemini tried to be detailed in its descriptions because it was aware that Guided vision is active and that the mode is designed to assist blind users.
I was able to ask it questions about the mode, and I got assurance from it that the mode is to assist blind users. It gave me guidance on how to move the camera to show certain objects or scenes better. When I asked it to tell me how to move to reach something, it told me that it needed to know exactly where I was in relation to the camera’s orientation. Using my hand as a reference, it gave me directions with respect to my hand. It also described the positions of items in the room relative to each other, like the position of a fan relative to the sofa.
Additionally, Gemini told me the book that I was holding and tried to assist me in putting the full page in view. When I asked it to read the page, it read a part of the page, though. However, when I asked it if that was the only text on the page, it correctly stated that there were other texts and visual diagrams and offered to describe them for me.
These observations are based on my initial tests, and I haven’t made direct comparisons between using Guided vision with the feature turned on and using the normal Live camera mode with it turned off.
Observed Limitations
Guided vision changes only how Gemini tries to provide information, as well as making it give more hints to assist a blind user. However, it is still a part of the Live experience. This means that current Live limitations are still present. The most important limitation is the need to prompt it again to tell you about any change, as Live doesn’t have real-time monitoring or continuous picture analysis capabilities.
Another factor worth mentioning is the model used, as Guided vision doesn’t switch Gemini to a more capable AI model than the one that you are using in your Gemini app.
Although image analysis has already advanced and helps in many scenarios, misidentifications are always a part of the formula that shouldn’t be forgotten. For example, a towel hanging on a door was identified more than once as pink, although it wasn’t pink.
Additionally, leveraging the new mode to take pictures still needs work. After Gemini gave me directions to centre my face and I told it to take a photo, it obeyed my order, opened the camera, and took a photo, but that led to the termination of the Live session. That behavior ended the conversation without being able to make sure that the photo was properly captured or without asking any additional questions about what it includes.
Final Remarks
Guided vision is an interesting step forward from Google in tailoring Live camera to blind users. Although Google still has the Lookout app, it decided to make Guided vision a part of the Gemini app. Google decided to launch the feature and mention TalkBack as one of the methods to turn it on before releasing a TalkBack update that includes the necessary shortcut. There is even no accessibility service that can be assigned to an accessibility shortcut at the time of writing. However, an interesting thing for users who rely on other screen readers is that using the feature is not tied to having TalkBack running. I am almost sure that Gemini doesn’t check whether a screen reader is running as a requirement to use the feature.
Although it is in its early stages, I am excited to keep Guided vision on, and I am not considering going back to the normal camera mode. This should minimize the need to remind Gemini that I am blind or to try to speak prompts asking it to help me guide my camera toward the subject I am asking about or the text I want it to help me read.
Google doesn’t lack the resources to improve how Guided vision works, with the money and ability to make Gemini Live a trusted assistant, as much as “trusted” can be used with AI services, that understands the needs of blind people and helps them live a more independent life.
Comments