Last updated on 27 September 2024
Google is rolling out version 15.0 of the TalkBack screen reader, with The most prominent new feature being the addition of Gemini-based image descriptions. Let’s take a look at what’s new in TalkBack 15.0. Please note that the rollout is gradual, so if you don’t see the update yet, be patient—it should reach your device in the coming days.
Table of Contents
Gemini-Based Image Descriptions
Google introduced image and icon descriptions in TalkBack starting with version 14.1. However, these descriptions were often undetailed and unreliable, and they were generated offline on the device after downloading the necessary data.
TalkBack 15.0 takes image descriptions to a new level with the implementation of Gemini-based image descriptions. Now, when you select the “Describe image” option from the TalkBack menu—or if you’ve assigned a gesture to it—you’ll be asked if you want to use the detailed AI image descriptions. You can also enable this from TalkBack settings > Image and icon descriptions > Detailed image descriptions. Note that the automatic descriptions still use the old offline data.
The result of using the new detailed image recognition is more detailed, richer descriptions. During our brief testing, we found the descriptions to be more useful and detailed. For example, people were recognized correctly (we tested with an image of two famous politicians, and they were identified). However, the current implementation has its shortcomings.
The new AI-based image descriptions are generated online for most phones. Although the results are obtained quickly with minimal waiting time, TalkBack does not revert to the offline, less detailed descriptions when there is no Internet connection. Another point worth mentioning is that for automatic image descriptions, you can choose to use on-device generative AI if you have Pixel 9 series phones. At the time of this writing, the on-deviced-based approach is limited to Pixel 9 handsets. Namely, Samsung phones have been excluded.
Another issue worth mentioning is related to text detection. Unlike the poor offline image descriptions, TalkBack has traditionally performed well in text and icon detection, accurately recognizing text in images and focused items. However, with the Gemini-based online descriptions, detected text is now part of the general description. This means Gemini decides how to read the text, often resulting in a summary or partial reading. Since there is no manual way to perform text detection, TalkBack has lost its accurate text detection capabilities, replaced by a less reliable solution that may miss parts of the text or produce hallucinations based on the detected text—unless the user chooses to disable detailed image descriptions and rely on the offline, less detailed descriptions. In summary, you either use the detailed image descriptions as they are or not use them at all.
It’s worth noting that automatic detections still use the offline data, including text detection, but this method is less convenient—especially if you enable icon and text recognition for all items, not just unlabeled ones.
Granular Punctuation Reading Control
TalkBack 15.0 introduces granular control over punctuation reading, though it’s quite basic. You can now choose between reading all, most, or some punctuation marks. “All” reads all punctuation, while “Some” reads only a few. This feature doesn’t affect character navigation and can be managed from verbosity settings or by adding it to reading controls in the Customize menus section of TalkBack settings.
New Commands for Text Selection with a Braille Display or TalkBack Braille Keyboard
When using a supported braille display or typing with the TalkBack braille keyboard, you can now select text from the cursor position to the beginning or end of the text.
- Braille Display: Press Space + Dots 1, 2, 3, 7, 8 to select from the cursor to the beginning of the text, and Space + Dots 4, 5, 6, 7, 8 to select from the cursor to the end of the text.
- TalkBack Braille Keyboard: Select from the cursor to the beginning of the text by holding Dots 4 and 5, then swiping up anywhere with two fingers. Select from the cursor to the end of the text by holding Dots 4 and 5, then swiping down anywhere with two fingers.
Miscellaneous Changes
- Read from Next Item is now called Read from Focused Item to reflect how this feature works now —it starts reading from the currently focused item instead of the next item.
- Some users have reported improved responsiveness, but we can’t confirm if this is real or just a placebo effect, and whether it applies across most devices.
- You can now open the Google disability support contact page from TalkBack settings using the new Contact Disability Support option.
Conclusion
The new TalkBack version doesn’t offer much beyond the Gemini-based image descriptions, which still have room for improvement. However, we appreciate it as a first step. We are still exploring the new TalkBack version and will update this post if we find anything worth mentioning. Google has also published a support article about TalkBack 15.0 and it can be accessed here.
Comments