Last updated on 27 January 2025
As a blind user, you might find an app that seems interesting. However, after installing and opening it, you discover that you can’t figure out what each button or control does. Proper labeling of UI elements by app developers is essential for screen readers to enable blind users to understand the functions of these elements. Labels are programmatically referred to as “ContentDescription”. Unfortunately, many app developers neglect to add content descriptions to their apps’ UI elements, making it challenging for screen reader users to navigate and use the apps efficiently. Screen readers on Android have label identification and adding capabilities, although these are limited in many scenarios. This article will cover the label-related features available in both TalkBack and Jieshuo screen readers.
Table of Contents
TalkBack
The TalkBack screen reader includes an option to label certain unlabeled elements. It also has an option to read element IDs if they aren’t labeled, as well as a feature to try to identify unlabeled elements even if they don’t have an ID.
The “Add a Label” Option
Introduced in Android 4.3, the “Add a Label” feature is one of TalkBack’s essential tools. It allows users to focus on an unlabeled button, select “Add a Label” from the TalkBack menu, and enter a custom label. Alternatively, users can add labels by issuing a voice command that begins with the word “label”, followed by the custom label text, after focusing on the unlabeled button. Once a custom label is added, TalkBack will use it to identify the item whenever it is focused.
To edit a label, focus on the same button and choose the “Edit Label” option from the TalkBack menu or issue the label voice command mentioned above with the edited text. Additionally, open the TalkBack settings, activate the advanced settings, and select the custom labels section. There, you can find all the custom labels, where you can remove or edit them. In the same section, you can export the created labels and import them. However, as far as I know, the importing feature is currently broken, which makes label sharing practically impossible.
While this feature is helpful, it has limitations. For TalkBack to label an element, the element must be a button with an assigned ID. IDs are given by app developers to identify elements, but some developers may choose not to assign IDs to certain buttons. Additionally, if multiple buttons share the same ID, labeling any one of them will apply the same label to all buttons with that ID, making custom labeling ineffective. Moreover, TalkBack’s labeling feature is limited to buttons only and doesn’t support other UI elements, even if they have IDs.
Speak Element ID for Unlabeled Buttons
Located in TalkBack settings > Verbosity Settings, this option allows TalkBack to read a button’s ID if it lacks a label or ContentDescription but has an ID. Sometimes, developers assign IDs that hint at the button’s function—for example, an ID of “play” for a button in a music player app indicates that the button is meant to play the media. In such situations, hearing the ID is helpful, giving users the choice to either use the ID directly or assign a custom label using the “Add a Label” option mentioned earlier. The same ID use limitations mentioned previously apply here.
Automatic Icon Descriptions
The introduction of icon descriptions marked a significant improvement for TalkBack, enabling it to identify more UI elements and buttons without IDs. TalkBack downloads icon description data to use offline on focused elements, allowing users to choose automatic icon descriptions for unlabeled icons, for all icons, or to turn the feature off. This setting is found in TalkBack settings under “Image and Icon Descriptions” > “Automatic Icon Descriptions”.
Additionally, TalkBack 15’s “Detailed Image Description” feature can sometimes provide descriptions for unlabeled elements that hint at their functions.
Automatic icon descriptions have shown impressive results and have proven to be a reliable way to identify many UI elements in apps that don’t respect accessibility labeling. However, users cannot assign a custom label to some of these items, as discussed earlier. Also, TalkBack 15’s icon description response time is noticeably slow, taking a few seconds sometimes to provide a description, which can disrupt smooth navigation.
Jieshuo
similar to TalkBack, Jieshuo screen reader includes the option to label buttons that have attributed ids but lack screen reader labels. Additionally, Jieshuo attempts to allow labeling of elements that do not have IDs by using its item indexing. While this method is not always effective, it is available for cases when it does work.
Adding a Label
When the user focuses on an element that can be labeled, they can open Jieshuo’s main menu. The dynamic button, typically located near the bottom left of the screen before the “Cancel” button, will change to “Add a Label.” The user can then input a custom label and decide whether to share it with Jieshuo’s server. There’s also an option to translate the provided text using Jieshuo’s translation feature. The text box shows either the item’s ID or its index (if the item lacks an ID and index reading is enabled). Tapping “OK” saves the label. Note that the same button used to add labels is used to edit them.
An example of the extended labeling capability in Jieshuo is seen with the two unlabeled tabs in the CX File Explorer home window. While these tabs cannot be labeled using TalkBack, I was able to successfully label them in Jieshuo.
Adding and Editing Labels with the Custom Label Manager
As users navigate through apps, Jieshuo saves unlabeled elements in “json” files within the “Labels” folder inside the Jieshuo main directory. Simply detecting an unlabeled element is enough for it to be included in the file. To view these files, navigate to Settings > Reading Settings > Custom Label and Alias Settings > Custom Label Manager. Here, users can access JSON files that Jieshuo creates for different apps. Each app typically has a file using the app name and additional files for various versions of the same app. When an app is updated, Jieshuo retains old files, which can cause the list to grow over time.
In the Custom Label Manager, users can review all added labels and identified unlabeled elements by tapping any JSON file to see a list. Selecting an item shows two text boxes: one for the identifier and another for Jieshuo’s label for the item. To modify a label, users can edit the second text box and save. An “Add” button is available for creating new entries, while a long press allows deletion of individual labels or entire files.
Label Reading Options
To enable ID reading for items without accessibility labels, go to Settings > Reading Settings > Label Reading Settings and select “Read Unlabeled Controls”. Additionally, selecting “Read the Index When the Control Has No Label” enables Jieshuo to read numbers associated with controls that lack IDs. While users can label these elements, it’s worth noting that this labeling feature may apply the same custom label to different items in different app screens, leading to potential incorrect labeling.
Recognizing Unlabeled Items
Jieshuo doesn’t provide an offline icon detection database. However, it includes an option that tries to identify unlabeled elements online. To enable this option, go to Advanced settings, then OCR settings, and check the Auto Image Description option. Once this option is enabled, after focusing on an unlabeled element, wait for the recognition to finish, and the automatic result will be read back.
Additionally, in OCR settings also, you can enable the “Automatically Use OCR” option, which detects any available text in unlabeled elements using the OCR engine you select as the default OCR engine for Jieshuo.
Moreover, users can manually detect icons using the “Recognize the Focused Icon” function, accessible in the main menu, the “Recognition Menu,” via voice commands, or by assigning it to a gesture. Note that The Vivo BlueHeart AI model, used to recognize unlabeled elements, is utilized for all image description functions in Jieshuo and currently doesn’t support English natively. Results are automatically translated by Jieshuo’s translation service based on the user’s selected language in translation settings. However, the functions are still useful for identifying icons or suggesting their possible functions.
Label Sharing
In Custom Label and Alias Settings, the “Use Cloud Labels” option allows users to download and use labels shared by others. Labels can also be uploaded directly through the “Share” option in the “Add a Label” dialog. However, enabling cloud labels may lead to inconsistencies or incorrectly labeled elements due to automatic sharing. With a large user base, the management of these labels by Jieshuo remains unclear, and there’s no way to monitor or select which downloaded labels to use.
Manual sharing of labels is possible by sharing the entire “Labels” folder or specific JSON files from this folder, allowing others to place them in the same folder on their devices.
Lastly, it’s possible to back up custom labels to the cloud through the Backup and Restore settings in Advanced Settings, and labels are included in the Export All option.
Changing the App Developers’ Provided Labels
In some apps, developers add random labels to elements that are even less helpful than having no label at all. While TalkBack cannot change these labels, Jieshuo allows users to replace them after adding the item to “content blacklist”. When a user applies this function to a focused item, the item is blacklisted. If the item has an ID, Jieshuo can read the ID instead of the original text; if it doesn’t, Jieshuo will read nothing when the element is focused. After blacklisting the item, the user can add a new label using the “Add Label” button (discussed in the previous section). This new label will override the original, but there are some limitations.
A notable issue occurs when the same text appears elsewhere in the app, such as in an edit box. For instance, if a messaging app has a “More Options” label that the user changes, and the same text is typed into a message, Jieshuo might read nothing in the text box, assuming the content should be ignored. This means that using this method to fix mislabeled elements isn’t always effective, so it should be applied cautiously.
Final Remarks
Identifying and labeling unlabeled UI elements can significantly improve the user experience for blind users who rely on screen readers to understand the functions of app elements. However, despite these workarounds, adding a “contentDescription” in the app’s code remains the best solution. When developers invest just a few minutes to add these descriptions, they create a much more accessible experience and spare users the task of identifying and labeling elements manually, if that’s even possible.
When developers add labels, they should avoid adding the element type or activation instructions to the description. Screen readers can independently identify element types and states and offer beginner hints, making redundant information unnecessary and potentially distracting.
As much as we appreciate advancements in label identification, we should continue to emphasize the importance of awareness among developers about proper labeling practices. Properly labeled elements reduce confusion and save users time and effort. Missing proper labels remains a significant accessibility issue across many apps, and no screen reader solution can substitute for well-implemented accessibility practices by app creators.
See Also
If you are a developer, you may take a look at the following resources:
Content labels,
Android Accessibility Guide
Items labeled with type or state,
Code examples for button labeling on Android accessibility.
Comments