In May 2018, a Google blog post introduced Lookout, a new app designed to help visually impaired users explore their surroundings. The announcement generated waves of excitement among blind Android users, but it took over a year for the app to become available on the Play Store. Fast forward to 2025, Lookout
remains active, regularly updated, and has undergone major improvements and changes since its initial release. This article explores the app’s current features and, more importantly, considers how it could evolve to become an indispensable tool for blind users.
Table of Contents
Lookout Current Features and Modes
Lookout includes several features that are accessed through modes ranging from short text and document scanning to image recognition, currency identification, food label recognition, and item detection in the surroundings.
Text and Document Scanning
Whether selecting the short text mode that reads any text in view instantly or opting for the documents mode, which is designed for scanning entire papers, Lookout demonstrates strong text recognition capabilities, especially after supporting more languages recently. I usually use the short text mode when checking the update status while updating the PC’s BIOS, for example.
The documents mode provides guidance on how to move the camera to capture the entire paper, such as instructions like “move away,” “move top left,” or “move top right.” It automatically takes the photo once the entire paper is detected, asking the user to hold still. On my phone, though, it can be slightly slow, often taking the photo while my hand is moving. Nonetheless, the text recognition itself remains very good.
A useful included addition is the ability to save or share the captured image.
When sharing an image to the app, Lookout also performs OCR and extracts any available text from the image.
Images Mode
Lookout allows describing photos either by capturing them with the camera, opening them through the app, or sharing them from other apps. The initial descriptions are brief and not detailed. However, after limiting the ability to ask follow-up questions to specific countries for over a year, this feature is now available globally. Using the questions feature, it is possible to ask Lookout for a detailed description of a photo.
In these detailed descriptions, a tendency to provide opinions and comments becomes apparent. For instance, when the app detected a solar panel on a building’s roof, it remarked that they are used as a means to obtain sustainable energy. It also noted that plants on a balcony added a touch of nature to the scene, mentioning a sense of depth and other similar observations here and there.
While testing the question-asking feature, I noticed that the model used appears outdated and less reliable. For example, when asked whether the Russian president is still in office, it referenced the Soviet Union unnecessarily while affirming the presidency, which was irrelevant to the context. Similarly, when asked about a Samsung phone image, specifically whether the Galaxy S24 had been released, it responded that the most recent Samsung S device it knows of is the S23, indicating that its database has not been updated to include information from 2024.
Unlike Gemini Assistant, Lookout does not shy away from politics and, in some cases, goes further than it should, such as questioning the legitimacy of a president’s leadership.
Questions can be typed or spoken, offering flexibility in interaction.
When using the images mode with the detection option enabled, the app announces the names of detected items, such as a person or curtain, though it does not provide approximate locations or directions. Despite this limitation, having the detection feature is a welcome addition as it could be useful.
It’s worth noting that when the front camera is selected, Lookout announces the positions of faces in the frame, which can assist in taking selfies. The captured image can then be saved using the download button or shared with other apps.
Other Features
- Food Labels:
This feature first prompts the user to select a country and download its corresponding database. It then attempts to detect barcodes and identify the food item. However, it requires significant improvements—more on this in the relevant section. - Find and Explore : The explore mode announces the names of items detected as the camera moves, while the find mode allows users to select a specific item from a list, such as cups, windows, or seats, and search for it. It also attempts to provide the distance and direction to the detected object. However, this is often impractical as it does not account for obstacles in the way. Additionally, the distance accuracy is unreliable, though the results may vary depending on the device being used.
- Currency: This feature recognizes only a few currencies. Although I haven’t tested it recently, I am fairly confident that it correctly identifies U.S. dollar bills.
Lookout’s Untapped Potential
Developed by Google—a company with immense financial resources and a significant presence in the generative AI and LLM race—Lookout has the potential to deliver much more. In its current state, however, it misses a significant opportunity to become a leading app for blind users leveraging AI.
Taking Advantage Of Gemini’s Advanced Capabilities
Despite the shortcomings, Gemini’s current models excel in providing detailed image and video descriptions. Features like TalkBack’s detailed image descriptions
and PiccyBot’s image and video recognition capabilities, powered by Gemini models, consistently impress me with their results. Unfortunately, Lookout does not yet utilize Gemini updated models.
Upgrading Gemini and Adding Video Descriptions
Lookout currently lacks any video description functionality. Additionally, the image descriptions it provides suggest that the Gemini model in use is an older, less capable version. Upgrading the Gemini model in Lookout is essential for staying competitive, especially since Google’s own TalkBack screen reader delivers more accurate and detailed image descriptions.
The importance of video descriptions cannot be overstated. As a blind user, I didn’t realize how much I was missing until I experienced them. Hearing details about facial expressions, visual reactions, and other elements not only enriches the imagination but also enhances understanding and enjoyment in many scenarios. Google should prioritize adding this functionality to make Lookout a more powerful tool for blind users.
Gemini 2.0 Flash Goodies
Gemini 2.0 Flash, which offers live camera use during a conversation, is a game-changer even in its initial state. While it still has its confusions and troubles, it can transform the experience of a blind user significantly. Being able to ask about surroundings, understand people’s gestures, and read text in real-time is undoubtedly a big upgrade.
Gemini 2.0 Flash also shows improvements in its conversational capabilities compared to previous models, making interactions more beneficial and less annoying.
Lookout has the potential to take the lead in blind-related tools by implementing this new model’s goodies. Even if Google decides to give this feature a beta or experimental label, as it often does with new features, incorporating these capabilities would still make a significant impact.
Imagine a Lookout mode that starts a live conversation with Gemini, utilizing its access to the camera. The support it could provide in delivering more visual analysis and assistance to the user would be revolutionary.
Extending OCR Capabilities and Merging Them with AI
While Lookout’s OCR detection is among the best available, there is still room for improvement. A key area for development is the support for reading PDF documents. Handling PDFs remains problematic for blind users, especially when dealing with documents that contain only image scans or those in languages like Arabic. These often result in misrendered text, leading to scattered, meaningless characters. Improving Lookout’s PDF support would make its text scanning abilities more complete and less limited.
Speaking of removing limits, it’s noticeable that Lookout lacks text formatting detection or a meaningful display of table data for blind users. Simply showing the recognized text is insufficient for certain documents, particularly when dealing with tables. Google has the tools to take text recognition to the next level by better understanding the document’s structure. This would improve how text is displayed and clarify the relationships between text elements.
AI could also play a crucial role in enhancing document analysis and recognition. The goal here isn’t to provide summaries or answer content-related questions, but to better detect the document’s formatting and present the recognized text in a more understandable way.
A Different Approach to Barcode Detection
Restricting barcode detection to food labels and making the database region-specific is quite limiting. Expanding the database would yield better results and allow for broader barcode detection. The ability to search for unrecognized products would also be a welcome change. Given that Google has the best search engine, why not use it to help blind users identify products?
Currently, there is no real guidance when trying to detect a product, unlike Seeing AI, which uses beeps to help users locate the barcode more easily. While Seeing AI isn’t perfect, it’s still better than having no feedback at all. Lookout could benefit from adopting this guidance approach, implementing it in some form to assist users in scanning the barcode, particularly since most of the time, there is nothing tactile to guide the user.
Not Being Shy of Venturing into New Experiments
Anyone who follows Google knows the company isn’t afraid to make bold moves—even when they backfire. Gemini is a prominent example. Google, initially far behind OpenAI in the LLM competition, rushed to release AI models that were criticized more than praised. However, Gemini has since started to catch up and excel in certain areas.
Yet, when it comes to blind-related products, Google tends to be more conservative. While it’s understandable that blind-related tools should be stable, building a better future requires embracing innovation and navigating through rugged terrain.
Lookout isn’t TalkBack, the built-in screen reader that demands stability and careful, calculated updates. Lookout can be a platform for experimenting with new ideas and taking advantage of cutting-edge AI models and technologies. Google should stop seeing the blind community as naive or weak—it should recognize that they are a part of the entire user base, which ranges from less knowledgeable to the smartest.
Experimental features could be hidden behind a license agreement and a few stern warnings, but they should be made available to anyone willing to contribute to building a better future.
Scrapping the Geo-Restriction Strategy
Lookout is one of the worst illustrations of Google’s geo-restriction policy. The app took a long time to be rolled out globally, and then the question-asking feature was available only in a few countries for no valid reason—not for a week or a month, but for over a year. If Google truly values inclusivity, it should think more globally. Everyone should have the opportunity to test new features, except in cases where legal considerations in certain countries apply. There are blind users who understand and use English all over the world, so language should not be used as an excuse.
Others
I’m not someone who wants an app to do everything or try to reinvent the wheel, so I’m not advocating for expanding or broadening the feature set. I believe it’s better for an app to excel in a few key areas rather than offering a bunch of half-baked features. That said, I still think Lookout could benefit from adding some navigation-related features. For example, a more advanced “Find” mode that better understands the camera’s orientation and provides directions relative to the user’s position. At the very least, it could identify obstacles when trying to guide the user to a particular object.
Another area for improvement is using object detection to assist in capturing photos. Lookout could do more to help users focus on specific objects, like a sign or a person, to help them capture a clear photo. While Lookout has a solid foundation, it just needs a few additional steps to make this a practical and effective feature.
Are the Discussed Improvements Viable or Just Impractical Ideas?
When evaluating the future of an app and its potential, several factors must be taken into account, such as the team behind it, the available financial resources, and access to tools and utilities.
Lookout is developed by Google, the company behind Google Search, with vast amounts of data at its disposal, enormous financial resources, and a significant stake in the AI and LLM competition. If someone argues that the team behind Lookout is small, the response could be that allocating human resources to support the app’s development is relatively easy, considering the scale of the company backing it.
There are features that Google offers as paid options in its other apps and services. Introducing a few of these, like the most advanced features from Gemini, into Lookout wouldn’t hurt Google. In fact, it could improve the company’s image, positioning Google as a company that truly cares about blind users and helps them analyze and interact with the visual world around them.
Furthermore, Google could implement measures to prevent those who aren’t blind from taking advantage of features, as well as ensuring that blind users do not misuse them.
Given all these factors, the discussed improvements are entirely viable and possible. Whether Google has the determination and willingness to move the app forward, however, is another matter. This will ultimately determine whether Lookout reaches the state that blind Android users long for.
I believe that Lookout, even in its current state, will always have a place on my device, but at the same time, I see it lagging behind the competition in several areas. I remain confident that there is untapped potential for this app to become a leader that others will follow. At present, Lookout is like a bird with its wings tied, waiting for the freedom to soar high.
Comments