Vist BlindVision. Read Usage below for how to use the app. Recommended to always allow camera and microphone for better usage.
BlindVision is a user-friendly web application designed for the visually impaired, that helps blind people to see and describe the world around them.
The app uses a combination of computer vision and artificial intelligence to identify objects and obstacles in the user's environment. BlindVision can then use this information to provide the user with a description of their surroundings through speech.
Compatible With Mobile and Desktop
Send images and optional text messages for a descriptive reply about your surroundings.
Send text messages for a continuous conversation. Remember past messages you sent.
See the chat mode history, messages from the model and user.
Check objects in the frame and relative position, fast response time.
Triple tap (anywhere below the canvas/keyboard except delete button) to switch between modes (can do double tap to switch modes when in Chat History Mode/Real Time Mode) (can also triple click arrow keys to switch modes when in Image Chat Mode/Chat Mode)
Double tap (anywhere below the canvas/keyboard except delete button) to switch between keyboard and drawing (For Image Chat Mode and Chat Mode)
Desktop: uses webcam. Mobile: uses rear-facing camera. On mobile you must click on any button or canvas on the screen after switching modes to allow for speech.
Using the on screen keyboard, type in your username when the screen loads, say password to switch to entering password. Enter your password and say login or sign up.
Delete character -> left button.
Clear entered text -> middle button.
Check password/username entered correctly -> right button.
Delete character -> left button.
Clear entered text -> middle button.
Stop speaking -> right button.
Space key -> above the 3 buttons.
Drawing mode: Draw characters on the canvas using mouse or tap, text you write will be automatically spoken.
Quickly tap/click and release on the canvas to clear the canvas.
Quickly tap/click and release again if you want to add the cleared text.
SWIPE EVENTS WORK ONLY BELOW CANVAS IN DRAW MODE
Swipe-down/up-arrow-key to do speech recognition (recommended to use Keyboard mode for better accuracy), text will be added once speech recognition ends.
Swipe-up/down-arrow-key to speak the entered text.
Swipe-right/left-arrow-key to speak the received response from the model.
Swipe-left/right-arrow-key to send the message (image and entered text) to the model.
Keyboard mode: Same as above, except swipe works anywhere, click keys on the keyboard to add them.
Also click 123 to enter other characters as well. The characters are spoken as you click them.
Also includes Drawing mode and Keyboard mode, but only entered text will be sent (no image) and past messages are remembered, so you can continue the conversation.
Same usage as Image Chat Mode.
Tap messages to speak them, can scroll, logout and swipe-right/left-arrow-key to stop speaking or click the stop speaking button.
Click the top button to speak the objects, the bottom button to speak objects and their relative position. Also using the Arduino, you can use the noise level to know the approximate distance to the object (louder means closer).