Systems and methods for providing feedback based on the state of an object

• US 9,189,973 B2
• Filed: 12/20/2013
• Issued: 11/17/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: OrCam Technologies Ltd
• Original Assignee: OrCam Technologies Ltd
• Inventors: Wexler, Yonatan, Shashua, Amnon
• Primary Examiner(s): RAHAMAN, MOHAMMED S
• Application Number:
  US14/135,775
• Publication Number:
  US 20140267645A1
• Time in Patent Office: 697 Days
• Field of Search: 348/62
• US Class Current: 1/1
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/243 by compensating for image skew or non-uniform image deformations
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/144 using a slot moved over the image; using discrete sensing elements at predetermined points; using automatic curve following means
• CPC Class Codes: G06V 30/1478 of characters or characters lines
• CPC Class Codes: G06V 30/226 of cursive writing
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: G06V 30/418 : Document matching, e.g. of document images
• CPC Class Codes: G06V 30/43 : Editing text-bitmaps, e.g. alignment, spacing; Semantic analysis of bitmaps of text without OCR
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G08B 3/10 : using electric transmission; using electromagnetic transmission
• CPC Class Codes: G08B 6/00 : Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G09B 21/00 : Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B
• CPC Class Codes: G09B 21/001 : Teaching or communicating with blind persons G09B21/02 - G09B21/06 take precedence
• CPC Class Codes: G09B 21/003 : using tactile presentation of the information, e.g. Braille displays
• CPC Class Codes: G09B 21/006 : using audible presentation of the information
• CPC Class Codes: G10L 13/00 : Speech synthesis; Text to speech systems
• CPC Class Codes: H04N 23/50 : Constructional details
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

A device and method are provided for providing feedback based on the state of an object. In one implementation, an apparatus for processing images is provided. The apparatus may include an image sensor configured to capture real time images from an environment of a user and at least one processor device configured to initially process at least one image to determine whether an object is likely to change its state. If a determination is made that the object is unlikely to change its state, the at least one processor device may additionally process the at least one image and provide a first feedback. If a determination is made that the object is likely to change its state, the at least one processor device may continue to capture images of the object and alert the user with a second feedback after a change in the state of the object occurs.

Claims

1. An apparatus for processing images, the apparatus comprising:

an image sensor configured to be worn by a visually impaired user and configured to capture real time images from an environment of the user; and

at least one processor device configured to;

initially process at least one image of the captured real time images to identify an object in the environment of the user that is likely to change its state;

provide a first feedback to the user that indicates a current state of the object;

after providing the first feedback, determine that the object has left a field of view of the image sensor;

after determining that the object has left the field of view of the image sensor, provide a second feedback to the user that indicates the object is no longer in the field of view of the image sensor;

after providing the second feedback, determine that the object has reappeared in the field of view of the image sensor; and

after determining that the object has reappeared in the field of view of the image sensor, continue to capture images of the object and alert the user with a third feedback after a change in the state of the object occurs, wherein the third feedback indicates a new state of the object,

wherein the change of state includes at least one of a change in color, a change in position, and a change in display of information, and the change of state is determined while the object remains in the field of view of the image sensor.

2. The apparatus of claim 1, wherein the first feedback, the second feedback, and the third feedback are audible.

3. The apparatus of claim 1, wherein the object includes a traffic light and the change of state includes a change in color of the traffic light.

4. The apparatus of claim 3, wherein the at least one processor device is further configured to provide the first feedback when the traffic light is green or red, and alert the user with the third feedback after the traffic light turns from green to red or after the traffic light turns from red to green.

5. The apparatus of claim 1, wherein the object includes text on an electronic display and the change of state includes an update to the text on the electronic display.

6. The apparatus of claim 1, wherein the object includes a person, and the change of state includes a change in a facial expression of the person.

7. The apparatus of claim 1, wherein the object includes a moving vehicle and the change of state includes a change in direction or speed of the moving vehicle.

8. The apparatus of claim 1, wherein the at least one processor device is further configured to that a second object is unlikely to change its state based on at least one of:

a type of the object in the at least one image, an identification of at least a portion of a user'"'"'s hand, identification of a trigger, and a particular context of a background of the object.

9. The apparatus of claim 1, wherein the determination that the object is likely to change its state is based on at least one of:

a type of the object in the image, an identification of at least a portion of a user'"'"'s hand, identification of a trigger, and a particular context of a background of the object.

10. The apparatus of claim 1, wherein the image sensor is further configured to be movable with a head of the user such that a field of view of the image sensor substantially coincides with a field of view of the user.

11. The apparatus of claim 10, wherein upon determining that the object is likely to change its state, the at least one processor device is further configured to track the object, while the object remains within the field of view of the image sensor, to determine when the change of state occurs.

12. The apparatus of claim 11, wherein after the object exits the field of view of the image sensor, the at least one processor device is further configured to automatically resume following the object when the object reappears in the field of view of the image sensor.

13. The apparatus of claim 11, wherein after the object exits the field of view of the image sensor, the at least one processor device is further configured to resume following the object if the object reappears in the field of view of the image sensor within a predetermined period of time.

14. An apparatus for processing images, the apparatus comprising:

an image sensor configured to be worn by a visually impaired user such that a field of view of the image sensor substantially coincides with a field of view of the user, and to capture real time images from an environment of the user; and

at least one processor device configured to;

process the captured real time images to determine an existence of an object in the field of view of the image sensor;

determine that a state of the object is likely to change;

provide a first feedback to the user that indicates a current state of the object;

after providing the first feedback, determine that the object has left the field of view of the image sensor;

after determining that the object has left the field of view of the image sensor, provide a second feedback to the user that indicates the object is no longer in the field of view of the image sensor;

after providing the second feedback, determine that the object has reappeared in the field of view of the image sensor; and

after determining that the object has reappeared in the field of view of the image sensor, track the object while the object remains within the field of view of the image sensor and provide a third feedback to the user after the state of the object has changed, wherein the third feedback indicates a new state of the object,

wherein the change of state includes at least one of a change in color, a change in position, and a change in display of information, and the change of state is determined while the object remains in the field of view of the image sensor.

15. The apparatus of claim 14, wherein the at least one processor device is further configured to provide a plurality of audible feedbacks, each audible feedback being associated with a specific change in the state of the object.

16. The apparatus of claim 14, wherein the at least one processor device is further configured to stop tracking the object after a predetermined number of changes in the state of the object have occurred.

17. The apparatus of claim 14, wherein the at least one processor device is further configured to stop tracking the object after a predetermined period of time.

18. The apparatus of claim 14, wherein the at least one processor device is further configured to stop tracking the object after a predetermined period of time without any change to the state of the object.

19. A method for providing feedback to a visually impaired user, the method comprising:

receiving from an image sensor real time image data from an environment of the user, wherein the image sensor is configured to be positioned for movement with a head of the user;

processing the image data to determine that a state of an object in the environment of the user is likely to change;

providing a first audible feedback to the user indicating a current state of the object;

after providing the first audible feedback, determining that the object has left a field of view of the image sensor;

after determining that the object has left the field of view of the image sensor, providing a second audible feedback to the user that indicates the object is no longer in the field of view of the image sensor;

after providing the second audible feedback, determining that the object has reappeared in the field of view of the image sensor; and

after determining that the object has reappeared in the field of view of the image sensor, tracking the object while the object remains within the field of view of the image sensor and providing a third audible feedback to the user after the state of the object has changed, wherein the third audible feedback indicates a new state of the object,

wherein the change of state includes at least one of a change in color, a change in position, and a change in display of information, and the change of state is determined while the object remains in the field of view of the image sensor.

20. A software product stored on a non-transitory computer readable medium and comprising data and computer implementable instructions for carrying out the method of claim 19.