Method and apparatus for a multisensor imaging and scene interpretation system to aid the visually impaired

• US 7,598,976 B2
• Filed: 06/12/2003
• Issued: 10/06/2009
• Est. Priority Date: 06/13/2002
• Status: Active Grant

• Current Assignee: I See Tech Limited
• Original Assignee: I See Tech Limited
• Inventors: Tidhar, Amnon, Sofer, Eli
• Primary Examiner(s): VO, TUNG T
• Application Number:
  US10/519,483
• Publication Number:
  US 20060098089A1
• Time in Patent Office: 2,308 Days
• Field of Search: 348/62, 348/63, 340/407.1, 340/825.1, 340/573.1, 340/539.1, 340/825.19, 356/601, 382/293, 709/229, 434/112
• US Class Current: 348/62
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G09B 21/006 using audible presentation of the information

Abstract

An apparatus and method for aiding a visually impaired or blind person to detect, identify and avoid objects in the field of vision of the surroundings. The apparatus includes electro-optical devices to detect and identify objects, a control unit, which receives and processes information from the devices and a vocal representation unit to receive instructions from the control unit for purpose of audibly describing the objects to the person in the field of vision, thereby enabling the person to cope and proceed safely in the surroundings. The method for aiding a visually impaired person to detect, identify and objects in the field of vision surroundings includes the steps of receiving visual images of the objects, receiving range information concerning the objects, processing the image information, and audibly describing to the person the objects in the field of vision, thereby enabling the person to cope with, and proceed in his surroundings.

Claims

1. A system for aiding a person with a visual impairment or visual obstruction, comprising:

means for sensing time/space characteristics and physical characteristics information about a plurality of objects in a field of view of said person,

said sensing means operating by reflecting and capturing laser pulses from the surfaces of said plurality of objects present in close range to said person to produce a cloud of spatial pixels (voxels) representing points on all said surfaces which can be observed from said person'"'"'s point of view;

means for identifying, interpreting, decomposing, and synthesizing said time/space characteristics and physical characteristics information and for high-level characterizing said plurality of objects in an object class,

said means for identifying operating by;

applying a set of low-level algorithms to decompose said cloud into a set of regular surfaces defined by parameters of type, dimensions, orientation in space, and relative to said user velocity vector, and

applying a set of high-level algorithms to act on said surfaces to combine them into identified 3-D objects with assigned names;

and

means for verbally communicating output and input information between said system and said user,

wherein said output information further comprises a scene description about said surrounding environment within the field of view of said person,

wherein said scene description is verbally communicated in conversational speech to said person based on the distance of said plurality of objects to said user, the closest being communicated first as having a higher priority,

wherein said scene description comprises said plurality of objects identified by said system, together with properties such as their position information, physical characteristics, and relative velocity vectors indicating motion, and

wherein said properties are provided by an Object Synthesis and Recognition module which associates said properties with an identified object class.

2. The system of claim 1, wherein said means for sensing comprises image sensing devices selected from the group consisting of:

CCD sensors, and a 3-D laser imaging sensor.

3. The system of claim 1, wherein said time/space characteristics and physical characteristics information about an object includes any of:

physical dimensions, general shape description, texture, color, the distance and position of said user from said object, motion of said object, spatial relationships between objects.

4. The system of claim 1, wherein said means for interpreting said time/space and physical characteristics information comprises:

a Sensors Processor and Control Unit (SPC) for interpreting low-level sensing data from said sensing means, and a High-level Vision Processing module for interpreting high-level sensing data from said sensing means.

5. The system of claim 1, further comprising a World Model knowledge database for performing said high-level characterizing.

6. The system of claim 5, wherein said World Model database comprises data selected from the group of:

physical descriptions of known objects and components thereof;

the relations of said objects to their respective said components;

common relationships among said components;

groupings of said known objects into classes and subclasses;

common properties of said classes, subclasses and groupings; and

common relations among and between said classes, subclasses and groupings.

7. The system of claim 5, wherein said World Model database is continuously enriched by adaptation to new experiences of said person and implemented in at least one option selected from the modes:

direct teaching mode;

generalization teaching mode by positive enforcement or default; and

refinement teaching mode by negative enforcement or correction.

8. The system of claim 7 wherein said direct teaching mode comprises giving a name to a part of a scene by said user.

9. The system of claim 7 wherein said generalization teaching mode comprises identifying a part of a given scene as belonging to a specific object class, although some of the relationships, or some of the components, may deviate from that identified in said database.

10. The system of claim 7 wherein said refinement teaching mode comprises refining the set of definitions to permanently exclude in the future an identification of only part of a scene as belonging to a specific object class when said user has rejected said partial identification.

11. The system of claim 7 wherein verbal instructions are received from said person in a direct teaching mode to improve said World Model knowledge database capabilities for naming and classifying a plurality of objects within said field of view.

12. The system of claim 1, wherein said means for interpretation comprises scene interpretation data and sensed data streams.

13. The system of claim 1, wherein said means for interpretation further comprises computations for relating distances and color information of said plurality of objects and components thereof within the field of vision of said person.

14. The system of claim 1, wherein said output information comprises said characterization and interpretation of said plurality of objects.

15. The system of claim 1, further comprising an automatic speech recognition unit for verbal input of instructions from said person for controlling said system.

16. The system of claim 15, wherein said instructions comprise naming and classifying objects known to said user.

17. A method for detecting, analyzing, identifying, and reporting a plurality of objects by name to a visually impaired or blind person in a field of view of said person, comprising the steps of:

providing a 3-D laser imaging system to accurately measure both the coordinates of said plurality of objects in both static and dynamic motion and the characterization of the relationships between said plurality of objects in relation to said person;

acquiring a plurality of 3-D visual images and information about said plurality of objects;

processing said plurality of 3-D visual images and said information;

storing said information of said plurality of objects by names in a hierarchy utilizing a structured World Model;

identifying said plurality of objects, and

communicating verbally with said person to report said information about said plurality of objects in said field of view,

reflecting and capturing laser pulses from the surfaces of said plurality of objects present in close range to said person to produce a cloud of spatial pixels (voxels) representing points on all said surfaces which can be observed from said person'"'"'s point of view;

applying a set of low-level algorithms to decompose said cloud into a set of regular surfaces defined by parameters of type, dimensions, orientation in space, and relative to said user velocity vector; and

applying a set of high-level algorithms to act on said surfaces to combine them into identified 3-D objects with assigned names.

18. The method of claim 17 further comprising 3-D object rendering.

19. The method of claim 17 further comprising:

providing a process of learning new facts about said physical surroundings of said person for growth and expansion of said World Model.

20. The method of claim 17 wherein said World Model hierarchically represents existing knowledge of the physical surroundings of said person;

classes of said plurality of objects;

the common relationships of said plurality of objects to their components; and

the way low-level components in said hierarchy can be composed of regular surfaces.