Systems and methods for audible facial recognition

• US 9,025,016 B2
• Filed: 12/20/2013
• Issued: 05/05/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: OrCam Technologies Ltd
• Original Assignee: OrCam Technologies Ltd
• Inventors: Wexler, Yonatan, Shashua, Amnon
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): Itskovich, Mikhail
• Application Number:
  US14/135,727
• Publication Number:
  US 20140267642A1
• Time in Patent Office: 501 Days
• US Class Current: 348/62
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/243 by compensating for image skew or non-uniform image deformations
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/144 using a slot moved over the image; using discrete sensing elements at predetermined points; using automatic curve following means
• CPC Class Codes: G06V 30/1478 of characters or characters lines
• CPC Class Codes: G06V 30/226 of cursive writing
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: G06V 30/418 : Document matching, e.g. of document images
• CPC Class Codes: G06V 30/43 : Editing text-bitmaps, e.g. alignment, spacing; Semantic analysis of bitmaps of text without OCR
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G08B 3/10 : using electric transmission; using electromagnetic transmission
• CPC Class Codes: G08B 6/00 : Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G09B 21/00 : Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B
• CPC Class Codes: G09B 21/001 : Teaching or communicating with blind persons G09B21/02 - G09B21/06 take precedence
• CPC Class Codes: G09B 21/003 : using tactile presentation of the information, e.g. Braille displays
• CPC Class Codes: G09B 21/006 : using audible presentation of the information
• CPC Class Codes: G10L 13/00 : Speech synthesis; Text to speech systems
• CPC Class Codes: H04N 23/50 : Constructional details
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

A device and method are provided for audible facial recognition. In one implementation, an apparatus for aiding a visually impaired user to identify individuals is provided. The apparatus may include a portable image sensor configured to be worn by the visually impaired user and to capture real-time image data from an environment of the user. The apparatus may also include at least one portable processor device configured to determine an existence of face-identifying information in the real-time image data, and access stored facial information and audible indicators. The at least one portable processor device may also be configured to compare the face-identifying information with the stored facial information, and identify a match. Based on the match, the at least one portable processor may be configured to cause an audible indicator to be announced to the visually impaired user.

Claims

1. An apparatus for aiding a visually impaired user to identify individuals, the apparatus comprising:

a portable image sensor configured to be worn by the visually impaired user and to capture real-time image data from an environment of the user;

a memory storing instructions; and

at least one portable processor device configured to execute the instructions to;

determine an existence of first face-identifying information in the real-time image data;

access at least one database storing facial information associated with selected individuals and audible indicators of identities of the individuals;

compare the first face-identifying information in the real-time image data with the stored facial information in the at least one database;

identify a match between the first face-identifying information in the real-time image data and the stored facial information in the at least one database;

based on the match, cause an audible indicator of an identity of an associated individual to be announced to the visually impaired user;

determine that the first face-identifying information related to the associated individual is no longer present in the real-time image data;

after the determination that the first face-identifying information related to the associated individual is rip longer present in the real-time image data, determine that new face-identifying information related to the associated individual appears in the real-time image data within a predetermined time period after the first face-identifying information was present in the real-time image data; and

based on the determination that the new face-identifying information related to the associated individual appears in the real-time image data within the predetermined time period, withhold an audible identification of the associated individual.

2. The apparatus of claim 1, wherein the at least one processor device is further configured to process the image data to identify a trigger associated with a user'"'"'s desire to audibly re-identify the associated individual during the predetermined period, and audibly re-identify the associated individual during the predetermined period.

3. The apparatus of claim 1, wherein the at least one processor device is further configured to repeat the audible indicator upon identifying a trigger associated with a user'"'"'s desire to repeat the audible indicator.

4. The apparatus of claim 3, wherein the at least one processor device is further configured to process the image data to identify the trigger associated with a user'"'"'s desire to repeat the audible indicator.

5. The apparatus of claim 4, wherein the trigger includes a pointing finger.

6. The apparatus of claim 1, wherein the at least one processor device is further configured to store facial information and an audible indicator for a non-recognized individual.

7. The apparatus of claim 6, wherein the at least one processor device is further configured to store the facial information and the audible indicator for the non-recognized individual upon identifying a trigger in the image data associated with a user'"'"'s desire to add the non-recognized individual to the at least one database.

8. The apparatus of claim 7, wherein the trigger includes a pointing finger.

9. The apparatus of claim 1, wherein the at least one processor device is further configured to store additional facial information for an individual whose facial image is already stored in the database.

10. The apparatus of claim 1, wherein the at least one processor device is further configured to capture an audio recording of an individual'"'"'s voice, and compare the captured audio recording to stored audio features of the associated individual'"'"'s voice to verify the match.

11. The apparatus of claim 1, wherein the at least one processor device is further configured to capture an audio recording of a non-recognized individual'"'"'s voice, compare the captured audio recording to stored audio features, and identify a match between the captured audio recording and stored audio features associated with an individual.

12. The apparatus of claim 1, wherein the at least one processor device is further configured to identify a match for two or more associated individuals in the real-time image data, and announce audible indicators associated with the two or more associated individuals according to a pre-defined rule.

13. The apparatus of claim 12, wherein the at least one processor device is further configured to determine an order in which to announce the audible indicators associated with the two or more individuals according to the pre-defined rule.

14. The apparatus of claim 1, wherein the at least one processor device is further configured to update the stored facial information in the at least one database with current facial information derived from the face-identifying information in the real-time image data upon identifying the match between the face-identifying information and the stored facial information.

15. The apparatus of claim 1, wherein the facial information of an individual, stored in the at least one database, is derived from multiple images of the individual.

16. The apparatus of claim 1, wherein the at east one processor device is further configured to provide feedback to the visually impaired user that no match was found between the face-identifying information and the stored facial information.

17. The apparatus of claim 1, wherein the image sensor is configured to capture images in various resolutions.

18. The apparatus of claim 17, wherein the at least one processor device is further configured to operate in a low power consumption mode by determining an existence of face-identifying information in image data taken at a resolution lower than a resolution of image data used to compare the face-identifying information in the real-time image data with the stored facial information in the at least one database.

19. The apparatus of claim 1, wherein the at least one processor device is further configured to implement a memory managing algorithm when comparing the face-identifying information with the stored facial information.

20. The apparatus of claim 19, wherein the at least one processor device is further configured to store statistical information associated with previous matches and to use the statistical information when implementing the memory managing algorithm.

21. The apparatus of claim 20, wherein the statistical information includes at least one of:

information about a frequency of specific matches, information about relationships between specific individuals, information about a time of specific matches, sound recording, and information about location of specific matches.

22. The apparatus of claim 1, wherein the at least one processor device is further configured to determine if the associated individual is looking in a direction of the visually impaired user, and if it is determined that the associated individual is looking in a direction of the visually impaired user, audibly identify the associated individual.

23. The apparatus of claim 1, wherein the at least one processor device is further configured to determine if the associated individual and at least one other individual are looking in a direction of the visually impaired user, and if it is determined that the associated individual and the at least one other individual are looking in a direction of the visually impaired user, audibly identify the associated individual and the at least one other individual.

24. A non-transitory computer readable medium storing computer implementable instructions for carrying out a method for identifying individuals in an environment of a user, the method comprising:

receiving real-time image data from a portable image sensor configured to be worn by the user and to capture real-time image data from the environment of the user;

determining an existence of first face-identifying information in the real-time image data;

accessing at least one database storing facial information associated with selected individuals and indicators associated with identities of the selected individuals;

comparing the first face-identifying information in the real-time image data with the stored facial information in the at least one database;

identifying a match between the first face-identifying information in the real-time image data and the stored facial information in the at least one database;

outputting to the user an indicator of an associated individual;

determining that the first face-identifying information related to the associated individual is no longer present in the real-time image data;

after the determination that the first face-identifying, information related to the associated individual is no longer present in the real-time image data, determining that new face-identifying information related to the associated individual appears in the real-time image data within a predetermined time period after the first face-identifying information was present in the real-time image data; and

based on the determination that the newface-identifying information related to the associated individual appears in the real-time image data within the predetermined time period, withholding outputting of the indicator of the associated individual.

25. The software product of claim 24, wherein upon identifying a trigger associated with a user'"'"'s desire to audibly re-identify the associated individual during the predetermined time period, re-outputting the indicator of the associated individual during the predetermined time period.

26. The software product of claim 24, wherein the indicator is an audible indicator.