Systems and methods for audibly presenting textual information included in image data

• US 8,891,817 B2
• Filed: 12/20/2013
• Issued: 11/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: OrCam Technologies Ltd
• Original Assignee: OrCam Technologies Ltd
• Inventors: Wexler, Yonatan, Shashua, Amnon
• Primary Examiner(s): Liew, Alex
• Application Number:
  US14/137,384
• Publication Number:
  US 20140270398A1
• Time in Patent Office: 333 Days
• Field of Search: 382/103, 382124-127
• US Class Current: 382/100
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/243 by compensating for image skew or non-uniform image deformations
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/144 using a slot moved over the image; using discrete sensing elements at predetermined points; using automatic curve following means
• CPC Class Codes: G06V 30/1478 of characters or characters lines
• CPC Class Codes: G06V 30/226 of cursive writing
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: G06V 30/418 : Document matching, e.g. of document images
• CPC Class Codes: G06V 30/43 : Editing text-bitmaps, e.g. alignment, spacing; Semantic analysis of bitmaps of text without OCR
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G08B 3/10 : using electric transmission; using electromagnetic transmission
• CPC Class Codes: G08B 6/00 : Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G09B 21/00 : Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B
• CPC Class Codes: G09B 21/001 : Teaching or communicating with blind persons G09B21/02 - G09B21/06 take precedence
• CPC Class Codes: G09B 21/003 : using tactile presentation of the information, e.g. Braille displays
• CPC Class Codes: G09B 21/006 : using audible presentation of the information
• CPC Class Codes: G10L 13/00 : Speech synthesis; Text to speech systems
• CPC Class Codes: H04N 23/50 : Constructional details
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

An apparatus and method are provided for identifying and audibly presenting textual information within captured image data. In one implementation, a method is provided for audibly presenting text retrieved from a captured image. According to the method, at least one image of text is received from an image sensor, and the text may include a first portion and a second portion. The method includes identifying contextual information associated with the text, and accessing at least one rule associating the contextual information with at least one portion of text to be excluded from an audible presentation associated with the text. The method further includes performing an analysis on the at least one image to identify the first portion and the second portion, and causing the audible presentation of the first portion.

Claims

1. A system for audibly presenting text retrieved from a captured image, the system comprising:

at least one processor device configured to;

receive at least one image of text to be audibly presented, the text including a first portion and a second portion;

identify contextual information associated with the text;

access at least one rule associating the contextual information with at least one portion of text to be excluded from an audible presentation associated with the text;

perform an analysis on the at least one image to identify the first portion and the second portion; and

cause the audible presentation, wherein the audible presentation includes the first portion and excludes the second portion.

2. The system of claim 1, wherein the second portion includes at least one of:

a name associated with the author of the text, a page number, and a date.

3. The system of claim 1, wherein the second portion includes at least one predefined word.

4. The system of claim 1, wherein the first portion and the second portion are part of a same paragraph.

5. The system of claim 1, wherein the contextual information includes information about a type of object associated with the text.

6. The system of claim 1, wherein the contextual information includes information about a type of document associated with the text.

7. The system of claim 6, wherein the contextual information includes predefined information about preferences of a user associated with the type of document.

8. The system of claim 1, wherein the contextual information includes information indicative of a location of the user.

9. The system of claim 1, wherein the first portion includes a plurality of sub-portions, and the at least one rule further associates the contextual information with a presenting order of the plurality of sub-portions.

10. The system of claim 1, wherein the analysis includes initiating an optical character recognition process and application of the at least one rule to identify the first portion and the second portion in the text.

11. The system of claim 1, wherein the analysis includes detecting different zones in the text, each zone being associated with differing logical roles, and applying the at least one rule to identify the first portion and the second portion in the text.

12. The system of claim 1, wherein the at least one processor device is further configured to initiate an optical character recognition process to recognize at least the first portion.

13. The system of claim 12, wherein the at least one processor device is further configured to recognize characters defined by 10 or less pixels.

14. The system of claim 12, wherein the at least one processor device is further configured to recognize characters defined by 6 or less pixels.

15. An apparatus for audibly presenting text retrieved from a captured image, the apparatus comprising;

an image sensor configured to capture images from an environment of a user;

at least one processor device configured to;

receive at least one image of text to be audibly presented, the text including a first portion and a second portion;

identify contextual information associated with the text;

access at least one rule associating the contextual information with at least one portion of text to be excluded from an audible presentation associated with the text;

perform an analysis on the at least one image to identify the first portion and the second portion; and

cause the audible presentation, wherein the audible presentation includes the first portion and excludes second portion.

16. The apparatus of claim 15, wherein the image sensor is further configured to be connected to glasses worn by the user to enable the image sensor to move with a head of the user.

17. The apparatus of claim 15, wherein the image sensor is further configured to be movable with a head of the user and an aiming direction of the image sensor substantially coincides with a field of view of the user.

18. The apparatus of claim 15, wherein the at least one processor device is further configured to automatically stop the audible presentation when the user looks away from the text.

19. The apparatus of claim 18, wherein the at least one processor device is further configured to automatically resume the audible presentation when the user looks back at the text.

20. The apparatus of claim 18, wherein the at least one processor device is further configured to resume the audible presentation from a previous stopping point when the user looks back at the text.

21. The apparatus of claim 15, wherein the at least one processor device is further configured to determine an existence of a trigger in the at least one image, the trigger being associated with a desire of the user to hear the text read aloud.

22. A method for audibly presenting text retrieved from a captured image, the method comprising:

receiving at least one image of text to be audibly presented, the text including a first portion and a second portion;

identifying contextual information associated with the text;

accessing at least one rule associating the contextual information with at least one portion of text to be excluded from an audible presentation associated with the text;

performing an analysis on the at least one image to identify the first portion and the second portion; and

causing the audible presentation, wherein the audible presentation includes the first portion and excludes second portion.

23. A software product stored on a non-transitory computer readable medium and comprising data and computer implementable instructions for carrying out the method of claim 22.