APPARATUS AND METHOD FOR USING BACKGROUND CHANGE TO DETERMINE CONTEXT

• US 20140267651A1
• Filed: 12/20/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: OrCam Technologies Ltd
• Original Assignee: OrCam Technologies Ltd
• Inventors: Wexler, Yonatan, Shashua, Amnon
• Granted Patent: US 10,339,406 B2
• Time in Patent Office: Days
• US Class Current: 348/62
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/243 by compensating for image skew or non-uniform image deformations
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/144 using a slot moved over the image; using discrete sensing elements at predetermined points; using automatic curve following means
• CPC Class Codes: G06V 30/1478 of characters or characters lines
• CPC Class Codes: G06V 30/226 of cursive writing
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: G06V 30/418 : Document matching, e.g. of document images
• CPC Class Codes: G06V 30/43 : Editing text-bitmaps, e.g. alignment, spacing; Semantic analysis of bitmaps of text without OCR
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G08B 3/10 : using electric transmission; using electromagnetic transmission
• CPC Class Codes: G08B 6/00 : Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G09B 21/00 : Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B
• CPC Class Codes: G09B 21/001 : Teaching or communicating with blind persons G09B21/02 - G09B21/06 take precedence
• CPC Class Codes: G09B 21/003 : using tactile presentation of the information, e.g. Braille displays
• CPC Class Codes: G09B 21/006 : using audible presentation of the information
• CPC Class Codes: G10L 13/00 : Speech synthesis; Text to speech systems
• CPC Class Codes: H04N 23/50 : Constructional details
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Devices and a method are provided for providing feedback to a user. In one implementation, the method comprises obtaining a plurality of images from an image sensor. The image sensor is configured to be positioned for movement with the user'"'"'s head. The method further comprises monitoring the images, and determining whether relative motion occurs between a first portion of a scene captured in the plurality of images and other portions of the scene captured in the plurality of images. If the first portion of the scene moves less than at least one other portion of the scene, the method comprises obtaining contextual information from the first portion of the scene. The method further comprises providing the feedback to the user based on at least part of the contextual information.

Claims

1. An apparatus for providing feedback to a user, the apparatus comprising:

an image sensor configured to be positioned for movement with a head of the user as the head moves, and to capture real time images from an environment of the user; and

at least one processor device for determining contextual information based on the real time images, the processor device being configured to;

monitor a plurality of images captured by the image sensor to determine whether relative motion occurs between a first portion of a scene captured in the plurality of images and other portions of the scene captured in the plurality of images;

if the first portion of the scene moves less than at least one other portion of the scene, obtain contextual information from the first portion of the scene; and

provide the feedback to the user based on at least part of the contextual information.

2. The apparatus of claim 1, wherein the first portion of the scene is associated with an object in a background of the scene, and the at least one other portion of the scene is associated with an object in a foreground of the scene.

3. The apparatus of claim 2, wherein the contextual information associated with the object in the background is used to identify the object in the foreground.

4. The apparatus of claim 2, wherein the at least one processor device is further configured to retrieve contextual information from a stationary object in the background when the object in the foreground is moving.

5. The apparatus of claim 4, wherein the stationary object includes a branded product and the contextual information includes information associated with the branded product.

6. The apparatus of claim 1, wherein the first portion of the scene is associated with an object in a foreground and the at least one other portion of the scene is associated with an object in the background.

7. The apparatus of claim 6, wherein the at least one processor device is further configured to retrieve contextual information from a moving object in a foreground of the scene when the user'"'"'s head tracks the moving object.

8. The apparatus of claim 7, wherein the moving object includes a public transportation vehicle and the contextual information includes information associated with the public transportation vehicle.

9. The apparatus of claim 1, wherein obtaining contextual information includes performing optical character recognition on an area of at least one of the plurality of images associated with the first portion of the scene.

10. The apparatus of claim 1, wherein obtaining contextual information includes comparing the first portion of the scene with stored image data.

11. The apparatus of claim 1, wherein the contextual information is used for selecting an action to execute from a plurality of context-based actions.

12. The apparatus of claim 1, wherein the image sensor is further configured to capture real time images at a plurality of resolutions, and the contextual information is used for selecting which of the plurality of resolutions to use.

13. The apparatus of claim 1, wherein the at least one processor device is configured to select between a plurality of differing processing schemes based on the contextual information obtained.

14. The apparatus of claim 13, wherein the plurality of differing processing schemes includes a processing scheme to identify an object, a processing scheme to identify an individual, a processing scheme to audibly reading a text, and a processing scheme to continuously monitor an object.

15. An apparatus for providing feedback to a user, the apparatus comprising:

an image sensor for capturing real time images from an environment of the user, the image sensor configured to be positioned for movement with a head of the user as the head moves such that an aiming direction of the image sensor falls within a field of view of the user; and

at least one processor device for determining contextual information based on the real time images, the processor device being configured to;

monitor the real time images captured by the image sensor to automatically determine whether an object in the user'"'"'s field of view lingers within a plurality of the real time images;

identify at least one object determined to linger within the plurality of real time images as an object of interest;

obtain contextual information associated with the object of interest; and

provide the feedback to the user based on at least part of the contextual information.

16. The apparatus of claim 15, wherein the at least one processor device is further configured to use the contextual information to identify the object of interest.

17. The apparatus of claim 16, wherein the object of interest includes an individual or a branded product.

18. The apparatus of claim 15, wherein the at least one processor device is further configured to initiate performance of an action associated with the object of interest, and suspend performance of the action when the object of interest moves outside a field of view of the image sensor.

19. The apparatus of claim 18, wherein the object of interest includes text and the action includes optical character recognition.

20. The apparatus of claim 15, wherein the at least one processor device is further configured to obtain additional contextual information associated with at least one object other than the object of interest, and to use at least part of the additional contextual information to output the feedback.

21. The apparatus of claim 15, wherein the at least one processor device is further configured to obtain the contextual information when the object of interest lingers within the plurality of real time images for a predetermined period of time.

22. The apparatus of claim 21, wherein the predetermined period of time depends on a characteristic associated with the object of interest.

23. A method for providing feedback to a user, the method comprising:

obtaining from an image sensor a plurality of images, wherein the image sensor is configured to be positioned for movement with a head of the user;

monitoring the plurality of images captured by the image sensor;

determining whether relative motion occurs between a first portion of a scene captured in the plurality of images and other portions of the scene captured in the plurality of images;

if the first portion of the scene moves less than the at least one other portion of the scene, obtaining contextual information from the first portion of the scene; and

providing the feedback to the user based on at least part of the contextual information.

24. A software product stored on a non-transitory computer readable medium and comprising data and computer implementable instructions for carrying out the method of claim 23.

25. An apparatus for processing real time images of an environment of a user, the apparatus comprising:

a camera including an image sensor configured to be positioned for movement with a head of the user as the head moves, and to capture real time images; and

at least one processor device for determining contextual information based on the real time images, the processor device being configured to;

monitor a plurality of images captured by the image sensor to determine whether relative motion occurs between a first portion of a scene captured in the plurality of images and other portions of the scene captured in the plurality of images;

if the first portion of the scene moves less than at least one other portion of the scene, obtain contextual information from the first portion of the scene; and

adjust at least one parameter associated with the camera based on at least part of the contextual information.

26. The apparatus of claim 25, wherein the contextual information used to identify an object in the first portion of the scene.

27. The apparatus of claim 25, wherein determining the contextual information includes comparing an object in the first portion of the scene with stored image data.

28. The apparatus of claim 25, wherein the at least one parameter associated with the camera includes at least one of:

a focus point, an exposure time, an aperture size, light sensitivity, an image resolution, and a frame rate.

29. The apparatus of claim 25, wherein the first portion of the scene is associated with an object in a background of the scene, and the at least one other portion of the scene is associated with an object in a foreground of the scene.

30. The apparatus of claim 25, wherein the first portion of the scene is associated with an object in a foreground and the at least one other portion of the scene is associated with an object in the background.