Point-of-view object selection

• US 8,203,605 B1
• Filed: 05/11/2011
• Issued: 06/19/2012
• Est. Priority Date: 05/11/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US13/105,828
• Time in Patent Office: 405 Days
• Field of Search: None
• US Class Current: 348/115
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

Example methods and systems are disclosed that utilize optical-flow analysis to help recognize the performance of a predetermined user-action by a user of a wearable computer, such as a user-action for selecting an object within the user'"'"'s view. An example method may involve a wearable computing device: (a) receiving a video from a video camera of a wearable computing device: (b) determining optical flow for the video; (c) determining that an optical-flow differential exists at an area within the optical flow for the video; and (d) responsive to determining that the optical-flow differential exists at the area within the optical flow: (i) identifying an object that is located at the area where the optical-flow differential exists; (ii) performing a search on the object that is located at the area where the optical-flow differential exists; and (iii) causing a graphical display to display information that corresponds to results from the search.

Claims

1. A method comprising:

receiving a video from a video camera of a wearable computing device;

determining optical flow for the video;

determining that an optical-flow differential exists at an area within the optical flow for the video wherein determining that the optical-flow differential exists comprises;

determining that optical flow within the area is of substantially different magnitude than optical flow outside of the area; and

determining that at least a portion of the optical flow for the video has a curvature that is characteristic of a predetermined user-action associated with selection of the object; and

responsive to determining that the optical-flow differential exists at the area within the optical flow;

identifying an object that is located at the area where the optical-flow differential exists;

performing a search on the object that is located at the area where the optical-flow differential exists; and

causing a graphical display to display information that corresponds to results from the search.

2. The method of claim 1, wherein the optical-flow differential corresponds to a predetermined user-action of a user of the wearable computing device.

3. The method of claim 1:

wherein the optical flow within the area is of substantially lesser magnitude than the optical flow outside of the area; and

wherein determining that at least a portion of the optical flow for the video has the curvature that is characteristic of the predetermined user-action comprises determining that the characteristic curvature exists in the optical flow outside of the area.

4. The method of claim 3, wherein the predetermined user-action comprises the object remaining in substantially the same location within the user'"'"'s field of view as the user moves along a substantially circular path around the object.

5. The method of claim 1:

wherein the optical flow within the area is of substantially greater magnitude than the optical flow outside of the area; and

wherein determining that at least a portion of the optical flow for the video has the curvature that is characteristic of the predetermined user-action comprises determining that the characteristic curvature exists within the area.

6. The method of claim 5, wherein the predetermined user-action comprises fixation on the object while the object rotates.

7. The method of claim 1,

wherein the optical flow comprises a plurality of optical-flow vectors at a plurality of sample points in the video, and wherein determining the optical flow comprises generating optical-flow data comprising, for each of the sample points in the video:

(a) coordinates of the sample point, and (b) an indication of a magnitude and a direction of the optical-flow vector at the sample point, and

wherein determining that the optical-flow differential exists at the area within the optical flow for the video comprises;

determining a threshold magnitude such that a magnitude for each of a predetermined percentage of the optical-flow vectors is less than the threshold magnitude;

determining which of the optical-flow vectors have magnitudes less than the threshold magnitude; and

determining that the area includes at least a predetermined percentage of the optical-flow vectors that have a magnitude less than the threshold magnitude.

8. The method of claim 1,

wherein the optical flow comprises a plurality of optical-flow vectors at a plurality of sample points in the video,

wherein determining the optical flow comprises generating optical-flow data comprising, for each of the sample points in the video:

(a) coordinates of the sample point, and (b) an indication of a magnitude and a direction of the optical-flow vector at the sample point, and

wherein determining that an optical-flow differential exists at the area within the optical flow for the video comprises;

determining a threshold magnitude such that magnitudes of a predetermined percentage of the optical-flow vectors are greater than the threshold magnitude;

determining which of the optical-flow vectors have magnitudes greater than the threshold magnitude; and

determining that the area includes at least a predetermined percentage of the optical-flow vectors that have a magnitude greater than the threshold magnitude.

9. The method of claim 1, further comprising:

using video frames from the video to generate a three-dimensional reconstruction of the object from the video; and

wherein performing the search on the object comprises performing a three-dimensional search on the three-dimensional reconstruction of the object.

10. The method of claim 9, further comprising:

evaluating the three-dimensional reconstruction of the object in order to determine whether there is information that has not been captured, which could improve the three-dimensional reconstruction; and

if it is determined that information that has not been captured and could improve the three-dimensional reconstruction, then initiating a prompt to the user, wherein the prompt directs the user to perform a user-action that is likely to capture, in the video, at least some of the information could improve the three-dimensional reconstruction.

11. The method of claim 1, wherein determining that optical-flow differential exists comprises continually determining optical flow of the video in order to determine when the optical-flow differential exists at the area in the video.

12. The method of claim 1, wherein determining that the optical-flow differential exists at the area comprises:

detecting the optical-flow differential at the area in the video; and

determining that the optical-flow differential persists in the area for at least a predetermined period of time.

13. The method of claim 1, wherein the optical-flow differential is determined to exist between two frames of the video, and wherein identifying the object in the video that is located at the area of the optical-flow differential comprises:

applying an image-processing technique to a first frame of the two frames in order to visually isolate the object in the first frame; and

sending an object-identification request to another computing device, wherein the object-identification request comprises a copy of the first frame having the image-processing technique applied thereto.

14. A non-transitory computer readable medium having instructions stored thereon, the instructions comprising:

instructions for receiving video from a video camera of a wearable computing device;

instructions for determining optical flow for the video;

instructions for determining when an optical-flow differential exists at an area in the optical flow for the video by;

determining that optical flow within the area is of substantially different magnitude than optical flow outside of the area; and

determining that at least a portion of the optical flow for the video has a curvature that is characteristic of a predetermined user-action associated with selection of the object; and

instructions for, responsive to determining that the optical-flow differential exists;

identifying an object in the video that is located at the area of the optical-flow differential;

causing a search to be performed on the object that is located at the area of the optical-flow differential; and

causing a graphical display to display information that corresponds to results from the search.

15. The non-transitory computer readable medium of claim 14, wherein the optical-flow differential corresponds to a predetermined user-action of a user of the wearable computing device.

16. A wearable-computing system comprising:

an interface to a head-mounted display, wherein the head-mounted display is configured to display computer-generated information related to a point-of-view of a user;

an interface to a video camera configured to substantially track the point-of-view of the user; and

an on-board computing system configured to;

determine optical flow for video from the video camera;

determine when an optical-flow differential exists at an area in the optical flow for the video based on (a) a determination that optical flow within the area is of substantially different magnitude than optical flow outside of the area and (b) a determination that at least a portion of the optical flow for the video has a curvature that is characteristic of a predetermined user-action associated with selection of the object;

responsive to determining that the optical-flow differential exists;

identify an object in the video that is located at the area of the optical-flow differential;

perform a search on the object that is located at the area of the optical-flow differential; and

cause the head-mounted display to display graphical information that corresponds to results from the search.

17. The wearable-computing system of claim 16,

wherein the determination that the optical-flow differential exists at the area is dependent upon the determination that the optical flow within the area is of substantially different magnitude and the determination that at least a portion of the optical flow for the video has a curvature that is characteristic of the predetermined user-action.

18. The wearable-computing system of claim 16, wherein the on-board computing system configured to:

differentiate between a plurality of types of optical-flow differentials comprising at least a first type and a second type;

determine which type of optical-flow differential exists at the area; and

adjust a manner in which the search on the object is performed based on the determined type of the optical-flow differential.