Point-of-View Object Selection

• US 20120287123A1
• Filed: 05/09/2012
• Published: 11/15/2012
• Est. Priority Date: 05/11/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad
• Granted Patent: US 9,429,990 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

Example methods and systems may be implemented by a wearable computer to utilize optical-flow analysis to recognize a predetermined user-action, such as selecting an object. An example method may involve a wearable computing device: (a) receiving video data from a video camera of a wearable computing device: (b) determining optical flow for the video data; (c) determining that an optical-flow differential exists at an area within the optical flow for the video data; and (d) responsive to determining that the optical-flow differential exists at the area: (i) identifying an object that is located at the area where the optical-flow differential exists; (ii) performing a search on the object; and (iii) causing a graphical display to display information that corresponds to results from the search.

Claims

1. A method comprising:

receiving video data;

determining optical flow for the video data;

determining that an optical-flow differential exists at an area within the optical flow for the video data wherein determining that the optical-flow differential exists comprises;

determining that optical flow within the area is of substantially different magnitude than optical flow outside of the area; and

determining that at least a portion of the optical flow for the video data has a curvature that is characteristic of a predetermined user-action associated with selection of the object; and

responsive to determining that the optical-flow differential exists at the area within the optical flow;

identifying an object that is located at the area where the optical-flow differential exists;

performing a search on the object that is located at the area where the optical-flow differential exists; and

causing a graphical display to display information that corresponds to results from the search.

2. The method of claim 1:

wherein the optical flow within the area is of substantially lesser magnitude than the optical flow outside of the area; and

wherein determining that at least a portion of the optical flow for the video data has the curvature that is characteristic of the predetermined user-action comprises determining that the characteristic curvature exists in the optical flow outside of the area.

3. The method of claim 2, wherein the predetermined user-action comprises the object remaining in substantially the same location within the user'"'"'s field of view as the user moves along a substantially circular path around the object.

4. The method of claim 1:

wherein the optical flow within the area is of substantially greater magnitude than the optical flow outside of the area; and

wherein determining that at least a portion of the optical flow for the video data has the curvature that is characteristic of the predetermined user-action comprises determining that the characteristic curvature exists within the area.

5. The method of claim 4, wherein the predetermined user-action comprises fixation on the object while the object rotates.

6. The method of claim 1,

wherein the optical flow comprises a plurality of optical-flow vectors at a plurality of sample points in the video data, and wherein determining the optical flow comprises generating optical-flow data comprising, for each of the sample points in the video data:

(a) coordinates of the sample point, and (b) an indication of a magnitude and a direction of the optical-flow vector at the sample point, and

wherein determining that the optical-flow differential exists at the area within the optical flow for the video data comprises;

determining a threshold magnitude such that a magnitude for each of a predetermined percentage of the optical-flow vectors is less than the threshold magnitude;

determining which of the optical-flow vectors have magnitudes less than the threshold magnitude; and

determining that the area includes at least a predetermined percentage of the optical-flow vectors that have a magnitude less than the threshold magnitude.

7. The method of claim 1,

wherein the optical flow comprises a plurality of optical-flow vectors at a plurality of sample points in the video data,

wherein determining the optical flow comprises generating optical-flow data comprising, for each of the sample points in the video data:

(a) coordinates of the sample point, and (b) an indication of a magnitude and a direction of the optical-flow vector at the sample point, and

wherein determining that an optical-flow differential exists at the area within the optical flow for the video data comprises;

determining a threshold magnitude such that magnitudes of a predetermined percentage of the optical-flow vectors are greater than the threshold magnitude;

determining which of the optical-flow vectors have magnitudes greater than the threshold magnitude; and

determining that the area includes at least a predetermined percentage of the optical-flow vectors that have a magnitude greater than the threshold magnitude.

8. The method of claim 1, further comprising:

using video frames from the video data to generate a three-dimensional reconstruction of the object from the video data; and

wherein performing the search on the object comprises performing a three-dimensional search on the three-dimensional reconstruction of the object.

9. The method of claim 8, further comprising:

evaluating the three-dimensional reconstruction of the object in order to determine whether there is information that has not been captured, which could improve the three-dimensional reconstruction; and

if it is determined that information that has not been captured and could improve the three-dimensional reconstruction, then initiating a prompt to the user, wherein the prompt directs the user to perform a user-action that is likely to capture, in the video data, at least some of the information could improve the three-dimensional reconstruction.

10. The method of claim 1, wherein determining that optical-flow differential exists comprises continually determining optical flow of the video data in order to determine when the optical-flow differential exists at the area in the video data.

11. The method of claim 1, wherein determining that the optical-flow differential exists at the area comprises:

detecting the optical-flow differential at the area in the video data; and

determining that the optical-flow differential persists in the area for at least a predetermined period of time.

12. The method of claim 1, wherein the optical-flow differential is determined to exist between two frames of the video data, and wherein identifying the object in the video data that is located at the area of the optical-flow differential comprises:

applying an image-processing technique to a first frame of the two frames in order to visually isolate the object in the first frame; and

sending an object-identification request to another computing device, wherein the object-identification request comprises a copy of the first frame having the image-processing technique applied thereto.

13. A non-transitory computer readable medium having instructions stored thereon, the instructions comprising:

instructions for receiving video data;

instructions for determining optical flow for the video data;

instructions for determining when an optical-flow differential exists at an area in the optical flow for the video data by;

determining that optical flow within the area is of substantially different magnitude than optical flow outside of the area; and

determining that at least a portion of the optical flow for the video data has a curvature that is characteristic of a predetermined user-action associated with selection of the object; and

instructions for, responsive to determining that the optical-flow differential exists;

identifying an object in the video data that is located at the area of the optical-flow differential;

causing a search to be performed on the object that is located at the area of the optical-flow differential; and

causing a graphical display to display information that corresponds to results from the search.

14. The non-transitory computer readable medium of claim 13:

wherein the optical flow within the area is of substantially lesser magnitude than the optical flow outside of the area; and

wherein the instructions for determining that at least a portion of the optical flow for the video data has the curvature that is characteristic of the predetermined user-action comprise instructions for determining that the characteristic curvature exists in the optical flow outside of the area.

15. The non-transitory computer readable medium of claim 14, wherein the predetermined user-action comprises the object remaining in substantially the same location within the user'"'"'s field of view as the user moves along a substantially circular path around the object.

16. The non-transitory computer readable medium of claim 13:

wherein the optical flow within the area is of substantially greater magnitude than the optical flow outside of the area; and

wherein the instructions for determining that at least a portion of the optical flow for the video data has the curvature that is characteristic of the predetermined user-action comprise instructions for determining that the characteristic curvature exists within the area.

17. The non-transitory computer readable medium of claim 16, wherein the predetermined user-action comprises fixation on the object while the object rotates.

18. A system comprising:

an interface to a display of a computing device;

an interface to a video camera configured to record video data; and

an on-board computing system operable to;

determine optical flow for video data;

determine when an optical-flow differential exists at an area in the optical flow for the video data based on (a) a determination that optical flow within the area is of substantially different magnitude than optical flow outside of the area and (b) a determination that at least a portion of the optical flow for the video data has a curvature that is characteristic of a predetermined user-action associated with selection of the object;

responsive to determining that the optical-flow differential exists;

identify an object in the video data that is located at the area of the optical-flow differential;

perform a search on the object that is located at the area of the optical-flow differential; and

cause the head-mounted display to display graphical information that corresponds to results from the search.

19. The system of claim 18, wherein the determination that the optical-flow differential exists at the area is dependent upon the determination that the optical flow within the area is of substantially different magnitude and the determination that at least a portion of the optical flow for the video data has a curvature that is characteristic of the predetermined user-action.

20. The system of claim 18, wherein the on-board computing system is operable to:

differentiate between a plurality of types of optical-flow differentials comprising at least a first type and a second type;

determine which type of optical-flow differential exists at the area; and

adjust a manner in which the search on the object is performed based on the determined type of the optical-flow differential.

21. The system of claim 18:

wherein the optical flow within the area is of substantially lesser magnitude than the optical flow outside of the area; and

wherein the portion of the optical flow for the video data that is determined to have the curvature that is characteristic of the predetermined user-action comprises a portion of the optical flow that is substantially outside of the area.

22. The system of claim 21, wherein the predetermined user-action comprises the object remaining in substantially the same location within the user'"'"'s field of view as the user moves along a substantially circular path around the object.

23. The system of claim 18:

wherein the optical flow within the area is of substantially greater magnitude than the optical flow outside of the area; and

wherein the portion of the optical flow for the video data that is determined to have the curvature that is characteristic of the predetermined user-action comprises a portion of the optical flow that is substantially within the area.

24. The system of claim 23, wherein the predetermined user-action comprises fixation on the object while the object rotates.