Device Control Utilizing Optical Flow

• US 20160118015A1
• Filed: 01/04/2016
• Published: 04/28/2016
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Johnson, Michael P., Stamer, Thad E.
• Granted Patent: US 10,032,429 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2380/10 Automotive applications
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

A computing device includes an interface configured to receive image data that is indicative of a field-of-view (FOV) that is associated with a head-mountable display (HMD). The computing device also includes a computing system configured to determine optical flow from the image data and to determine, based on the optical flow, whether or not the HMD is associated with operation of a vehicle. Further, the computing device is configured to control the HMD to display information in a first mode, if the HMD is associated with the operation of the vehicle, and to control the HMD to display information in a second mode, if the HMD is not associated with the operation of the vehicle.

Claims

1. A computing device comprising:

a camera configured to receive image data that is indicative of a field-of-view (FOV) that is associated with a head-mountable display (HMD);

one or more sensors disposed on the HMD; and

a computing system configured to;

determine optical flow of the FOV from the image data;

determine, based on an output from the one or more sensors, that the HMD is positioned at an operator location in a vehicle as opposed to a passenger location in the vehicle;

determine, based on the optical flow and the determination regarding the position of the HMD in the vehicle, that the HMD is associated with operation of the vehicle;

control the HMD to display information in a first mode, if the HMD is associated with the operation of the vehicle; and

control the HMD to display information in a second mode different from the first mode, if the HMD is not associated with the operation of the vehicle.

2. The computing device of claim 1, wherein the computing system is configured to determine from the optical flow that the HMD is associated with operation of the vehicle by determining that one or more of foreground pixels or background pixels of the image data of the FOV are moving toward the HMD at a rate of speed consistent with being inside a moving vehicle.

3. The computing device of claim 1, wherein the first mode provides a less obstructed display of information than the second mode.

4. The computing device of claim 1, wherein the first mode provides navigational information.

5. The computing device of claim 1, wherein the one or more sensors include one or more of an accelerometer, a gyroscope, a compass, or a global positioning system sensor.

6. The computing device of claim 1, wherein the one or more sensors include a gaze tracking sensor.

7. The computing device of claim 1, wherein the computing system is configured to control the HMD in response to the determination that the HMD is associated with operation of the vehicle by modifying communication settings of the HMD, wherein the communication settings include one or more of text messaging settings, phone call settings, or internet settings.

8. The computing device of claim 1, wherein the computing system is configured to determine optical flow from the image data by compensating for optical flow contributions caused by rotation of the camera.

9. A computer-implemented method comprising:

receiving image data that is indicative of a field-of-view (FOV) that is associated with a head-mountable display (HMD), wherein the HMD includes a camera, and wherein the camera captures the image data;

processing the image data to determine optical flow of the FOV;

receiving sensor data from an output of one or more sensors, wherein the one or more sensors are disposed on the HMD;

identifying an activity from the optical flow and the received sensor data; and

controlling the HMD in response to the identified activity, wherein controlling the HMD includes adjusting a user interface displayed by the HMD.

10. The method of claim 9, further comprising processing the received sensor data to determine that the HMD is positioned at an operator location in a vehicle as opposed to a passenger location in the vehicle, wherein identifying the activity utilizes the determination regarding the position of the HMD in the vehicle, and wherein the activity is associated with operation of the vehicle.

11. The method of claim 9, further comprising:

identifying, using the optical flow, a sub-state of the activity; and

controlling the HMD in response to the identified sub-state,

and wherein the identified sub-state includes a sub-state of the operation of the vehicle.

12. The method of claim 9, wherein the sensor data is captured by one or more of an accelerometer, a gyroscope, a compass, or a global positioning system sensor.

13. The method of claim 9, wherein the sensor data is captured by a gaze tracking sensor.

14. The method of claim 9, wherein processing the image data to determine the optical flow includes compensating for optical flow contributions caused by rotation of the camera.

15. The method of claim 9, further comprising controlling the HMD in response to the identified activity by modifying communication settings of the HMD, wherein the communication settings include one or more of text messaging settings, phone call settings, or internet settings.

16. A wearable computing device comprising:

a display;

an imaging device;

one or more sensors; and

a computing system coupled to the display, the imaging device, and the one or more sensors, wherein the computing system is configured to;

determine optical flow by processing image data captured by the imaging device,

identify, based on an output from the one or more sensors, that the imaging device is positioned at an operator location in a vehicle as opposed to a passenger location in the vehicle,

identify an activity from the optical flow and the identification regarding the position of the imaging device in the vehicle, wherein the activity is associated with operation of the vehicle, and

control the display to display information in response to the identified activity.

17. The wearable computing device of claim 16, wherein the computing system is further configured to measure rotation of the imaging device, and to determine the optical flow by processing the image data and by compensating for optical flow contributions caused by the measured rotation of the imaging device,

18. The wearable computing device of claim 17, wherein the computing system being configured to determine optical flow by compensating for optical flow contributions caused by the measured rotation of the imaging device further includes the computing system being configured to:

measure a rotational angular velocity of the imaging device,

model a predicted optical flow contribution based on the measured rotational angular velocity, and

removing the predicted optical flow contribution from an optical flow field using a vector sum of a calculated optical flow field and an anticipated optical flow field.

19. The wearable computing device of claim 17, wherein the computing system being configured to determine optical flow by compensating for optical flow contributions caused by the measured rotation of the imaging device further includes the computing system being configured to:

determine if the measured rotation of the imaging device exceeds a given threshold, and

based on the measured rotation of the imaging device exceeding the given threshold, suspending the processing of the image data to determine the optical flow.

20. The wearable computing device of claim 16, wherein the one or more sensors include one or more of an accelerometer, a gyroscope, a compass, a gaze tracking sensor, or a global positioning system sensor.