Use of comparative sensor data to determine orientation of head relative to body

• US 9,268,136 B1
• Filed: 09/28/2012
• Issued: 02/23/2016
• Est. Priority Date: 09/28/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad Eugene, Johnson, Michael Patrick
• Primary Examiner(s): Ghebretinsae, Temesgh
• Assistant Examiner(s): Jansen, II, Michael J
• Application Number:
  US13/631,454
• Time in Patent Office: 1,243 Days
• Field of Search: 345/156, 345 7- 9
• US Class Current: 1/1
• CPC Class Codes: G01B 7/00 Measuring arrangements characterised by the use of electric or magnetic techniques
• CPC Class Codes: G01C 17/02 Magnetic compasses
• CPC Class Codes: G01C 19/00 Gyroscopes; Turn-sensitive devices using vibrating masses; Turn-sensitive devices without moving masses; Measuring angular rate using gyroscopic effects
• CPC Class Codes: G01P 15/00 Measuring acceleration; Measuring deceleration; Measuring shock, i.e. sudden change of acceleration
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface

Abstract

Methods and systems are described that involve a head-mountable display (HMD) or an associated device determining the orientation of a person'"'"'s head relative to their body. To do so, example methods and systems may compare sensor data from the HMD to corresponding sensor data from a tracking device that is expected to move in a manner that follows the wearer'"'"'s body, such a mobile phone that is located in the HMD wearer'"'"'s pocket.

Claims

1. A computer-implemented method comprising:

detecting, by a computing device, sensor data that is indicative of an association between movement of a tracking device and body movement;

in response to detecting the sensor data that is indicative of the association between movement of the tracking device and body movement;

determining a direction of the body movement that is associated with the movement of the tracking device;

determining, based on the direction of the body movement, a forward-backward body axis of a body; and

determining a base orientation of a tracking device relative to the forward-backward body axis;

determining a first orientation of a head-mountable device (HMD) relative to the tracking device, wherein the HMD and the tracking device are separate devices; and

determining a first head orientation relative to the body based on both;

(a) the first orientation of the HMD relative to the tracking device and (b) the base orientation of the tracking device relative to the forward-backward body axis, wherein determining the first head orientation relative to the body comprises offsetting the first orientation of the HMD relative to the tracking device by the base orientation of the tracking device relative to the forward-backward body axis.

2. The method of claim 1, wherein the computing device is the HMD.

3. The method of claim 1, wherein the computing device is the tracking device.

4. The method of claim 1, wherein the tracking device is a mobile phone.

5. The method of claim 1, wherein detecting the sensor data that is indicative of the association between movement of the tracking device and body movement comprises:

receiving sensor data from at least one of the HMD and the tracking device, wherein the sensor data is indicative of movement; and

determining that the sensor data is characteristic of movement along the forward-backward body axis.

6. The method of claim 5, wherein determining that the sensor data is characteristic of movement along the forward-backward body axis comprises determining that the sensor data is characteristic of walking.

7. The method of claim 5, wherein the sensor data comprises at least one of:

(a) gyroscope data associated with the HMD, (b) accelerometer data associated with the HMD, (c) gyroscope data associated with the tracking device, and (d) accelerometer data associated with the tracking device.

8. The method of claim 1, wherein determining a forward-backward body axis of a body comprises determining a direction of forward body movement.

9. The method of claim 1, wherein determining a base orientation of the tracking device relative to the forward-backward body axis comprises determining an angle between a forward-facing direction of the tracking device and the directional component along the forward-backward body axis.

10. The method of claim 1, wherein the first orientation of the HMD relative to the tracking device is determined based on both:

(a) magnetometer data associated with the HMD and (b) magnetometer data associated with the tracking device.

11. The method of claim 10, wherein determining the first orientation of the HMD relative to the tracking device comprises:

determining a first orientation of the HMD relative to magnetic north based on the magnetometer data associated with the HMD;

determining a first orientation of the tracking device relative to magnetic north based on the magnetometer data associated with the tracking device; and

determining the first orientation of the HMD relative to the tracking device based on a difference between (a) the first orientation of the HMD relative to magnetic north and (b) the first orientation of the tracking device relative to magnetic north.

12. The method of claim 1, further comprising:

subsequently detecting a calibration event, wherein detecting the calibration event comprises receiving an indication that the tracking device has moved in relation to the body; and

in response to detecting the calibration event;

analyzing the sensor data to detect subsequent sensor data that is indicative of an association between second movement of a tracking device and second body movement; and

in response to detecting the subsequent sensor data that is indicative of the association, re-determining, based on a direction of the second body movement, the base orientation of the tracking device relative to the forward-backward body axis.

13. The method of claim 1, further comprising causing a display of the HMD to display graphics based on the first head orientation relative to the body.

14. The method of claim 1, further comprising initiating a computing action based on the first head orientation relative to the body.

15. The method of claim 1, wherein determining a first head orientation relative to the body comprises determining a rotation of the head relative to the forward-backward body axis.

16. The method of claim 1, wherein determining a first head orientation relative to the body comprises determining two or more of:

(a) a rotation of the head relative to the forward-backward body axis, (b) a pitch of the head relative to an upward-downward body axis and (c) a yaw of the head relative to the forward-backward body axis and the upward-downward body axis.

25. The method of claim 1, wherein detecting sensor data that is indicative of the association between movement of the tracking device and body movement comprises determining that movement of at least one of the tracking device and the HMD is characteristic of walking.

26. The method of claim 1, further comprising:

analyzing the sensor data for any occurrence of a calibration event, wherein the detecting of the sensor data that is indicative of the association between the movement of a tracking device and the body movement is a calibration event.

17. A non-transitory computer readable medium having stored therein instructions that are executable to cause a computing device to perform functions comprising:

detecting sensor data that is indicative of an association between movement of a tracking device and body movement;

in response to detecting the sensor data that is indicative of the positional association between movement of the tracking device and body movement;

determining a direction of the body movement that is associated with the movement of the tracking device;

determining a forward-backward body axis of a body based on the direction of the body movement; and

determining a base orientation of a tracking device relative to the forward-backward body axis;

determining a first orientation of a head-mountable device (HMD) relative to the tracking device, wherein the HMD and the tracking device are separate devices; and

determining a first head orientation relative to the body based on both;

(a) the first orientation of the HMD relative to the tracking device and (b) the base orientation of the tracking device relative to the forward-backward body axis, wherein determining the first head orientation relative to the body comprises offsetting the first orientation of the HMD relative to the tracking device by the base orientation of the tracking device relative to the forward-backward body axis.

18. The non-transitory computer readable medium of claim 17, wherein the first orientation of the HMD relative to the tracking device is determined based on both:

(a) magnetometer data associated with the HMD and (b) magnetometer data associated with the tracking device.

19. The non-transitory computer readable medium of claim 18, wherein determining the first orientation of the HMD relative to the tracking device comprises:

determining a first orientation of the HMD relative to magnetic north based on the magnetometer data associated with the HMD;

determining a first orientation of the tracking device relative to magnetic north based on the magnetometer data associated with the tracking device; and

determining the first orientation of the HMD relative to the tracking device based on a difference between (a) the first orientation of the HMD relative to magnetic north and (b) the first orientation of the tracking device relative to magnetic north.

20. The non-transitory computer readable medium of claim 19, wherein determining a first head orientation relative to the body comprises determining two or more of:

(a) a rotation of the head relative to the forward-backward body axis, (b) a pitch of the head relative to an upward-downward body axis and (c) a yaw of the head relative to the forward-backward body axis and the upward-downward body axis.

21. A computing system comprising:

a non-transitory computer readable medium;

program instructions stored on the non-transitory computer readable medium and executable by at least one processor to;

detect first sensor data that is indicative of both (a) a device arrangement indicative of an association between movement of a tracking device and body movement and (b) an alignment of a forward-backward axis of the body with a direction of movement associated with a head-mountable device (HMD);

in response to detecting the first sensor data;

determine the direction of the body movement that is associated with the movement of the tracking device;

determine, based on the direction of the body movement, a forward-backward body axis of a body; and

determine a base orientation of a tracking device relative to the forward-backward body axis;

subsequently determine a first orientation of a head-mountable device (HMD) relative to the tracking device, wherein the HMD and the tracking device are separate devices; and

determine a first head orientation relative to the body based on both;

(a) the first orientation of the HMD relative to the tracking device and (b) the base orientation of the tracking device relative to the forward-backward body axis, wherein the determination of the first head orientation relative to the body comprises offsetting the first orientation of the HMD relative to the tracking device by the base orientation of the tracking device relative to the forward-backward body axis.

22. The computing system of claim 21, wherein the computing system is implemented in or takes the form of the HMD.

23. The computing system of claim 21, wherein the first orientation of the HMD relative to the tracking device is determined based on both:

(a) magnetometer data associated with the HMD and (b) magnetometer data associated with the tracking device.

24. The computing system of claim 21, wherein the determined first head orientation relative to the body comprises two or more of:

(a) a rotation of the head relative to the forward-backward body axis, (b) a pitch of the head relative to an upward-downward body axis and (c) a yaw of the head relative to the forward-backward body axis and the upward-downward body axis.