Head-mountable apparatus and systems

• US 9,811,908 B2
• Filed: 06/11/2014
• Issued: 11/07/2017
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Raghoebardajal, Sharwin Winesh, Bickerstaff, Ian Henry
• Primary Examiner(s): Bekele, Mekonen
• Application Number:
  US14/896,859
• Publication Number:
  US 20160155231A1
• Time in Patent Office: 1,245 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A head mountable display (HMD) is provided that has an upward facing camera associated with a wide angle (such as a fisheye) lens arranged to capture images of the environment of the HMD. The HMD also includes an image comparator arranged to derive an indication of yaw of the HMD from images captured by the upward facing camera at different points in time. The image comparator may be configured to perform two or more detection techniques to detect HMD yaw, and to combine the results of the two or more detection techniques. The HMD may also include a tilt detector. Here, the image comparator may be configured to ignore one or more portions of a captured image in dependence upon a current tilt of the HMD as detected by the tilt detector.

Claims

1. A head mountable display (HMD) comprising:

an upward facing camera associated with a wide angle lens configured to capture images of an environment of the HMD;

an image comparator operatively coupled to the upward facing camera, the image comparator being arranged to derive an indication of yaw of the HMD from images captured by the upward facing camera at different points in time; and

a processor configured to identify feature points in the images captured at different points in time, and to detect positions in the images of the identified feature points so as to provide an indication of rotational movement of the HMD between two respective images, in which the processor is configured to detect indications of the rotational movement from a plurality of pairs of image positions, each pair representing opposite viewpoints of the camera, and in which the processor is further configured to derive the indication of rotational movement from one or more pairs exhibiting a lowest difference between the viewpoints of that pair.

2. An HMD according to claim 1, in which the lens is a fisheye lens.

3. An HMD according to claim 2, in which the lens has a field of view of at least 140°

.

4. An HMD according to claim 1, in which the images are successively captured images.

5. An HMD according claim 1, configured to apply an image warping process to convert the captured images into a linear image format.

6. An HMD according to claim 1, wherein the processor is further configured to apply a correlation process as between the images captured at different points in time by the upward facing camera.

7. An HMD according to claim 1, in which the image comparator is configured to perform two or more detection techniques to detect HMD yaw, and to combine the results of the two or more detection techniques.

8. An HMD according to claim 1, further comprising:

a tilt detector;

and in which the image comparator is configured to ignore one or more portions of a captured image in dependence upon a current tilt of the HMD.

9. A system, comprising:

a head mountable display (HMD) including;

an upward facing camera associated with a wide angle lens configured to capture images of an environment of the HMD;

an image comparator operatively coupled to the upward facing camera, the image comparator being arranged to derive an indication of yaw of the HMD from images captured by the upward facing camera at different points in time; and

a processor configured to identify feature points in the images captured at different points in time, and to detect positions in the images of the identified feature points so as to provide an indication of rotational movement of the HMD between two respective images, in which the processor is configured to detect indications of the rotational movement from a plurality of pairs of image positions, each pair representing opposite viewpoints of the camera, and in which the processor is further configured to derive the indication of rotational movement from one or more pairs exhibiting a lowest difference between the viewpoints of that pair; and

one or more of a base device and an intermediate device operatively coupled to the HMD.

10. A method of operation of a head mountable display (HMD) having an upward facing camera associated with a wide angle lens arranged to capture images of the environment of the HMD, the method comprising:

deriving, by one or more processors, an indication of yaw of the HMD from images captured by the upward facing camera at different points in time;

identifying feature points in the images captured at the different points in time;

detecting positions in the images of the identified feature points so as to provide an indication of rotational movement of the HMD between two respective images;

detecting indications of the rotational movement from a plurality of pairs of image positions, each pair representing opposite viewpoints of the camera; and

deriving the indication of rotational movement from one or more pairs exhibiting a lowest difference between the viewpoints of that pair.

11. A non-transitory computer-readable storage medium on which computer readable instructions are stored, the instructions, when executed by one or more processors, causing the one or more processors to execute a method of operation of a head mountable display (HMD) having an upward facing camera associated with a wide angle lens arranged to capture images of the environment of the HMD, the method comprising:

deriving an indication of yaw of the HMD from images captured by the upward facing camera at different points in time;

identifying feature points in the images captured at the different points in time;

detecting positions in the images of the identified feature points so as to provide an indication of rotational movement of the HMD between two respective images;

detecting indications of the rotational movement from a plurality of pairs of image positions, each pair representing opposite viewpoints of the camera; and

deriving the indication of rotational movement from one or more pairs exhibiting a lowest difference between the viewpoints of that pair.

12. A head mountable display (HMD) comprising:

an upward facing camera associated with a wide angle lens configured to capture images of an environment of the HMD; and

an image comparator operatively coupled to the upward facing camera, the image comparator being arranged to derive an indication of yaw of the HMD from images captured by the upward facing camera at different points in time;

in which the image comparator is configured to detect inter-image scaling in two or more regions of the captured images, and not to use a region, in the detection of HMD yaw, in which inter-image scaling of more than a threshold scaling has been detected.

13. An HMD according to claim 12, comprising a processor configured to apply a correlation process as between the images captured at different points in time by the upward facing camera.

14. An HMD according to claim 12, further comprising:

a tilt detector;

and in which the image comparator is configured to ignore one or more portions of a captured image in dependence upon a current tilt of the HMD.

15. An HMD according to claim 12, in which the images are successively captured images.

16. An HMD according claim 12, configured to apply an image warping process to convert the captured images into a linear image format.

17. A system, comprising:

a head mountable display (HMD) including;

an upward facing camera associated with a wide angle lens configured to capture images of an environment of the HMD; and

an image comparator operatively coupled to the upward facing camera, the image comparator being arranged to derive an indication of yaw of the HMD from images captured by the upward facing camera at different points in time;

in which the image comparator is configured to detect inter-image scaling in two or more regions of the captured images, and not to use a region, in the detection of HMD yaw, in which inter-image scaling of more than a threshold scaling has been detected; and

one or more of a base device and an intermediate device operatively coupled to the HMD.

18. A method of operation of a head mountable display (HMD) having an upward facing camera associated with a wide angle lens arranged to capture images of the environment of the HMD, the method comprising:

deriving, by one or more processors, an indication of yaw of the HMD from images captured by the upward facing camera at different points in time;

detecting inter-image scaling in two or more regions of the captured images; and

not using a region, in the detection of HMD yaw, in which inter-image scaling of more than a threshold scaling has been detected.