HEAD-MOUNTABLE APPARATUS AND SYSTEMS

• US 20160140764A1
• Filed: 06/11/2014
• Published: 05/19/2016
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Myatt, Darren, Bickerstaff, Ian Henry, Benson, Simon Mark, Raghoebardajal, Sharwin Winesh
• Granted Patent: US 10,198,866 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A head mountable display (HMD) includes a camera operable to capture images of a peripheral and/or control device in use by a wearer of the HMD. A detector of the HMD is configured to detect occlusions in a captured image of the peripheral and/or control device. And an image renderer of the HMD is configured to render a virtual version of the peripheral and/or control device for display to the HMD wearer and to render a representation of a user'"'"'s hand at a position of a detected occlusion.

Claims

1. A head mountable display (HMD) system comprising:

a camera configured to capture images of a peripheral and/or control device in use by a wearer of the HMD;

a detector configured to detect occlusions in a captured image of the peripheral and/or control device; and

an image renderer configured to render a virtual version of the peripheral and/or control device for display to the HMD wearer and to render a representation of a user'"'"'s hand at a position of a detected occlusion.

2. An HMD system according to claim 1, in which the representation of the user'"'"'s hand is a virtual representation of the user'"'"'s hand.

3. An HMD system according to claim 2, in which:

the detector is configured to detect at least an angular orientation of the peripheral and/or control device from a shape of the peripheral and/or control device in a captured image; and

the image renderer is configured to render the virtual representation in dependence upon the detected angular orientation.

4. An HMD system according to claim 1, in which:

the detector is configured to detect pixels corresponding to skin colours around a detected occlusion, and

the image renderer is configured to render parts of the captured image corresponding to the user'"'"'s hand for display to the HMD wearer.

5. An HMD system according to claim 2, in which the image renderer is configured to select a hand to be rendered based at least in part on either or both of user profile data indicating the user'"'"'s dominant hand, and the position of the occlusion relative to the peripheral and/or control device.

6. An HMD system according to claim 1, in which the HMD is configured to display computer game material.

7. An HMD system according to claim 1, in which the detector is configured to detect one or both of markings and illumination on the peripheral and/or control device in the captured images.

8. An HMD system according to claim 1, in which the detector is configured to detect false occlusions by comparing a detected occlusion with sensor data relating to the peripheral and/or control device.

9. An HMD system according to claim 1, in which the detector is configured to detect false occlusions by detecting inconsistencies with control data from one or more other peripheral and/or control devices indicating the user touching the other peripheral and/or control devices.

10. An HMD system according to claim 1, in which the detector is configured to detect false occlusions by detecting images of the peripheral and/or control device captured by another camera.

11. An HMD system according to claim 1, in which the HMD system is configured to enable or disable haptic feedback for a given peripheral and/or control device in dependence upon whether an occlusion of the given peripheral and/or control device is detected.

12. A computer games system, comprising:

a head mountable display (HMD system including;

a camera configured to capture images of a peripheral and/or control device in use by a wearer of the HMD;

a detector configured to detect occlusions in a captured image of the peripheral and/or control device; and

an image renderer configured to render a virtual version of the peripheral and/or control device for display to the HMD wearer and to render a representation of a user'"'"'s hand at a position of a detected occlusion; and

one or more other devices selected from the list consisting of;

a games console and a break-out box.

13. A computer games system according to claim 12, in which one or more of the games console and the break-out box comprises a camera configured to detect images of the peripheral and/or control device.

14. A computer games system according to claim 12, in which one or more of the games console and the break-out box comprises a light source configured to illuminate at least part of a rear aspect, with reference to a user of the HMD, of the peripheral and/or control device.

15. A computer games system according to claim 14, in which the illuminator is an infra-red illuminator.

16. A method of operation of a head mountable display (HMD) comprising:

capturing images of a peripheral and/or control device in use by a wearer of the HMD;

detecting occlusions in a captured image of the peripheral and/or control device;

rendering, by one or more processors, a virtual version of the peripheral and/or control device for display to the HMD wearer; and

rendering, by the one or more processors, a representation of a user'"'"'s hand at a position of a detected occlusion.

17. (canceled)

18. A non-transitory computer-readable storage medium which stores computer instructions thereon, the instructions, when executed by one or more processors, causes the one or more processors to perform a method of operation of a head mountable display (HMD) comprising:

capturing images of a peripheral and/or control device in use by a wearer of the HMD;

detecting occlusions in a captured image of the peripheral and/or control device;

rendering, by one or more processors, a virtual version of the peripheral and/or control device for display to the HMD wearer; and

rendering, by the one or more processors, a representation of a user'"'"'s hand at a position of a detected occlusion.