Image processing for head mounted display devices

• US 9,910,275 B2
• Filed: 07/16/2015
• Issued: 03/06/2018
• Est. Priority Date: 05/18/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Gibbs, Simon J., Liot, Anthony S., Song, Yu, Hirase, Yoshiya
• Primary Examiner(s): Sharifi-Tafreshi, Koosha
• Application Number:
  US14/801,574
• Publication Number:
  US 20160341959A1
• Time in Patent Office: 964 Days
• Field of Search: 345 8, 345630, 345633
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0143 the two eyes not being equipped with identical nor symmetrical optical devices
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: H04N 7/142 : Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

Various devices, arrangements and methods for managing communications using a head mounted display device are described. In one aspect, tracking data is generated at least in part by one or more sensors in a head mounted display (HMD) device. The tracking data indicates one or more facial movements of a user wearing the HMD device. A patch image is obtained based on the tracking data. The patch image is merged with a facial image. Various embodiments relate to the HMD device and other methods for generating and using the patch and facial images.

Claims

1. A head mounted display (HMD) device, comprising:

a device body arranged to cover a first portion of a face of a user such that eyes of the user are hidden from view, wherein the HMD device, when worn by the user, is arranged to leave a second portion of the face of the user viewable;

one or more sensors positioned on the device body that are arranged to track one or more of eye movement, eyelid movement and eyebrow movement of the user;

a network interface unit arranged to receive a composite image of a face of another user wearing another HMD device that covers a first portion of the face of the other user including eyes of the other user, wherein the composite image is obtained by merging a patch image with a facial image of the other user and with a background patch image, the patch image being formed based on eye movement, eyelid movement and eyebrow movement of the other user, and wherein the background patch image is of a background of the other user that is behind non-facial regions covered by the other HMD device; and

a display positioned on the device body that is arranged to display an image including the composite image for the user when the HMD device is worn by the user.

2. The HMD device of claim 1 further comprising one or more of an electromyography (EMG) sensor, a camera arranged to track upper facial features, a camera arranged to track eye rotation and an infrared light source, and an electrooculography (EOG) sensor arranged to track eye rotation and eyelid movement.

3. The HMD device of claim 1 further comprising an electromyography (EMG) sensor that is pressed flush against a portion of the face of the user when the HMD device is worn by the user, wherein the EMG sensor is arranged to detect eyebrow movement of the user.

4. The HMD device of claim 1 wherein the network interface unit is arranged to transmit one or more composite images to an external device wherein the one or more composite images indicate the user who is wearing the HMD device and further indicate the tracked movements that underlie the HMD device and that were tracked by the one or more sensors.

5. An HMD device as recited in claim 1 wherein the patch image is formed based on an adjusted 3-D model of at least a portion of the face of the other user, the 3-D model being adjusted based on the tracked movements of the other user.

6. An HMD device as recited in claim 1 further comprising:

a light sensor positioned on the device body that is arranged to detect light level in the ambient environment; and

a light emitter positioned on the interior of the device body that is arranged to emit light to illuminate the first portion of the face of the user to assist the one or more sensors.