Using HMD Camera Touch Button to Render Images of a User Captured During Game Play

• US 20180001198A1
• Filed: 06/16/2017
• Published: 01/04/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Frappiea, Phillip
• Granted Patent: US 10,471,353 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 13/493 Resuming a game, e.g. after pausing, malfunction or power failure
• CPC Class Codes: A63F 13/497 Partially or entirely replaying previous game actions
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Methods and systems for presenting an image of a user interacting with a video game includes providing images of a virtual reality (VR) scene of the video game for rendering on a display screen of a head mounted display (HMD). The images of the VR scene are generated as part of game play of the video game. An input provided at a user interface on the HMD received during game play is used to initiate a signal to pause the video game and to generate an activation signal to activate an image capturing device. The activation signal causes the image capturing device to capture an image of the user interacting in a physical space. The image of the user captured by the image capturing device during game play is associated with a portion of the video game that corresponds with a time when the image of the user was captured. The association causes the image of the user to be transmitted to the HMD for rendering on the display screen of the HMD.

Claims

1. A method for processing content for head mounted display (HMD) use, comprising:

providing images of a virtual reality (VR) scene of a video game for rendering on a display screen of the HMD, the images of the VR scene generated in response to a request for game play of the video game received from a user wearing the HMD;

receiving a user input provided at a user interface on the HMD, during game play of the video game;

in response to receiving the user input, pausing the game play of the video game, wherein the pausing includes storing a current location of the video game for later resuming game play;

generating a signal to activate an image capturing device to capture an image of the user interacting in a physical space, wherein the image capturing device is external to the HMD and is oriented toward the user;

processing the image of the user received from the image capturing device, wherein the processing includes associating the image of the user to a portion of the game play of the video game, the association allowing the image of the user to be shared with other users,

wherein operations of the method are performed by a processor of a computer that is communicatively connected to the HMD and is executing the video game.

2. The method of claim 1, wherein associating the image includes linking the image of the user to a specific video frame within a video clip of the game play,

wherein the video clip is generated from a recording of the game play of the video game that is stored in a local buffer during game play, the video clip including the portion of the game play that corresponds with a time frame when the image of the user was captured.

3. The method of claim 2, wherein a length of the video clip of the recorded game play is pre-defined or user-defined.

4. The method of claim 2, further includes sharing the video clip of the recorded game play with the associated image of the user with other users identified from a social network of the user or other users from a game network to which the user belongs or on a game network site, or on a website.

5. The method of claim 1, wherein processing the image includes associating one or more rendering characteristics to the image of the user and to the images of the VR scene, the rendering characteristics causing the images of the VR scene of the video game to fade out of view and the image of the user to bring into view.

6. The method of claim 5, wherein a speed of fading the VR scene out of view and bringing the image of the user into view is based on one or more attributes of the user input.

7. The method of claim 1, wherein the user input is a gesture input that includes a forward swipe, a button press, a single tap, a directional swipe, a double tap, a finger press, or any combinations thereof.

8. The method of claim 1, further includes,

receiving a second user input provided at the user interface of the HMD, the second user input is used to,

resume the game play of the video game by transitioning the display screen from rendering the image of the user to rendering the VR scene, wherein the resuming is effectuated by rewinding the video game from a current location by a pre-defined number of frames and restarting the video game from a point to where the video game was rewound.

9. The method of claim 8, wherein the second user input is a backward swipe or a directional swipe or a single tap or a double tap or a triple tap or a button press or a finger press or any combinations thereof.

10. The method of claim 1, wherein processing the image further includes,

formatting the image of the user so as to render the image in a first portion of the display screen of the HMD during game play of the video game and rendering image of the VR scene in a second portion of the display screen, the image of the user being presented for a pre-defined period of time;

and

after expiration of the pre-defined period of time, resuming rendering of the VR scene in the first portion of the display screen of the HMD.

11. The method of claim 1, wherein the image capturing device is a video camera and the image of the user is captured as a video stream, the sharing of the image includes sharing a portion of the video stream containing the image of the user with the video clip of the recorded game play, the portion of the video stream being linked to the video clip.

12. The method of claim 1, further includes adjusting the image of the user by flipping the image of the user horizontally so as to cause the image of the user to switch from a mirror view orientation to a reverse mirror view orientation.

13. The method of claim 1, further includes receiving a second image of the user interacting in the physical space, the second image captured by a forward facing camera of the HMD, the image from the forward facing camera and the image from the image capturing device used to generate a three-dimensional image of the user interacting in the physical space, the three-dimensional image of the user stored for the game play for sharing with other users or posting to a website.

14. A system, comprising:

a computing device including

a memory to store game code of a video game;

a game processor that is configured to,

execute an instance of the video game and generate video frames of a virtual reality scene of the video game for rendering on a display screen of a head mounted display (HMD) for viewing;

receive user interaction during game play of the video game, the user interaction used to drive interactivity of the video game;

receive a user input provided at an user interface of the HMD that is communicatively coupled to the computing device;

pause the game play of the video game, in response to the user input received from the HMD, and store a current location of the video game as a pointer to re-start the video game; and

an input analyzer that is configured to,

receive the user input provided at the user interface of the head mounted display (HMD) during game play of the video game, analyze the user input and forward the analyzed user input for further processing;

a signal generator that is configured to,

generate a signal to activate an image capturing device that is communicatively coupled to the computing device, in response to the user input received from the HMD, the activation enables the image capturing device to capture an image of a user interacting in a physical space; and

an image data processor that is configured to receive the image of the user and process the image of the user for rendering on a display screen of the HMD.

15. The system of claim 14, wherein the image data processor of the computing device is further configured to,

retrieve a video clip from a recording of game play stored in a local buffer of the memory, the video clip identifying a portion of the VR scene of the video game that corresponds to the user input;

and

link the image of the user to the video clip by identifying a video frame of the VR scene captured in the video clip that corresponds to the user input received from the HMD, and associating the image of the user to the identified video frame, wherein the video frame of the VR scene is identified using a timeline.

16. The system of claim 14, further includes,

a first communication device that is used to,

transmit the video frames of the VR scene to the HMD for rendering on the display screen associated with the HMD;

receive interaction data from the HMD, the interaction data includes user input provided at the user interface of the HMD and user interaction provided through the HMD as part of game play;

receive user interaction provided at a controller during game play;

forward the user interaction provided through the HMD and the user interaction provided at the controller to the game processor for driving interactivity of the video game; and

forward the user input to the input analyzer, the user input used to generate a signal for activating the image capturing device.

17. The system of claim 14, further includes,

a second communication device that is configured to,

establish a communication channel with the image capturing device so as to communicate the signal to activate the image capturing device;

and

receive an image of the user interacting in the physical space from the image capturing device.

18. The system of claim 14, wherein the image data processor is further configured to generate a rendering signal that is used to adjust a speed at which the image of the user is brought into view when rendered in a region on the display screen of the HMD and the portion of the VR scene that corresponds to the region is faded out of view.

19. The system of claim 15, further includes a social media application interface that is used to identify social contacts of a user, the image data processor configured to share the video clip linked with the image of the user, with the identified social contacts of the user.

20. The system of claim 14, wherein the user interface is a touch pad that is configured to receive gesture input.

21. The system of claim 14, wherein the image capturing device is a video camera.

22. A method for processing content for head mounted display (HMD) use, comprising:

providing images of a virtual reality (VR) scene of a video game for rendering on a display screen of the HMD, the images of the VR scene generated in response to a request for game play of the video game received from a user wearing the HMD;

tracking gaze direction of the user using a gaze detection camera that is directed toward one or both eyes of the user wearing the HMD;

when the gaze direction of the user is directed toward a pre-defined area on the display screen of the HMD and last for a pre-defined amount of time, pausing the game play of the video game, wherein the pausing includes storing a current location of the video game for later resuming game play;

generating a signal to activate an image capturing device, wherein the image capturing device is external to the HMD and is oriented to capture an image of the user interacting in a physical space;

and

rendering the image of the user at the pre-defined area on the display screen of the HMD,

wherein operations of the method are performed by a processor of a computer that is communicatively connected to the HMD and is executing the video game.