Camera based safety mechanisms for users of head mounted displays

• US 9,873,049 B2
• Filed: 03/29/2017
• Issued: 01/23/2018
• Est. Priority Date: 09/30/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger
• Primary Examiner(s): D'AGOSTINO, PAUL ANTHONY
• Application Number:
  US15/473,469
• Publication Number:
  US 20170203213A1
• Time in Patent Office: 300 Days
• Field of Search: None
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/537 using indicators, e.g. showing the condition of a game character on screen
• CPC Class Codes: A63F 2300/306 for displaying a marker associated to an object or location in the game field
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L

Abstract

Methods and systems for warning a user of a head mounted display (HMD) during game play of a video game. A game is executed causing interactive scenes of the game to be transmitted for rendering on a display portion of the HMD. Coordinates of the HMD are determined in a three-dimensional space of a real-world environment in which the user wearing the HMD is present, to identify a current position of the user. A movement of the HMD is determined during execution of the game by identifying a change in one or more coordinates of the HMD in the three-dimensional space. When it is determined that the user is approaching a boundary of an interaction space in a real-world environment, a warning signal is conveyed to the HMD to indicate proximity of the user to the boundary of the interaction space.

Claims

1. A method, comprising:

executing a game on a client device, the execution causing interactive scenes of the game to be transmitted to a head mounted display (HMD) for rendering on a display portion, the HMD when worn by a user is configured to block a view to a real-world environment, the HMD being in wireless communication with the client device;

determining coordinates of the HMD in a three-dimensional space of a real-world environment in which the user is present, the coordinates used to identify a current position of the user wearing the HMD;

monitoring a movement of the HMD during execution of the game, the movement of the HMD determined by identifying a change in one or more coordinates of the HMD in the three-dimensional space;

evaluating the change in the one or more coordinates of the HMD worn by the user to determine when the user is approaching a boundary of an interaction space defined in the real-world environment, wherein the interaction space is an area in the real-world environment in which the user is present, the boundary is defined as an edge of the interaction space; and

conveying a warning signal for rendering on the HMD, wherein the warning signal is indicative of a proximity of the user to the boundary of the interaction space, the warning signal includes an image of an outline representing the boundary of the interaction space for presenting with at least a part of the interactive scenes of the game,

wherein the method operations are performed by a processor.

2. The method of claim 1, further includes adjusting an intensity of the warning signal based on the evaluating, wherein the intensity is adjusted to increase as the user continues to approach the boundary of the interaction space and to decrease as the user moves away from the boundary of the interaction space toward a center of the interaction space.

3. The method of claim 1, wherein the warning signal is further provided in an audio format or a textual format or a haptic format, or any combinations thereof.

4. The method of claim 1, wherein the warning signal further includes instructions to direct the user away from the boundary of the interaction space in the real-world environment and toward a center of the interaction space.

5. The method of claim 1, further includes detecting one or more real-world objects disposed along the boundary, the warning signal providing an image of the one or more real-world objects that the user is approaching.

6. The method of claim 1, wherein determining coordinates further includes,

receiving images of the HMD in the three-dimensional space of the real-world environment captured by one or more cameras that are external to the HMD;

and

computing the coordinates of the HMD in the three-dimensional space.

7. The method of claim 6, further includes receiving secondary images of the HMD in the three-dimensional space of the real-world environment captured by one or more forward facing cameras coupled to the HMD, while the game is being executed, the secondary images analyzed to verify the coordinates of the HMD in the three-dimensional space.

8. The method of claim 1, wherein the image of an outline representing the boundary is presented in a form of a grid.

9. A method, comprising:

receiving interactive scenes of a game at a head mounted display (HMD) for rendering on a display screen of the HMD, the game being executed on a client device, the HMD when worn by a user is configured to block a view to a real-world environment, data for the interactive scenes being communicated by the client device to the HMD using wireless communication;

capturing images of an interaction space of a real-world environment in a vicinity of the user using a forward facing camera coupled to the HMD, the images being captured while the user is interacting with the game that is providing the interactive scenes;

forwarding the images to the client device for processing to determine coordinates of the HMD in the interaction space, wherein the coordinates identify a current position of the HMD and of the user wearing the HMD;

capturing additional images of the interaction space of the real-world environment, the additional images forwarded to the client device for evaluating a change in one or more coordinates of the HMD during execution of the game, wherein the change in the coordinates identifies movement of the user in the interaction space of the real-world environment; and

receiving a warning signal from the client device for rendering on the display portion of the HMD, when it is determined that the user is approaching a boundary of the interaction space, the warning signal is indicative of proximity of the user to the boundary of the interaction space, wherein the warning signal received from the client device includes an image of an outline representing the boundary of the interaction space for rendering with at least a part of the interactive scenes of the game,

wherein operations of the method are performed by a processor of the HMD.

10. The method of claim 9, further includes,

receiving secondary images of the interaction space captured by one or more cameras that are external to the HMD;

and

forwarding the secondary images to the client device for verifying the movement of the user.

11. The method of claim 9, wherein rendering the warning signal includes dynamically adjusting an intensity of the image of the outline representing the boundary based on the current position of the user, the warning signal includes information for dynamically adjusting the intensity of the image of the outline, such that the intensity of the image of the outline is set to increase as the user continues to approach the boundary of the interaction space and decrease as the user moves away from the boundary of the interaction space.

12. The method of claim 9, wherein the warning signal is further provided in audio format or haptic format or textual format, or any combinations thereof, and wherein information provided with the warning signal causes a dynamic adjustment to an intensity of the warning signal, such that the intensity of the warning signal rendered at the HMD is set to increase when the user continues to approach the boundary of the interaction space and decrease when the user move away from the boundary toward a center of the interaction space.