Camera Based Safety Mechanisms for Users of Head Mounted Displays

• US 20160214016A1
• Filed: 03/31/2016
• Published: 07/28/2016
• Est. Priority Date: 09/30/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger
• Granted Patent: US 9,908,049 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/537 using indicators, e.g. showing the condition of a game character on screen
• CPC Class Codes: A63F 2300/306 for displaying a marker associated to an object or location in the game field
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L

Abstract

Methods and systems are provided for warning a user of a head mounted display during gameplay of a video game. A game is executed causing interactive scenes of the game to be rendered on a display portion of a head mounted display (HMD) worn by a user. Images of a physical environment in a vicinity of the user are received from a forward facing camera of the HMD, while the game is being executed. The images are analyzed to determine if the user is approaching one or more objects in the physical environment. Data is sent to be rendered on the display portion of the HMD while rendering the interactive scenes of the game. The data provides information to avoid interaction with the one or more objects in the physical environment.

Claims

1. A method, comprising:

executing a game, the execution causing interactive scenes of the game to be rendered on a display portion of a head mounted display (HMD) configured to be worn by a user;

receiving images of a physical environment in a vicinity of the user from a forward facing camera coupled to the HMD, while the game is being executed;

analyzing the images of the physical environment received from the forward facing camera of the HMD to determine if the user is approaching one or more objects in the physical environment;

sending data to be rendered on the display portion of the HMD while rendering the interactive scenes of the game, the data provides information to avoid interaction with the one or more objects in the physical environment,

wherein the method operations are performed by a processor.

2. The method of claim 1, further includes receiving secondary images of the physical environment from one or more cameras that are external to the HMD, the secondary images analyzed to determine relative position of the user wearing the HMD, wherein the relative position of the user is determined with reference to the one or more objects in the physical environment.

3. The method of claim 1, wherein the analyzing is performed by the processor of the HMD.

4. The method of claim 1, wherein the analyzing is performed by the processor of a computing system that is communicatively connected to the HMD, the computing system executing game code of the game, wherein results from analyzing is used to affect data being sent to the HMD for rendering.

5. The method of claim 1, wherein the analyzing is performed by the processor of a server that is communicatively connected to the HMD over a network, the server executing game code of the game, wherein results from analyzing is used to affect data being sent to the HMD for rendering.

6. The method of claim 1, wherein analyzing further includes monitoring change in relative position of the user wearing the HMD, the monitoring causes change in the data sent to the HMD for rendering.

7. The method of claim 1, wherein the information includes a descriptive context of the physical environment condition detected in the vicinity of the user wearing the HMD.

8. The method of claim 1, wherein the information sent for rendering includes a warning identifying severity level of a safety condition detected in the physical environment in a vicinity of the user wearing the HMD.

9. The method of claim 8, wherein when the warning is a low level warning, the information sent includes images to augment the interactive scenes currently rendering on the display portion of the HMD.

10. The method of claim 8, wherein when the warning is a low level warning, the information sent includes a signal to transition the display portion of the HMD from the interactive scenes to the images of the physical environment, wherein the transition is performed gradually.

11. The method of claim 8, wherein when the warning is a high level warning, the information sent includes a signal to interrupt the interactive scenes of the game.

12. The method of claim 11, wherein the information provided in the data includes descriptive context of the safety condition detected in the physical environment in the vicinity of the user wearing the HMD.

13. The method of claim 8, wherein the information sent includes a signal to stop rendering the interactive scenes of the game.

14. The method of claim 1, wherein the information is provided as an overlay for rendering with the interactive scenes of the game, wherein the information provided for overlay may be in any one of audio format, video format, haptic format, static graphic format, animated graphic format or any combinations thereof.

15. The method of claim 1, wherein the information that is sent includes instructions to guide the user away from the one or more objects of the physical environment.

16. A method, comprising:

executing an application, the execution causing content of the application to be rendered on a display portion of a head mounted display (HMD) configured to be worn by a user;

receiving images of a physical environment in a vicinity of the user from a forward facing camera coupled to the HMD, while the application is being executed;

analyzing the images of the physical environment received from the forward facing camera of the HMD to determine relative position of the user wearing the HMD in relation to one or more objects in the physical environment;

sending data to be rendered on the display portion of the HMD while rendering the content of the application, the data provides information to avoid interaction with the one or more objects in the physical environment.

17. The method of claim 16, wherein operations of receiving of images of the physical environment, analyzing the images and sending data are handled by an operating system (OS) of the HMD.

18. The method of claim 16, wherein the operations of executing of the application, receiving of images of the physical environment, analyzing the images and sending data are executed by a processor of the HMD.

19. The method of claim 16, wherein the operations of executing of the application, receiving of images of the physical environment, analyzing the images and sending data are executed by a processor of a computing device that is communicatively coupled to the HMD.

20. The method of claim 16, wherein the operations of executing of the application, receiving of images of the physical environment, analyzing the images and sending data are executed by a processor of a server that is connected to the HMD via a network.

21. The method of claim 17, wherein the OS of the HMD is further configured to enable processing interaction with the application.

22. A method for executing content to be presented on a head mounted display (HMD) by a computing device, the HMD being associated an operating system (OS), the OS enables processing of an application for rendering the content, comprising:

during execution of the application by the computing device, receiving images of a physical environment in a vicinity of the user from a forward facing camera of the HMD;

analyzing, by code of the OS, the images of the physical environment received from the forward facing camera of the HMD to determine a position of the user wearing the HMD in relation to one or more objects in the physical environment; and

sending data, by the OS, for rendering on the display portion of the HMD while rendering the content of the application, the data provides information regarding proximity of the user to the one or more objects in the physical environment.

23. The method of claim 22, wherein the computing device is part of the HMD, or is a game console that is connected to the HMD, or is a personal computer that is connected to the HMD, or is a server that is connected to the HMD over a network.

24. The method of claim 22, wherein operations of receiving of images of the physical environment, analyzing the images and sending data are handled by the OS that is associated with the HMD.