Camera based safety mechanisms for users of head mounted displays

• US 9,908,049 B2
• Filed: 03/31/2016
• Issued: 03/06/2018
• Est. Priority Date: 09/30/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger
• Primary Examiner(s): D'AGOSTINO, PAUL ANTHONY
• Application Number:
  US15/088,000
• Publication Number:
  US 20160214016A1
• Time in Patent Office: 705 Days
• Field of Search: None
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/537 using indicators, e.g. showing the condition of a game character on screen
• CPC Class Codes: A63F 2300/306 for displaying a marker associated to an object or location in the game field
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L

Abstract

Methods and systems are provided for warning a user of a head mounted display during gameplay of a video game. A game is executed causing interactive scenes of the game to be rendered on a display portion of a head mounted display (HMD) worn by a user. Images of a physical environment in a vicinity of the user are received from a forward facing camera of the HMD, while the game is being executed. The images are analyzed to determine if the user is approaching one or more objects in the physical environment. Data is sent to be rendered on the display portion of the HMD while rendering the interactive scenes of the game. The data provides information to avoid interaction with the one or more objects in the physical environment.

Claims

1. A method, comprising:

executing a game by a computing system having a processor, the execution causing interactive scenes of the game to be rendered on a display portion of a head mounted display (HMD) configured to be worn by a user;

receiving images of a physical environment in a vicinity of the user from a forward facing camera coupled to the HMD, while the game is being executed;

analyzing the images of the physical environment received from the forward facing camera of the HMD to determine if the user is approaching one or more objects in the physical environment;

sending data to be rendered on the display portion of the HMD while rendering the interactive scenes of the game, the data provides information to avoid interaction with the one or more objects in the physical environment, wherein the information provided is based on a safety condition detected in the physical environment in a vicinity of the user wearing the HMD, the information includes a signal to gradually transition the display portion of the HMD out from rendering interactive scenes,

wherein the method operations are performed by the processor.

2. The method of claim 1, further includes receiving secondary images of the physical environment from one or more cameras that are external to the HMD, the secondary images analyzed to determine relative position of the user wearing the HMD, wherein the relative position of the user is determined with reference to the one or more objects in the physical environment.

3. The method of claim 1, wherein the analyzing is performed by the processor of the HMD.

4. The method of claim 1, wherein the analyzing is performed by the processor of the computing system that is communicatively connected to the HMD, the computing system executing game code of the game, wherein results from analyzing is used to affect data being sent to the HMD for rendering.

5. The method of claim 1, wherein the analyzing is performed by the processor of a server that is communicatively connected to the HMD over a network, the server executing game code of the game, wherein results from analyzing is used to affect data being sent to the HMD for rendering.

6. The method of claim 1, wherein analyzing further includes monitoring change in relative position of the user wearing the HMD, the monitoring causes change in the data sent to the HMD for rendering.

7. The method of claim 1, wherein the information includes a descriptive context of the physical environment condition detected in the vicinity of the user wearing the HMD.

12. The method of claim 1, wherein the information is provided in audio format, or a video format, or a haptic format, or a static graphic format, or an animated graphic format or any two or more combinations thereof.

13. The method of claim 1, wherein the information that is sent includes instructions to guide the user away from the one or more objects of the physical environment.

8. A method, comprising:

executing a game by a computing system having a processor, the execution causing interactive scenes of the game to be rendered on a display portion of a head mounted display (HMD) configured to be worn by a user;

receiving images of a physical environment in a vicinity of the user from a forward facing camera coupled to the HMD, while the game is being executed;

analyzing the images of the physical environment received from the forward facing camera of the HMD to determine if the user is approaching one or more objects in the physical environment; and

sending data to be rendered on the display portion of the HMD while rendering the interactive scenes of the game, the data provides information to avoid interaction with the one or more objects in the physical environment,

wherein the information sent for rendering includes a warning identifying a severity level of a safety condition detected in the physical environment in a vicinity of the user wearing the HMD, wherein when the warning is a low level warning, the information sent includes a signal to transition the display portion of the HMD from the interactive scenes to the images of the physical environment, and wherein the transition is performed gradually.

11. The method of claim 8, wherein the information sent includes a signal to stop rendering the interactive scenes of the game.

9. A method, comprising:

executing a game by a computing system having a processor, the execution causing interactive scenes of the game to be rendered on a display portion of a head mounted display (HMD) configured to be worn by a user;

receiving images of a physical environment in a vicinity of the user from a forward facing camera coupled to the HMD, while the game is being executed;

analyzing the images of the physical environment received from the forward facing camera of the HMD to determine if the user is approaching one or more objects in the physical environment; and

sending data to be rendered on the display portion of the HMD while rendering the interactive scenes of the game, the data provides information to avoid interaction with the one or more objects in the physical environment,

wherein the information sent for rendering includes a warning identifying a severity level of a safety condition detected in the physical environment in a vicinity of the user wearing the HMD, and

wherein when the warning is a high level warning, the information sent includes a signal to interrupt the interactive scenes of the game.

10. The method of claim 9, wherein the information provided in the data includes descriptive context of the safety condition detected in the physical environment in the vicinity of the user wearing the HMD.

14. A method, comprising:

executing an application by a processor, the execution causing content of the application to be rendered on a display portion of a head mounted display (HMD) configured to be worn by a user;

receiving images of a physical environment in a vicinity of the user from a forward facing camera coupled to the HMD, while the application is being executed;

analyzing the images of the physical environment received from the forward facing camera of the HMD to determine relative position of the user wearing the HMD in relation to one or more objects in the physical environment;

sending data to the HMD while rendering the content of the application, the data provides information for interrupting the rendering of the content by the HMD to avoid interaction with the one or more objects in the physical environment.

15. The method of claim 14, wherein operations of receiving images of the physical environment, analyzing the images and sending data are handled by an operating system (OS) of the HMD.

16. The method of claim 14, wherein the processor is part of the HMD and operations of executing the application, receiving images of the physical environment, analyzing the images and sending data are executed by the processor of the HMD.

17. The method of claim 14, wherein the processor is part of a computing device that is communicatively coupled to the HMD and operations of executing the application, receiving images of the physical environment, analyzing the images and sending data are executed by the processor of the computing device.

18. The method of claim 14, wherein the processor is part of a server that is connected to the HMD via a network and wherein operations of executing the application, receiving images of the physical environment, analyzing the images and sending data are executed by the processor of the server.

19. The method of claim 15, wherein the OS of the HMD is further configured to enable processing interaction with the application.

20. A method for executing content to be presented on a head mounted display (HMD) by a computing device, the HMD being associated with an operating system (OS), the OS enables processing of an application for rendering the content, comprising:

during execution of the application by the computing device, receiving images of a physical environment in a vicinity of the user from a forward facing camera of the HMD;

analyzing, by code of the OS, the images of the physical environment received from the forward facing camera of the HMD to determine a position of the user wearing the HMD in relation to one or more objects in the physical environment; and

sending data, by the OS, for rendering on the display portion of the HMD while rendering the content of the application, the data provides information regarding proximity of the user to the one or more objects in the physical environment and a signal for interrupting the rendering of the content by the HMD.

21. The method of claim 20, wherein the computing device is part of the HMD, or is a game console that is connected to the HMD, or is a personal computer that is connected to the HMD, or is a server that is connected to the HMD over a network.

22. The method of claim 20, wherein operations of receiving of images of the physical environment, analyzing the images and sending data are handled by the OS that is associated with the HMD.