Camera based safety mechanisms for users of head mounted displays

• US 9,729,864 B2
• Filed: 01/31/2014
• Issued: 08/08/2017
• Est. Priority Date: 09/30/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger
• Primary Examiner(s): Bailey, Frederick
• Application Number:
  US14/170,470
• Publication Number:
  US 20150092015A1
• Time in Patent Office: 1,285 Days
• Field of Search: None
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]

Abstract

The disclosure provides methods and systems for warning a user of a head mounted display that the user approaches an edge of field of view of a camera or one or more tangible obstacles. The warning includes presenting audio and/or displayable messages to the user, or moving the display(s) of the head mounted displays away of the user'"'"'s eyes. The determination that the user approaches the edge of scene or a tangible obstacle is made by dynamically tracking motions of the users through analysis of images and/or depth data obtained from image sensor(s) and/or depth sensor(s) secured to either the head mounted display, arranged outside of the scene and not secured to the head mounted display, or a combination of both.

Claims

1. A method for cautioning a user of a head mounted display, the method comprising:

receiving, by a processor, images of a scene of a physical environment within which the user wearing the head mounted display is present, wherein the scene includes a predetermined three-dimensional area in which the user is allowed to move around during use of the head mounted display;

tracking, by the processor, location information of the head mounted display based at least in part on the images as the user moves around the physical environment;

dynamically determining, by the processor, while content is presented on a screen of the head mounted display and the user is wearing the head mounted display, that the user is approaching an edge of the scene, the edge of the scene is defined by a boundary of the predetermined three-dimensional area; and

based on the determination, warning, by the processor, the user via the head mounted display that the user wearing the head mounted display is approaching the edge of the scene.

2. The method of claim 1, wherein the warning includes a generated audio message or instructions to avoid moving toward the edge of the scene.

3. The method of claim 1, wherein the warning includes sending, by the processor, a displayable message to the screen of the head mounted display.

4. The method of claim 1, wherein the warning includes gradually fading-in an image of a tangible object present in the physical environment as the user wearing the head mounted display approaches closer to the tangible object, the fading-in is further configured to increase a brightness of the image of the tangible object as presented on the screen of the head mounted display while presenting the content on the head mounted display.

5. The method of claim 4, wherein the fading-in of the image of the tangible object is reversed to a fading-out of the image of the tangible object when the user wearing the head mounted display moves away from the tangible object present in the physical environment as the user approaches the edge of the scene.

6. The method of claim 1, wherein the warning includes stopping, by the processor, displaying of the content on the screen of the head mounted display when it is determined that the user wearing the head mounted display is approaching the edge of the scene.

7. The method of claim 1, wherein the warning includes gradually blending the content rendered on the screen of the head mounted display with an image of a tangible object from the physical environment, the blending assists in preventing the user from contacting the tangible object.

8. The method of claim 7, wherein the images of the scene include images in which the user is present, and the content rendered on the screen of the head mounted display include virtual reality (VR) media.

9. The method of claim 7, wherein images of the scene include images from a point of view of the user wearing the head mounted display.

10. The method of claim 1, wherein the scene is defined by a field of view of at least one image sensor or at least one depth sensor usable to determine when the user wearing the head mounted display is approaching the edge of the scene.

11. The method of claim 1, wherein the warning includes providing guiding instructions to direct the user back towards a center of the scene or a safe location within the predetermined three-dimensional area where the user can continue to interact with the content presented on the screen of the head mounted display, the content being virtual reality (VR) media.

12. A method for cautioning a user of a head mounted display, the method comprising:

receiving, by a processor, images of a scene of a physical environment within which the user using the head mounted display is present, wherein the scene includes a predetermined three-dimensional area in which the user moves around during use of the head mounted display and wherein the images are captured by one or more sensors secured to the head mounted display;

dynamically determining, by the processor, while content is presented on a screen of the head mounted display and the user is wearing the head mounted display, that the user is approaching one or more tangible obstacles in the physical environment based at least on the images, the determining includes analyzing the images to identify a boundary of the predetermined three-dimensional area and determining that the user is approaching the one or more tangible objects within the boundary; and

based on the determination, warning, by the processor, the user via the head mounted display that the user wearing the head mounted display is approaching the one or more tangible obstacles.

13. The method of claim 12, wherein the one or more sensors include an image sensor.

14. The method of claim 12, wherein the one or more sensors include a depth sensor.

15. The method of claim 12, wherein the images include depth information of the scene.

16. The method of claim 12, further comprising receiving, by the processor, secondary images of the scene from one or more secondary sensors arranged external to the head mounted display, the secondary images capturing the user within the scene of the physical environment.

17. The method of claim 16, wherein the determining that the user is approaching one or more tangible obstacles is further based at least on evaluating the secondary images.

18. The method of claim 12, wherein the one or more tangible obstacles are present within the physical environment at an edge of the scene.

19. The method of claim 12, wherein the one or more tangible obstacles at least partially define an edge of the scene where the user is considered safe during interaction with content provided to the screen of the head mounted display, the content being virtual reality (VR) media.

20. A method for cautioning a user of a head mounted display, the method comprising:

receiving, by a processor, depth data of a scene of a physical environment within which the user using the head mounted display is present, wherein the scene includes a predetermined three-dimensional area in which the user moves around during use of the head mounted display and wherein the depth data is captured by one or more depth sensors secured to the head mounted display;

dynamically determining, by the processor, while content is presented on a screen of the head mounted display and the user is wearing the head mounted display that the user is approaching one or more tangible obstacles present within a boundary of the predetermined three-dimensional area as the user moves around the physical environment, based at least on the depth data; and

based on the determination, warning, by the processor, the user via the head mounted display that the user wearing the head mounted display is approaching the one or more tangible obstacles.