Switching mode of operation in a head mounted display

• US 10,019,057 B2
• Filed: 03/12/2014
• Issued: 07/10/2018
• Est. Priority Date: 06/07/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Osman, Steven, Mao, Xiadong, Stafford, Jeffrey Roger
• Primary Examiner(s): Boddie, William
• Assistant Examiner(s): Parker, Jeffrey A
• Application Number:
  US14/206,219
• Publication Number:
  US 20140361976A1
• Time in Patent Office: 1,581 Days
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 2203/04804 Transparency, e.g. transparent or translucent windows
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Methods, systems, and computer programs are presented for managing the display of images on a head mounted device (HMD). One method includes an operation for tracking the gaze of a user wearing the HMD, where the HMD is displaying a scene of a virtual world. In addition, the method includes an operation for detecting that the gaze of the user is fixed on a predetermined area for a predetermined amount of time. In response to the detecting, the method fades out a region of the display in the HMD, while maintaining the scene of the virtual world in an area of the display outside the region. Additionally, the method includes an operation for fading in a view of the real world in the region as if the HMD were transparent to the user while the user is looking through the region. The fading in of the view of the real world includes maintaining the scene of the virtual world outside the region.

Claims

1. A method for managing display of images on a head mounted device (HMD), the method comprising:

presenting video data on a display of the HMD, the video data providing a view of a virtual world;

tracking a gaze of a user wearing the HMD that is displaying the video data;

detecting that the gaze of the user is fixed on a predetermined area of the display for a predetermined amount of time, which triggers modification to the video data presented at the display, without requiring user action, wherein the modification of the video data includes,

fading out a portion of the video data that is being rendered in the predetermined area of the display of the HMD while maintaining the view of the virtual world in an area of the display outside the predetermined area; and

fading in a portion of an image of a real world in the predetermined area, wherein the image of the real world is captured by a forward facing camera of the HMD and the portion of the image of the real world corresponds to a direction of the gaze of the user, the fading in simulating a see-through mode of the display in the predetermined area by extending a view of the user through the HMD into the real world in the direction of the gaze, while the area outside of the predetermined area continues to present the view of the virtual world,

wherein the fading out of the video data and the fading in of the image of the real world is performed for a period of time.

2. The method as recited in claim 1, wherein the predetermined area, where the gaze of the user has been detected for a predetermined amount of time, is one of an area defined below the display, an area defined to a side of the display, an area defined above the display, or an area defined on a corner of the display.

3. The method as recited in claim 1, wherein the predetermined area, where the gaze of the user has been detected for a predetermined amount of time, is defined below the display, and wherein the region is a bottom half of the display.

4. The method as recited in claim 1, wherein the predetermined area is a rectangle centered on the display.

5. The method as recited in claim 1, further including:

providing an option for user confirmation before fading in the portion of the image of the real world in the predetermined area.

6. The method as recited in claim 1, further including:

receiving a command from the user to terminate rendering of the image of the real world;

in response to the command, fading out the portion of the image of the real world in the predetermined area of the display; and

fading in the portion of the virtual world that corresponds with the predetermined area while keeping the view of the virtual world in the area outside the predetermined area.

7. The method as recited in claim 1, wherein tracking the gaze further includes:

capturing images of eyes of the user with a camera inside the HMD; and

analyzing the captured images to determine the gaze of the user.

8. The method as recited in claim 1, wherein the predetermined amount of time is two seconds.

9. The method of claim 1, further comprising,

detecting a notification from a device of the user, wherein the view in the direction of the gaze is toward the device.

10. The method of claim 9, wherein the notification is a phone call or a message being received via the device.

11. A method for managing images rendered on a display of a head mounted device (HMD), the method comprising:

detecting an operation to acclimate a user to the HMD;

presenting a view of a real world on the display to simulate a see-through mode for the display of the HMD, the view of the real world captured by a camera disposed on a front side of the HMD;

adding a virtual element from a virtual world to the view of the real world to simulate an augmented reality mode for the display, wherein the virtual element is added in an area on the display by fading out a portion of the view of the real world rendered in the area and fading in the virtual element, such that the view of the real world is not visible in the area occupied by the virtual element;

after adding the virtual element in the area, fading out a portion of the view of the real world that corresponds with a second area of the display that is outside the area occupied by the virtual element while maintaining the virtual element on the display in the area without fading the virtual element; and

fading in a portion of a view of the virtual world that corresponds with the second area of the display while keeping the virtual element, wherein fading in the portion of the view of the virtual world includes adding additional virtual elements that form the virtual world, wherein the virtual element is present during the fading out the portion of the view of the real world and the fading in of the virtual world to assist the user to acclimate into the view of the virtual world.

12. The method as recited in claim 11, wherein the augmented reality mode includes presenting the virtual element as if the virtual element was located in the real world, wherein a motion of the HMD with respect to the real world causes a change in the view of the real world while maintaining the virtual element in a same location in the real world.

13. The method as recited in claim 11, wherein the real world and the virtual world are not blended on the display except for the virtual element.

14. The method as recited in claim 11, wherein the real world and the virtual world are blended during the fade out of the real world.

15. The method as recited in claim 11, wherein operations of the method are performed by a computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.

16. A non-transitory computer-readable storage medium storing a computer program, the computer-readable storage medium comprising:

program instructions for presenting video data on a display of the HMD, the video data providing a view of a virtual world;

program instructions for tracking a gaze of a user wearing the HMD that is displaying the video data;

program instructions for detecting that the gaze of the user is fixed on a predetermined area of the display for a predetermined amount of time, which triggers modification to the video data presented at the display without requiring user action, wherein the modification of the video data includes,

program instructions for fading out a portion of the video data that is being rendered in the predetermined area of the display of the HMD while maintaining the view of the virtual world in an area of the display outside the predetermined area; and

program instructions for fading in a portion of an image of a real world in the predetermined area, wherein the image of the real world is captured by a forward facing camera of the HMD and the portion of the image of the real world corresponds to a direction of the gaze of the user, the fading in simulating a see-through mode of the display in the predetermined area by extending a view of the user through the HMD into the real world in the direction of the gaze, while the area outside of the predetermined area continues to present the view of the virtual world,

wherein the fading out of the video data and the fading in of the image of the real world is performed for a period of time.

17. The computer program as recited in claim 16, wherein the predetermined area is one of an area defined below the display, an area defined to a side of the display, an area defined above the display, or an area defined on a corner of the display.

18. The computer program as recited in claim 16, wherein the predetermined area is defined below the display, and wherein the region is a bottom half of the display.

19. The computer program as recited in claim 16, further including:

program instructions for detecting a command from the user to terminate the view of the real world;

program instructions for fading out the view of the real world in the region; and

program instructions for fading in a view of the virtual world in the region.

20. The computer program as recited in claim 16, wherein program instructions for tracking the gaze further includes:

program instructions for capturing images of eyes of the user with a camera inside the HMD; and

program instructions for analyzing the captured images to determine the gaze of the user.

21. A method for managing display of images on a head mounted device (HMD), the method comprising:

notifying a user wearing the HMD to enter a signature by moving a gaze of the user in an empty area of the display;

tracking the gaze of the user as the gaze of the user moves across the empty area of the display of the HMD;

identifying a trajectory of the gaze as the gaze is tracked;

comparing the identified trajectory to a signature for the user, the signature including a signature trajectory for gaze; and

enabling access to a resource on the HMD when the identified trajectory corresponds to the signature trajectory, wherein accessing the resource causes images provided by the resource to be forwarded to the HMD for rendering on the display.