Methods and Systems for User Interaction within Virtual Reality Scene using Head Mounted Display

• US 20160187969A1
• Filed: 06/19/2015
• Published: 06/30/2016
• Est. Priority Date: 12/29/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Larsen, Eric, Umminger, Frederick, Ye, Xiaoyong, Lou, Xia, Rimon, Noam, Stafford, Jeffrey Roger
• Granted Patent: US 10,073,516 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Inertial sensors within a head mounted display are used to track movement of the head mounted display. The tracked movement of the head mounted display is correlated to an action within a virtual reality scene that is currently displayed to a user wearing the head mounted display. The action within the virtual reality scene is based on a context of the virtual reality scene that is currently displayed. The detected movement of the head mounted display can be combined with other sensor data, such as gaze detection data, to determine the action within the virtual reality scene. In this manner, movements of the user as detected using the inertial sensors within the head mounted display are used as inputs to cause actions within the current context of the virtual reality scene as displayed to the user within the head mounted display.

Claims

1. A method, comprising:

transmitting video data to a head mounted display for display of a virtual reality scene within the head mounted display;

receiving inertial sensor data from one or more inertial sensors within the head mounted display, the inertial sensor data indicating movement of the head mounted display in accordance with movement of a user by whom the head mounted display is worn, the movement of the user based on the virtual reality scene displayed within the head mounted display;

correlating the received inertial sensor data to an action within the virtual reality scene, wherein the correlating includes determining a route of movement of the head mounted display and determining a rate of movement of the head mounted display, wherein the correlating includes determining a set of allowable virtual movements within a current context of the virtual reality scene, and wherein the correlating includes mapping the route of movement of the head mounted display to a corresponding virtual movement within the set of allowable virtual movements; and

generating an update to the virtual reality scene to reflect implementation of the action.

2. The method as recited in claim 1, wherein mapping the route of movement of the head mounted display to the corresponding virtual movement includes querying a calibration table, the calibration table defining associations between a number of routes of movement of the head mounted display and resulting actions within the virtual reality scene.

3. The method as recited in claim 2, wherein the correlating includes mapping the rate of movement of the head mounted display to a rate of the corresponding virtual movement within the virtual reality scene.

4. The method as recited in claim 1, further comprising:

transmitting a feedback signal to the head mounted display, the feedback signal notifying the user that the movement of the head mounted display does not correspond to any virtual movement within the set of allowable virtual movements as determined by the correlating.

5. The method as recited in claim 1, further comprising:

receiving external inertial sensor data from one or more inertial sensors disposed on the user at locations external to the head mounted display;

using the external inertial sensor data to facilitate correlating the inertial sensor data received from the one or more inertial sensors within the head mounted display to the action within the virtual reality scene.

6. The method as recited in claim 1, wherein the action within the virtual reality scene is one or more of a movement of a viewpoint of the user within the virtual reality scene, a zoom-in or a zoom-out of a viewpoint of the user within the virtual reality scene, and a movement of an object displayed within the virtual reality scene.

7. The method as recited in claim 1, further comprising:

receiving gaze direction data from one or more gaze detectors within the head mounted display, the gaze direction data indicating a vision focus direction of the user upon the virtual reality scene displayed within the head mounted display; and

synchronizing the gaze direction data with the inertial sensor data to determine a synchronized vision focus direction.

8. The method as recited in claim 7, wherein the correlating includes determining a route of movement of the head mounted display and determining a rate of movement of the head mounted display, and wherein the correlating includes using both the route of movement of the head mounted display and the synchronized vision focus direction to determine the action within the virtual reality scene.

9. A method, comprising:

receiving video data at a head mounted display, the video data defining a virtual reality scene;

displaying the virtual reality scene within the head mounted display;

operating one or more inertial sensors within the head mounted display to generate inertial sensor data indicating movement of the head mounted display in accordance with movement of a user by whom the head mounted display is worn, the movement of the user corresponding to an action within the virtual reality scene;

transmitting the inertial sensor data to a computing system from which the video data is received, wherein the inertial sensor data provides for determination of a route of movement of the head mounted display by the computing system and determination of a rate of movement of the head mounted display by the computing system, wherein the route of movement of the head mounted display and the rate of movement of the head mounted display are mapped by the computing system to a corresponding virtual movement within a set of allowable virtual movements within a current context of the virtual reality scene, the corresponding virtual movement directing the action within the virtual reality scene;

receiving, from the computing system, video data defining an updated virtual reality scene reflective of the action within the virtual reality scene corresponding to the movement of the user; and

displaying the updated virtual reality scene within the head mounted display.

10. The method as recited in claim 9, further comprising:

receiving, from the computing system, a feedback signal indicating that the route of movement of the head mounted display does not have any corresponding virtual movement within the set of allowable virtual movements within the current context of the virtual reality scene; and

operating the head mounted display to provide one or more of an audible signal, a tactile signal, and a visual signal to notify the user of receipt of the feedback signal.

11. The method as recited in claim 9, further comprising:

transmitting external inertial sensor data to the computing system from one or more inertial sensors disposed on the user at locations external to the head mounted display, the external inertial sensor data used by the computing system to facilitate determination of the route of movement of the head mounted display.

12. The method as recited in claim 9, wherein the action within the virtual reality scene is one or more of a movement of a viewpoint of the user within the virtual reality scene, a zoom-in or a zoom-out of a viewpoint of the user within the virtual reality scene, and a movement of an object displayed within the virtual reality scene.

13. The method as recited in claim 9, further comprising:

operating one or more gaze detectors within the head mounted display to generate gaze direction data indicating a vision focus direction of the user upon the virtual reality scene displayed within the head mounted display; and

transmitting the gaze direction data to the computing system from which the video data is received, the gaze direction data synchronized with the inertial sensor data to determine a synchronized vision focus direction.

14. The method as recited in claim 13, wherein both the route of movement of the head mounted display and the synchronized vision focus direction are used to determine the action within the virtual reality scene corresponding to the movement of the user.

15. A system, comprising:

an inertial data processing module configured to receive inertial sensor data from a head mounted display, the inertial sensor data indicating movement of the head mounted display in accordance with movement of a user by whom the head mounted display is worn, the movement of the user based on a virtual reality scene displayed within the head mounted display, the inertial data processing module configured to determine from the inertial sensor data a route of movement of the head mounted display and a rate of movement of the head mounted display;

an inertial data correlation module configured to correlate the route of movement of the head mounted display and the rate of movement of the head mounted display to an action within the virtual reality scene, wherein the inertial data correlation module is configured to determine a set of allowable virtual movements within a current context of the virtual reality scene, wherein the inertial data correlation module is configured to map the route of movement of the head mounted display to a corresponding virtual movement within the set of allowable virtual movements; and

an inertial data command output module configured to provide a command signal to a rendering engine, the command signal directing the rendering engine to update the virtual reality scene to reflect implementation of the action.

16. The system as recited in claim 15, wherein the inertial data correlation module is configured to map the rate of movement of the head mounted display to a rate of the corresponding virtual movement within the virtual reality scene.

17. The system as recited in claim 16, wherein the inertial data correlation module is configured to query a calibration table in order to map the route of movement and the rate of movement of the head mounted display to the corresponding virtual movement within the virtual reality scene, the calibration table defining associations between a number of combinations of routes and rates of movement of the head mounted display and resulting actions within the virtual reality scene.

18. The system as recited in claim 15, further comprising:

an inertial data feedback module configured to generate and transmit a feedback signal to the head mounted display, the feedback signal notifying the user that the movement of the head mounted display does not correspond to any virtual movement within the set of allowable virtual movements.

19. The system as recited in claim 15, further comprising:

a gaze direction processing module configured to receive gaze direction data from the head mounted display, the gaze direction data indicating a vision focus direction of the user upon the virtual reality scene displayed within the head mounted display; and

a synchronization module configured to synchronize the gaze direction data with the inertial sensor data to determine a synchronized vision focus direction, wherein the inertial data correlation module is configured to use both the route of movement of the head mounted display and the synchronized vision focus direction to determine the action within the virtual reality scene.

20. The system as recited in claim 19, wherein the inertial data correlation module is configured to use the synchronized vision focus direction as a directional weighting parameter in determining the action within the virtual reality scene.