SYSTEMS AND METHODS FOR PROVIDING FEEDBACK TO A USER WHILE INTERACTING WITH CONTENT

• US 20170322655A1
• Filed: 07/25/2017
• Published: 11/09/2017
• Est. Priority Date: 08/19/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger
• Granted Patent: US 10,228,811 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

Systems and methods for providing feedback to a user are described. One of the methods includes sending interactive data for rendering on a head mounted display (HMD). The HMD is configured for attachment to a head of a user for playing a game. The method further includes receiving image data associated with a body part of the user while the HMD is worn by the user and the HMD renders interactive content. The method includes tracking a spatial position associated with the body part of the user from the image data of the user. In the method, physical placement of a remote-controlled device at the spatial position associated with the body part of the user is rendered in the HMD as virtual contact with an object in the interactive content.

Claims

1. A system comprising:

a head mounted display (HMD) for displaying a game, wherein the HMD is configured to be worn by a user; and

a game console coupled to the head mounted display, the game console including a gaming processor configured to;

send interactive content for rendering in the HMD;

receive image data associated with a body part of the user while the HMD is worn by the user;

determine a position associated with the body part of the user in a real-world environment from the image data;

generate an instruction based on the interactive content rendered in the HMD and the position associated with the body part; and

send the instruction to a remotely-controlled device (RCD),

wherein the RCD includes a communication circuit configured to receive the instruction from the game console,

wherein the RCD further includes a contact surface,

wherein upon the reception of the instruction, the RCD is configured to fly to place the contact surface to be proximate to the position associated with the body part, wherein the placement of the contact surface proximate to the position associated with the body part of the user is rendered in the HMD as virtual contact with a virtual object in the interactive content.

2. The system of claim 1, wherein the RCD includes:

a mechanical component; and

a position controller coupled to the mechanical component and to the communication circuit,

wherein the position controller is configured to receive the instruction from the gaming processor,

wherein the position controller is configured to control, based on the instruction, the mechanical component to further control a position of the contact surface, wherein the position controller is configured to control the position of the contact surface to place the contact surface proximate to the position associated with the body part of the user.

4. The system of claim 1, wherein the RCD is a drone.

5. The system of claim 1, wherein the gaming processor is configured to:

determine a spatial position of the virtual object displayed within the interactive content;

determine a spatial position of a representation of the body part, the representation displayed within the interactive content; and

determine whether the spatial position of the representation of the body part is within a pre-determined range from the spatial position of the virtual object,

wherein the instruction is sent to control the position of the contact surface when the spatial position of the representation of the body part is within the pre-determined range from the spatial position of the virtual object.

6. The system of claim 1, wherein the gaming processor is configured to:

determine a spatial position of a representation of the body part from the image data;

associate the spatial position of the representation of the body part in the image data with a spatial position of a representation of the body part in the interactive content;

determine a spatial position of the virtual object in the interactive content;

determine whether the spatial position of the representation of the body part in the interactive content is within a pre-determined distance from the spatial position of the virtual object in the interactive content,

wherein the instruction is sent to control the position of the contact surface when the spatial position of the representation of the body part in the interactive content is within the pre-determined distance from the spatial position of the virtual object in the interactive content.

7. The system of claim 1, wherein the gaming processor is configured to determine x, y, and z co-ordinates of the body part or of a position device attached to the body part from a reference point in the real-world environment to determine the position associated with the body part from the image data.

8. The system of claim 1, wherein the interactive data includes image data used to render a virtual display of the game.

9. The system of claim 1, further comprising an image capturing device configured to capture the image data and send the image data to the gaming processor.

3. The system of claim 3, wherein the mechanical component is a propeller of a drone.

10. A method comprising:

sending interactive content from a game console to for rendering in a head-mounted display (HMD);

receiving image data associated with a body part of a user while the HMD is worn by a user;

determining a position associated with the body part of the user in a real-world environment from the image data;

generating an instruction based on the interactive content rendered in the HMD and the position associated with the body part; and

sending the instruction to a remotely-controlled device (RCD),

receiving the instruction from the game console;

upon said receiving of the instruction, flying to place a contact surface of the RCD to be proximate to the position associated with the body part, wherein the placement of the contact surface proximate to the position associated with the body part of the user is rendered in the HMD as virtual contact with a virtual object in the interactive content.

11. The method of claim 10, further comprising:

controlling a mechanical component of the RCD to control the position of the contact surface to place the contact surface proximate to the position associated with the body part of the user during a play of the game.

12. The method of claim 11, wherein the mechanical component is a propeller of a drone.

13. The method of claim 10, wherein the RCD is a drone.

14. The method of claim 10, further comprising:

determining a spatial position of the virtual object displayed within the interactive content;

determining a spatial position of a representation of the body part, the representation displayed within the interactive content; and

determining whether the spatial position of the representation of the body part is within a pre-determined range from the spatial position of the virtual object,

wherein sending the instruction to the RCD is performed when the spatial position of the representation of the body part is within the pre-determined range from the spatial position of the virtual object.

15. The method of claim 10, further comprising:

determining a spatial position of a representation of the body part from the image data;

associating the spatial position of the representation of the body part in the image data with a spatial position of a representation of the body part in the interactive content;

determining a spatial position of the virtual object in the interactive content;

determining whether the spatial position of the representation of the body part in the interactive content is within a pre-determined distance from the spatial position of the virtual object in the interactive content,

wherein sending the instruction to the RCD is performed when the spatial position of the representation of the body part in the interactive content is within the pre-determined distance from the spatial position of the virtual object in the interactive content.

16. The method of claim 10, further comprising determining x, y, and z co-ordinates of the body part or of a position device attached to the body part from a reference point in the real-world environment to determine the position associated with the body part from the image data.

17. The method of claim 10, wherein the interactive data includes image data used to render a virtual display of the game.

18. The method of claim 10, further comprising:

capturing the image data; and

sending the image data to the gaming processor.

19. A method comprising:

sending interactive content for rendering on a head mounted display (HMD), the HMD configured to be worn by a user for playing a game;

receiving image data associated with a body part of the user while the HMD is worn by the user;

tracking a plurality of positions associated with the body part of the user in a real-world environment from the image data associated with the body part of the user; and

for a transition from a first state of rendering by the HMD to a second state of rendering by the HMD and for a transition from one of the positions associated with the body part to another one of the positions associated with the body part, controlling a position of a remotely-controlled device (RCD) to fly from one position to another position to be placed proximate to the other position associated with the body part, wherein the placement of the RCD proximate to the other position associated with body part of the user is rendered in the HMD as virtual contact with a virtual object in the interactive content.

20. The method of claim 19, further comprising:

determining a spatial position of the virtual object displayed within the interactive content;

determining a spatial position of a representation of the body part, the representation of the body part displayed within the interactive content;

determining whether the spatial position of the representation of the body part is within a pre-determined range from the spatial position of the virtual object,

wherein controlling the position of the RCD is performed when the spatial position of the representation of the body part is within the pre-determined range from the spatial position of the virtual object.

21. The method of claim 19, wherein the RCD is a drone.