Systems and methods for providing feedback to a user while interacting with content

• US 9,746,984 B2
• Filed: 09/30/2014
• Issued: 08/29/2017
• Est. Priority Date: 08/19/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger
• Primary Examiner(s): LIM, SENG HENG
• Application Number:
  US14/503,275
• Publication Number:
  US 20160054837A1
• Time in Patent Office: 1,064 Days
• Field of Search: None
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: A63F 2300/1037 being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

Systems and methods for providing feedback to a user are described. One of the methods includes sending interactive data for rendering on a head mounted display (HMD). The HMD is configured for attachment to a head of a user for playing a game. The method further includes receiving image data associated with a body part of the user while the HMD is worn by the user and the HMD renders interactive content. The method includes tracking a spatial position associated with the body part of the user from the image data of the user. In the method, physical placement of a remote-controlled device at the spatial position associated with the body part of the user is rendered in the HMD as virtual contact with an object in the interactive content.

Claims

1. A method comprising:

sending interactive data for rendering on a head mounted display (HMD), the HMD configured for attachment to a head of a user for playing a game;

receiving image data associated with a body part of the user while the HMD is worn by the user and the HMD renders interactive content;

tracking a spatial position associated with the body part of the user from the image data associated with the body part of the user; and

for a state of rendering by the HMD and for the spatial position associated with the body part, controlling a position of a remotely-controlled device (RCD) to fly and place the RCD proximate to the spatial position associated with the body part, wherein the placement of the RCD at the spatial position associated with the body part of the user is rendered in the HMD as virtual contact with an object in the interactive content.

2. The method of claim 1, further comprising:

determining a spatial position of the object displayed within the interactive content;

determining a spatial position of a representation of the body part, the representation of the body part displayed within the interactive content;

determining whether the spatial position of the representation of the body part is within a pre-determined range from the spatial position of the object,

wherein controlling the position of the RCD is performed when the spatial position of the representation of the body part is within the pre-determined range from the spatial position of the object.

3. The method of claim 1, further comprising scaling a position of the body part that is represented within the image data to the spatial position associated with the body part within a real-world in which the user is located.

4. The method of claim 1, wherein the image data associated with the body part includes image data received from an image capturing device that captures an image of the body part or an image of a position device that is attached to the body part.

5. The method of claim 1, wherein the body part includes a hand or a leg or the head of the user.

6. The method of claim 1, wherein receiving the image data associated with the body part of the user while the HMD is worn by the user and the HMD renders the interactive content comprises receiving the image data from an image capturing device at a time the HMD is displaying the interactive content and the HMD is worn by the user.

7. The method of claim 1, wherein tracking the spatial position associated with the body part from the image data includes determining x, y, and z co-ordinates of the body part or of a position device attached to the body part from a reference point in a real-world environment.

8. The method of claim 1, wherein the interactive data includes image data used to render a virtual game.

9. The method of claim 1, wherein the state of rendering by the HMD is based on a state of an interactive program that is executed to generate the interactive data that is further rendered to display the interactive content.

10. The method of claim 1, wherein the interactive content includes a virtual reality scene or an augmented reality scene or a video or streaming data.

11. The method of claim 1, wherein the RCD includes a drone or a robot or an exercise machine.

12. The method of claim 1, wherein the RCD is proximate to the spatial position associated with the body part when a contact surface of the RCD touches the body part or a position device that is attached to the body part.

13. The method of claim 1, wherein the RCD is proximate to the spatial position associated with the body part when a contact surface of the RCD has the same position as that of the body part or of a position device that is attached to the body part.

14. The method of claim 1, wherein the RCD is proximate to the spatial position associated with the body part when a contact surface of the RCD is within a pre-determined distance of the body part or of a position device that is attached to the body part.

15. A system comprising:

a head mounted display (HMD) for displaying a game, the head mounted display for mounting on a head of a user for playing the game; and

a game console coupled to the head mounted display, the game console including a gaming processor for;

sending interactive content for rendering in the HMD;

receiving image data associated with a body part of the user while the HMD is worn by the user and the HMD renders the interactive content;

tracking a spatial position associated with the body part of the user from the image data;

generating an instruction based on a current state of rendering by the HMD and the spatial position associated with the body part; and

sending the instruction to a remotely-controlled device (RCD) to control a position of the RCD to fly and place the RCD proximate to the spatial position associated with the body part, wherein the placement of the RCD at the spatial position associated with body part of the user is rendered in the HMD as virtual contact with an object in the interactive content.

16. The system of claim 15, wherein the gaming processor is for:

determining a spatial position of the object displayed within the interactive content;

determining a spatial position of a representation of the body part, the representation displayed within the interactive content; and

determining whether the spatial position of the representation of the body part is within a pre-determined range from the spatial position of the object,

wherein sending the instruction to control the position of the RCD is performed when the spatial position of the representation of the body part is within the pre-determined range from the spatial position of the object.

17. The system of claim 15, wherein the gaming processor is for:

determining a spatial position of a representation of the body part from the image data;

associating the spatial position of the representation of the body part in the image data with a spatial position of a representation of the body part in the interactive content;

determining a spatial position of a virtual object in the interactive content;

determining whether the spatial position of the representation of the body part in the interactive content is within a pre-determined distance from the spatial position of the virtual object in the interactive content,

wherein sending the instruction to control the position of the RCD is performed when the spatial position of the representation of the body part in the interactive content is within the pre-determined distance from the spatial position of the virtual object in the interactive content.

18. The system of claim 15, wherein the RCD includes a drone.

19. A remotely-controlled device (RCD) comprising:

a communication circuit for receiving an instruction from a game console;

a mechanical component;

a contact surface; and

a position controller coupled to the mechanical component and to the communication circuit, the position controller for controlling a position of the contact surface based on the instruction, the position controller for controlling the mechanical component to fly to further control the position of the contact surface, the position controller to control the position of the contact surface to place the contact surface proximate to a spatial position associated with a body part of a user during a play of a game, the game having interactive content, wherein the placement of the contact surface at the spatial position associated with the body part of the user is rendered in a head-mounted display (HMD) as virtual contact with an object in the interactive content.

20. The RCD of claim 19, wherein the mechanical component includes a propeller of a drone.