Control of separate computer game elements

• US 9,259,643 B2
• Filed: 09/20/2011
• Issued: 02/16/2016
• Est. Priority Date: 04/28/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lansdale, Thomas William, Griffiths, Charles Robert
• Primary Examiner(s): Lewis, David L
• Assistant Examiner(s): HALL, SHAUNA-KAY N
• Application Number:
  US13/237,584
• Publication Number:
  US 20120276994A1
• Time in Patent Office: 1,610 Days
• Field of Search: 463/36, 463/31
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics

Abstract

Control of separate computer game elements is described. In one example, a computer game system displays a computer game to a user, the computer game comprising at least one avatar and an environment. Manual input commands are received from a hand-operated controller operated by the user, and are used to control the display of the avatar in the computer game. Substantially concurrently with the manual input, a stream of images from an image capture device depicting the user is also received. A portal, separate to the avatar, is displayed in a region of the computer game display, which shows the received stream of images to the user. The stream of images of the user is analyzed and a gesture performed by the user is recognized. The recognized gesture is used to control displayed interaction between the portal and the environment.

Claims

1. A method of controlling a computer game system comprising:

displaying a computer game at a display viewable by a user of the game, the display comprising at least one avatar and an environment;

receiving manual input from a hand-operated controller operated by the user;

receiving a stream of images from an image capture device depicting the user;

displaying a portal as part of the displayed computer game, the portal being a region of the display representing the stream of images depicting the user;

recognizing a gesture of the user by analyzing the stream of images;

controlling the display of the avatar on the basis of the manual input and controlling displayed interaction between the portal and the environment on the basis of the recognized gesture of the user; and

one or more of;

recognizing a gesture of the by analyzing the stream, of images further comprising the recognition being based at least in part on tracking the location of at least one body part of the user in the stream of imaged, determining whether the at least one body part being tracked is located within a predefined portion of the stream of images wherein the predefined portion is a predefined distance from an edge of the stream of images;

recognizing a gesture of the user further comprising determining whether the user is holding the controller with both hands by comparing the stream of images depicting the user with at least on pre-recorder stance example;

recognizing a gesture of the user further comprising determining whether the user id holding the controller with both hands by tracking a location for the hands and determining whether they are within a predetermined distance of each other;

or

recognizing a gesture of the user by analyzing the stream of images, determining that the user is reaching out to an edge of the portal, and rendering a representation of a body part of the user outside the edge of the portal.

2. A method as claimed in claim 1, wherein the steps of recognizing the gesture and receiving manual input occur substantially concurrently.

3. A method as claimed in claim 1, wherein the step of recognizing a gesture of the user by analyzing the stream of images comprises tracking at least one body part of the user in the stream of images.

4. A method as claimed in claim 3, wherein the tracking comprises tracking a location for the at least one body part of the user in the stream of images.

5. A method as claimed in claim 4, wherein the step of recognizing a gesture further comprises determining whether the at least one body part being tracked is located within a predefined portion of the stream of images.

6. A method as claimed in claim 5, wherein the predefined portion is a predefined distance from an edge of the stream of images.

7. A method as claimed in claim 6, wherein responsive to determining that the at least one body part being tracked is located within a predefined portion of the stream of images, rendering a computer-generated representation of the at least one body part being tracked outside the portal at a position corresponding to the edge of the stream of images.

8. A method as claimed in claim 7, wherein the step of controlling displayed interaction between the portal and the environment comprises controlling displayed interaction between the computer-generated representation of the at least one body part and the environment.

9. A method as claimed in claim 4, wherein the location tracked for the at least one body part is a distance between the at least one body part and the image capture device.

10. A method as claimed in claim 9, wherein the step of recognizing a gesture further comprises determining whether the distance between the at least one body part and the image capture device is less than a predetermined value.

11. A method as claimed in claim 3, wherein the tracking comprises tracking a motion velocity for the at least one body part of the user in the stream of images.

12. A method as claimed in claim 11, wherein the step of recognizing a gesture further comprises determining whether the at least one body part being tracked has a rate of change of velocity greater than a predetermined value.

13. A method as claimed in claim 3, wherein the at least one body part of the user is the hands of the user.

14. A method as claimed in claim 3, wherein the capture device comprises a depth camera arranged to capture a sequence of depth images of the user, and wherein the step of tracking is performed using a body-pose estimator taking an input from the depth camera.

15. A method as claimed in claim 1, wherein the step of recognizing a gesture comprises determining whether the user is holding the controller with both hands.

16. A method as claimed in claim 15, wherein the step of determining whether the user is holding the controller with both hands comprises comparing the stream of images depicting the user with at least one pre-recorded stance example.

17. A method as claimed in claim 15, wherein the step of determining whether the user is holding the controller with both hands comprises tracking a location for the hands and determining whether they are within a predetermined distance of each other.

18. A computer game system, comprising:

an input interface arranged to receive a stream of images depicting a user from a capture device, and receive manual input commands from a hand-operated controller operable by the user;

an output interface arranged to provide a computer game display to a display device viewable by a user of the computer game system, the computer game display comprising at least one avatar and an environment;

a processor arranged to;

control the output interface to display a plurality of portal as including one active portal and at least one inactive portal as part of the computer game display, the active portal being a region of the display representing the stream of images depicting the user, recognizing a gesture of the user by analyzing the stream of images, control the display of the avatar on the basis of the manual input, control displayed interaction between the portal and the environment on the basis of the recognized gesture of the user, and change an inactive portal to an active portal based on a recognized gesture of the user.

19. A system according to claim 18, wherein the capture device comprises a depth camera.

20. A method of controlling a computer game system comprising:

displaying a computer game at a display viewable by a user of the game, the display comprising at least one avatar and an environment;

receiving manual input from a hand-operated controller operated by the user;

at the same time as receiving manual input, receiving a stream of images from an image capture device depicting the user;

displaying a portal as part of the displayed computer game, the portal being a region of the display representing the stream of images depicting the user;

recognizing a gesture of the user by analyzing the stream of images and determining that the user is reaching out to an edge of the portal;

rendering a representation of a body part of the user outside the edge of the portal; and

controlling the display of the avatar on the basis of the manual input and controlling displayed interaction between the representation of the body part and the environment on the basis of the recognized gesture of the user.