Interacting with a computer based application

• US 9,069,381 B2
• Filed: 03/02/2012
• Issued: 06/30/2015
• Est. Priority Date: 03/12/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin, Markovic, Relja, Latta, Stephen G., Mihelich, Mark T., Willoughby, Christopher, Steed, Jonathan T., Bennett, Darren, Wright, Shawn C., Coohill, Matt
• Primary Examiner(s): GALKA, LAWRENCE STEFAN
• Application Number:
  US13/410,561
• Publication Number:
  US 20120165096A1
• Time in Patent Office: 1,215 Days
• Field of Search: 463/1, 463/30
• US Class Current: 1/1
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/47 involving branching, e.g. choosing one of several possible scenarios at a given point in time
• CPC Class Codes: A63F 2300/10 characterized by input arrangements for converting player-generated signals into game device control signals
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/8023 the game being played by multiple players at a common site, e.g. in an arena, theatre, shopping mall using a large public display
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A computing system runs an application (e.g., video game) that interacts with one or more actively engaged users. One or more physical properties of a group are sensed. The group may include the one or more actively engaged users and/or one or more entities not actively engaged with the application. The computing system will determine that the group (or the one or more entities not actively engaged with the application) have performed a predetermined action. A runtime condition of the application is changed in response to determining that the group (or the one or more entities not actively engaged with the computer based application) have performed the predetermined action. Examples of changing a runtime condition include moving an object, changing a score or changing an environmental condition of a video game.

Claims

1. A method for interacting with a computer based application, comprising:

using a sensor system to automatically track motion of limbs of one or more users actively engaged with the computer based application;

based at least in part on the automatically tracking motion of limbs of one or more users actively engaged with the computer based application, automatically determining that the one or more users actively engaged with the computer based application have performed a first predetermined motion;

performing the computer based application including performing a first action in the computer based application based at least in part on the determining that the one or more users actively engaged with the computer based application have performed the first predetermined motion;

using the sensor system to automatically track motion of limbs of one or more entities not actively engaged with the computer based application;

based at least in part on the automatically tracking motion of limbs of one or more entities not actively engaged with the computer based application, automatically determining that the one or more entities not actively engaged with the computer based application have performed a second predetermined motion, the first predetermined motion is a different type of gesture than the second predetermined motion;

performing a second action in the computer based application based at least in part on the determining that the one or more entities not actively engaged with the computer based application have performed the second predetermined motion, the second action is a different type of action than the first action; and

automatically reporting the-first action and the second action in a user interface of the computer based application.

2. The method of claim 1, wherein:

the using the sensor system to automatically track motion of limbs of one or more entities not actively engaged with the computer based application includes sensing a two dimensional depth image with pixels having magnitudes that represent distance;

the second predetermined motion is a gesture that involves motion; and

the determining that the one or more entities not actively engaged with the computer based application have performed the second predetermined motion includes using the depth image to identify the gesture.

3. The method of claim 1, wherein:

using the sensor system to automatically track motion of limbs of one or more entities not actively engaged with the computer based application includes sensing a two dimensional depth image with pixels having magnitudes that represent distance and sensing a visual image;

the second predetermined motion is a gesture that involves motion; and

the determining that the one or more entities not actively engaged with the computer based application have performed the second predetermined motion includes using the depth image and the visual image to identify the gesture.

4. The method of claim 1, wherein:

the computer based application is a video game; and

the performing the second action includes changing an appearance of an item in the video game.

5. The method of claim 1, wherein:

the computer based application is a video game; and

the performing the second action includes changing the score of the video game.

6. The method of claim 1, further comprising:

detecting an emotion of the one or more entities not actively engaged with the computer based application based at least in part on facial expression using the sensor system, the performing the second action is performed in response to detecting the emotion.

7. The method of claim 1, wherein:

the determining that the one or more entities not actively engaged with the computer based application have performed a second predetermined motion comprises identifying an action performed as a group.

8. One or more processor readable hardware storage devices having processor readable code embodied on the one or more processor readable hardware storage devices, the processor readable code for programming one or more processors to perform a method comprising:

performing a video game including interacting with one or more users who are bound to the video game;

receiving information from a first sensor about moving objects, the moving objects include the one or more bound users and one or more persons who are not bound to the video game;

during the video game, automatically tracking and characterizing motion of the one or more bound users and the one or more persons who are not bound to the video game based at least in part on the information from the first sensor including recognizing different types of motion being performed by the one or more bound users as compared to the one or more persons who are not bound to the video game, the tracking and characterizing motion of the one or more bound users includes skeletal tracking using one or more models of the bodies of the one or more bound users, the tracking and characterizing motion of the one or more persons who are not bound to the video game includes skeletal tracking using one or more models of the bodies of the one or more persons who are not bound to the video game; and

automatically changing the computer based video game in response to the tracked and characterized motion of the one or more bound users and one or more persons who are not bound to the video game.

9. The one or more processor readable hardware storage devices of claim 8, wherein:

the receiving information from a first sensor includes receiving a two dimensional depth image with pixels having magnitudes that represent distance; and

automatically tracking and characterizing motion includes recognizing a gesture using the depth image.

10. The one or more processor readable hardware storage devices of claim 8, wherein the method further comprises:

receiving sound information from a second sensor, the sound information is from the one or more persons who are not bound to the computer based video game, the changing the video game in performed partially in response to the sound information.

11. The one or more processor readable hardware storage devices of claim 8, wherein:

the changing the video game includes moving an item in the video game in response to motion of the one or more persons who are not bound to the video game.

12. The one or more processor readable hardware storage devices of claim 8, wherein:

the automatically tracking and characterizing motion comprises identifying an action performed as a group.

13. The one or more processor readable hardware storage devices of claim 8, wherein:

the receiving information from a first sensor about moving objects includes sensing motion of a group;

the automatically tracking and characterizing motion includes identifying an action of an individual of the group;

the changing the video game is performed based on the action of the individual.

14. The one or more processor readable hardware storage devices of claim 8, wherein:

the tracking and characterizing motion of the one or more persons who are not bound to the video game includes skeletal tracking to track motion of limbs of the one or more persons who are not bound to the video game.

15. The one or more processor readable hardware storage devices of claim 8, wherein the method further comprises:

using the information from the first sensor to detect an emotion of the one or more persons who are not bound to the video game based on facial expression, the automatically changing the computer based video game is performed in response to detecting the emotion.

16. A computing system, comprising:

a camera; and

a computer connected to the camera, the computer includes;

a tracking engine to receive data from the camera, the tracking engine operable to provide output information indicative of tracking,

a software application for communication with the tracking engine, the software application operable to interact with one or more actively engaged users based on the output information from the tracking engine and operable to interact with one or more entities not actively engaged with the software application, the tracking engine operable to track motion of limbs of the one or more entities not actively engaged with the software application, the output information includes information about the tracked motion of limbs of the one or more entities not actively engaged with the software application,

a recognizer engine to receive data from the camera and the output information from the tracking engine, and

a plurality of filters, the recognizer engine operable to selectively provide the data from the camera and output information from the tracking engine to one or more of the filters as input data, each filter of the plurality of filters configured to test for a different predetermined motion and configured to output to the software application whether one or more entities not actively engaged with the software application have performed the respective predetermined motion based on the tracking of motion of limbs of the one or more entities not actively engaged with the software application, the software application operable to make a change to a runtime condition reported in a user interface of the software application in response to one or more of the filters indicating that one or more entities not actively engaged with the software application have performed a predetermined action.

17. The computing system of claim 16, wherein:

the camera includes a depth sensor; and

the data from the camera includes a depth image.

18. The computing system of claim 17, wherein:

the software application is a video game; and

the change to the runtime condition is a movement of an object in response to movement of one or more entities not actively engaged with the video game.

19. The computing system of claim 16, wherein:

the software application is a video game; and

each filter of the plurality of the plurality of filters is configured to determine and output to the video game whether one or more entities not actively engaged with the software application have performed a different gesture that involves motion.

20. The computing system of claim 16, wherein:

the software application is a video game; and

one particular filter of the plurality of the plurality of filters is configured to determine and output to the video game whether a group has performed a predetermined motion as an aggregate, the group including the entities not actively engaged with the software application and the one or more actively engaged users.