Visual based identity tracking

• US 8,926,431 B2
• Filed: 03/02/2012
• Issued: 01/06/2015
• Est. Priority Date: 01/29/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Leyvand, Tommer, Dernis, Mitchell Stephen, Li, Jinyu, Wei, Yichen, Sun, Jian, Meekhof, Casey Leon, Keosababian, Timothy Milton
• Primary Examiner(s): Shah, Milap
• Application Number:
  US13/410,626
• Publication Number:
  US 20120165097A1
• Time in Patent Office: 1,040 Days
• Field of Search: 463/9, 463 30- 34, 463 36- 43, 382/103, 382/154, 345/156, 345/157
• US Class Current: 463/36
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

A video game system (or other data processing system) can visually identify a person entering a field of view of the system and determine whether the person has been previously interacting with the system. In one embodiment, the system establishes thresholds, enrolls players, performs the video game (or other application) including interacting with a subset of the players based on the enrolling, determines that a person has become detectable in the field of view of the system, automatically determines whether the person is one of the enrolled players, maps the person to an enrolled player and interacts with the person based on the mapping if it is determined that the person is one of the enrolled players, and assigns a new identification to the person and interacts with the person based on the new identification if it is determined that the person is not one of the enrolled players.

Claims

1. A method of configuring interaction between a set of already enrolled players and a computer based application, comprising:

executing the computer based application using a computing system, the computer based application interacting with one or more players of the set of the already enrolled players who are physically co-located in a physical real-world play space, the already enrolled players are users of the computing system wherein the computing system previously identified the set of already enrolled players using a sensor system by having detected one or more physical features of each of the set of already enrolled players with respect to one or more tracked locations in the physical real-world play space and having generated and having stored, for each of the set of already enrolled players, one or more stored visual signatures of the detected one or more physical features for each tracked location;

automatically, via the sensor system and the computing system, determining that a person has become detectable in the physical real-world play space, wherein the person that has become detectable in the physical real-world play space is not one of the one or more players of the set of the already enrolled players currently interacting with the computer based application in the physical real-world play space;

automatically via the computing system and the sensor system,

detecting one or more physical features of the person and tracking one or more locations in the physical real-world play space of the detected one or more physical features of the person,

generating one or more respective visual signatures of the detected one or more physical features of the person for each tracked location,

identifying whether the person is a specific already enrolled player of the set of already enrolled players based on comparing the one or more respective visual signatures to the one or more stored visual signatures of the set of already enrolled players, and

responsive to the one or more stored visual signatures of a specific already enrolled player matching the one or more respective visual signatures of the person, concluding that the person is the specific already enrolled player of the set of already enrolled players; and

mapping the person to the specific already enrolled player and interacting with the person using the computer based application running on the computing system based on the mapping.

2. The method of claim 1, wherein:

the automatically identifying the person that has become detectable in the real-world play space as a specific already enrolled player of the set of already enrolled players includes visually identifying the person as the specific already enrolled player.

3. The method of claim 1, further comprising:

sensing visual data of the person using the sensor system; and

sensing depth data of the person using the sensor system, the automatically identifying the person as a specific enrolled player of the set of already enrolled players is based on the visual data and the depth data, the depth data includes a two dimensional depth image with pixels of the depth image each having magnitudes that represent distance from the sensor system.

4. The method of claim 1, further comprising:

establishing thresholds, the thresholds are used when automatically identifying the person that has become detectable in the real-world play space as a specific already enrolled player of the set of already enrolled players; and

dynamically updating the thresholds while and based on performing the computer based application.

5. The method of claim 1, further comprising:

receiving visual data and depth data from the sensor system; and

wherein generating one or more respective visual signatures of the detected one or more physical features of the person for each tracked location further comprises

creating a set of visual signatures of the person for each tracked location indicative of appearance of the person based on the visual data and depth data, the set of visual signatures are created by applying one or more mathematical functions to the visual data to create a mathematical representation of the visual data.

6. The method of claim 5, wherein:

the creating the set of visual signatures of the person for each tracked location includes creating a signature for an appearance of the person'"'"'s face, creating a signature for an appearance of the person'"'"'s clothing and creating a signature for the person'"'"'s height; and

the detected physical features of the person includes arrangement of the person'"'"'s face, color of the person'"'"'s clothing and the person'"'"'s height.

7. The method of claim 6, wherein identifying the person as a specific already enrolled player of the set of already enrolled players based on comparing the one or more respective visual signatures to the one or more stored visual signatures of the set of already enrolled players and concluding that the person is the specific already enrolled player of the set of already enrolled players having the one or more stored visual signatures that closest match the one or more respective visual signatures of the person further comprises:

(a) determining whether the signature for the appearance of the person'"'"'s face corresponds to stored signatures for each tracked location for faces of any of the set of enrolled players;

(b) determining whether the signature for the appearance of the person'"'"'s clothing corresponds to stored signatures for each tracked location for clothing of any of the set of enrolled players;

(c) determining whether the signature for the person'"'"'s height corresponds to stored signatures for each tracked location for height of any of the set of enrolled players; and

making an aggregate decision as to whether the person is the specific enrolled player based on steps (a), (b) and (c).

8. The method of claim 1, wherein:

the interacting with the person based on the mapping includes controlling an avatar in a video game based on motion of the person and outputting display signals indicating the video game with the avatar.

9. The method of claim 1, wherein the computer based application is a video game, the method further comprises:

determining that insufficient data exists for identifying the person;

requesting the person to perform a specific act of motion as part of the video game in response to determining that insufficient data exists for identifying the person; and

visually detecting sufficient data in response to the specific act.

10. A video game apparatus, comprising:

an interface to a display;

a sensor system;

a memory; and

one or more processors in communication with the interface and the sensor system and having access to the memory;

the one or more processors establish thresholds, enroll players in a video game including identifying users of the video game apparatus as members of a set of already enrolled players, wherein the one or more processors previously identified the set of already enrolled players using a sensor system by having detected one or more physical features of each of the set of already enrolled players with respect to one or more tracked locations in a physical real-world play space and having generated and having stored in the memory, for each of the set of already enrolled players, one or more stored visual signatures of the detected one or more physical features for each tracked location;

the one or more processors execute the video game including interacting with one or more of the already enrolled players in the physical real-world play space based on the enrolling, and determine that a person has become detectable in the physical real-world play space of the video game;

the one or more processors automatically determine based on data from the sensor system whether the person that has become detectable in the physical real-world play space is one of the set of already enrolled players by

detecting one or more physical features of the person and tracking one or more locations in the physical real-world play space of the detected one or more physical features of the person,

generating one or more respective visual signatures of the detected one or more physical features of the person for each tracked location,

identifying whether the person is a specific already enrolled player of the set of already enrolled players based on comparing the one or more respective visual signatures to the one or more stored visual signatures of the set of already enrolled players,

responsive to the one or more stored visual signatures of a specific already enrolled player matching the one or more respective visual signatures of the person, concluding that an identification of the person is the identification of the specific already enrolled player of the set of already enrolled players, and

responsive to no match between the one or more respective visual signatures of the person and the one or more stored visual signatures for the set of already enrolled players, assigning a new identification as the identification of the person and interacting with the person in the video game based on the new identification if it is determined that the person is not one of the enrolled players, and

mapping the person to the identification of the person, and interacting with the person in the video game based on the mapping.

11. The video game apparatus of claim 10, wherein:

the sensor system senses visual data of the person and depth data of the person, the one or more processors automatically determine whether the person is one of the enrolled players based on the visual data and the depth data.

12. The video game apparatus of claim 10, wherein:

the one or more processors dynamically update the thresholds while executing the video game.

13. The video game apparatus of claim 10, wherein:

at least one of the one or more respective visual signatures of the person is generated using depth data from the sensor system, the depth data includes a two dimensional depth image with pixels having magnitudes that represent distance from the sensor system.

14. The video game apparatus of claim 13, wherein:

the one or more respective visual signatures of the person includes a visual signature for an appearance of the person'"'"'s face, a visual signature for an appearance of the person'"'"'s clothing and a visual signature for the person'"'"'s height.

15. The video game apparatus of claim 10, wherein:

the one or more processors interact with the person in the video game by controlling an avatar in the video game based on motion of the person.

16. One or more processor readable storage devices having processor readable code embodied on said one or more processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising:

receiving visual data and depth data from one or more sensors pertaining to a first entity in a physical real world space, the first entity not currently interacting with an application, the depth data includes a two dimensional depth image with pixels of the depth image each having amagnitude that represents a distance from the one or more sensors;

detecting one or more physical features of the first entity and tracking one or more locations in the physical real-world space of the detected one or more physical features of the first entity;

creating a first set of visual signatures of the detected one or more physical features of the first entity for each tracked location based on the visual data and depth data, the first set of visual signatures are created by applying one or more mathematical functions to the visual data and depth data to create a mathematical representation of the visual data and depth data;

determining whether the first set of visual signatures for each tracked location of the first entity not currently interacting with the application matches at least one of previously created stored sets of visual signatures for each tracked location for one or more entities already enrolled for the application based on comparing the first set of visual signatures for each tracked location of the first entity not currently interacting with the application to the previously created stored sets of visual signatures for each tracked location for the one or more entities already enrolled for the application; and

responsive to the first set of visual signatures for each tracked location of the first entity not currently interacting with the application matching the at least one of the stored sets of visual signatures, mapping the first entity to the at least one entity already enrolled for the application associated with the at least one of the stored sets of visual signatures, the mapping allows the first entity to interact with the application.

17. One or more processor readable storage devices according to claim 16, wherein:

the first set of visual signatures for each location includes a signature for an appearance of the first entity'"'"'s face, a signature for an appearance of the first entity'"'"'s clothing and a signature for the first entity'"'"'s height.