Visual based identitiy tracking

• US 8,864,581 B2
• Filed: 01/29/2010
• Issued: 10/21/2014
• Est. Priority Date: 01/29/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Leyvand, Tommer, Dernis, Mitchell Stephen, Li, Jinyu, Wei, Yichen, Sun, Jian, Meekhof, Casey Leon, Keosababian, Timothy Milton
• Primary Examiner(s): Shah, Milap
• Application Number:
  US12/696,282
• Publication Number:
  US 20110190055A1
• Time in Patent Office: 1,726 Days
• Field of Search: 345156-158, 463/1, 463 30- 33, 463 40- 43, 382/103, 382/118, 382/119, 382/154
• US Class Current: 463/30
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

A video game system (or other data processing system) can visually identify a person entering a field of view of the system and determine whether the person has been previously interacting with the system. In one embodiment, the system establishes thresholds, enrolls players, performs the video game (or other application) including interacting with a subset of the players based on the enrolling, determines that a person has become detectable in the field of view of the system, automatically determines whether the person is one of the enrolled players, maps the person to an enrolled player and interacts with the person based on the mapping if it is determined that the person is one of the enrolled players, and assigns a new identification to the person and interacts with the person based on the new identification if it is determined that the person is not one of the enrolled players.

Claims

1. A method of creating a signature of a person'"'"'s face for use in a computer based gaming application, comprising:

executing, by a computing system, the computer based gaming application, the computer based gaming application including interaction with a subset of a set of previously enrolled players;

determining, by the computing system, that a person has become detectable in a physical play space of the computer based gaming application, the person not being one of the subset of the set of previously enrolled players currently interacting with the computer based gaming application;

receiving, at the computing system, a depth image of the person obtained from a depth image sensor, the depth image comprises a two dimensional arrangement of pixels, each pixel of the depth image stores a number identifying a distance from the depth image sensor to a body of the person;

receiving, at the computing system, a visual image of the person from a visual image sensor, the visual image being different than the depth image;

correlating, by the computing system, the depth image to the visual image so that pixels in the visual image are identified to correspond to pixels in the depth image;

identifying, by the computing system, foreground pixels and background pixels in the two dimensional arrangement of pixels in the depth image;

removing, by the computing system, the background pixels from the depth image;

assigning, by the computing system, the foreground pixels of the depth image to different parts of the body of the person;

using, by the computing system, the different parts of the body of the person to construct a model;

utilizing, by the computing system, the model to estimate a position of the person'"'"'s face in the depth image by using one or more criteria to locate eyes and nose of the person;

automatically, by the computing system, determining a bounding box for the depth image that surrounds the person'"'"'s face;

automatically, by the computing system, identifying pixels in the visual image that correspond to pixels in the depth image inside the bounding box;

automatically, by the computing system, creating and storing a new signature of the person'"'"'s face that represents the pixels in the visual image that correspond to the pixels in the depth image inside the bounding box;

automatically, by the computing system, identifying the person as a specific previously enrolled player of the set of previously enrolled players based on comparing the new signature of the person'"'"'s face to stored signatures of the set of previously enrolled players; and

mapping, by the computing system, the person to the determined specific enrolled player and enabling the person to interact with the computer based gaming application based on the mapping.

2. The method of claim 1, further comprising:

adjusting, by the computing system, the bounding box to more tightly bound the eyes and nose.

3. The method of claim 1, further comprising:

realigning, by the computing system, the visual image so that the face is at a predetermined reference alignment.

4. The method of claim 1, further comprising:

smoothing, by the computing system, high variance noise from the depth image.

5. The method of claim 1, further comprising:

creating, by the computing system, a multi-dimensional vector for pixels of the visual image that correspond to pixels in the depth image inside the bounding box and transforming the vector to Principal Component Analysis space.

6. The method of claim 1, further comprising:

establishing thresholds, by the computing system, the thresholds are used when automatically identifying the person as the specific enrolled player; and

dynamically updating, by the computing system, the thresholds while and based on performing the computer based gaming application.

7. The method of claim 1, further comprising:

mapping, by the computing system, the person to an object associated with the computer based gaming application based on the signature of the person'"'"'s face; and

interacting, by the computing system, with the person using the computer based gaming application based on the mapping.

8. The method of claim 7, further comprising:

the interacting, by the computing system, with the person includes controlling an avatar in a video game based on motion of the person.

9. The method of claim 1, further comprising:

determining, by the computing system, that insufficient data exists for identifying the person;

requesting, by the computing system, the person to perform a specific act in response to determining that insufficient data exists for identifying the person; and

obtaining, by the computing system, sufficient data in response to the specific act.

10. A computing apparatus, comprising:

a sensor system for providing one or more depth images and one or more visual images; and

one or more processors for communication with the sensor system, the one or more processors for execution of a computer based gaming application, the computer based gaming application including interaction with a subset of a set of previously enrolled players;

the one or more processors for determining that a person has become detectable in a physical play space of the computer based gaming application, the person not being one of the subset of the set of previously enrolled players currently interacting with the computer based gaming application;

the one or more processors for receiving a depth image of the person from the sensor system, the depth image comprises a two dimensional arrangement of pixels, each pixel of the depth image stores a number identifying a distance from the sensor system to a body of the person;

the one or more processors for identifying foreground pixels and background pixels in the two dimensional arrangement of pixels in the depth image;

the one or more processors for receiving a visual image of the person from the sensor system, the visual image being different than the depth image;

the one or more processors for correlating the depth image to the visual image so that pixels in the visual image are identified to correspond to pixels in the depth image;

the one or more processors for assigning the foreground pixels to different parts of the body of the person;

the one or more processors for using the different parts of the body to construct a model and using the model to estimate a position of a person'"'"'s face in the visual image that corresponds to the position of the person'"'"'s face in the depth image by using one or more criteria to locate eyes and nose of the person;

the one or more processors for determining a bounding box for the depth image that surrounds the person'"'"'s face;

the one or more processors for identifying pixels in the visual image that correspond to pixels in the depth image inside the bounding box;

the one or more processors for creating a new signature of the person'"'"'s face representing pixels in the visual image that correspond to the pixels in the depth image inside the bounding box;

the one or more processors for identifying the person as a specific previously enrolled player of the set of previously enrolled players based on comparing the new signature of the person'"'"'s face to stored signatures of the set of previously enrolled players; and

the one or more processors for mapping the person to the determined specific enrolled player and enabling the person to interact with the computer based gaming application based on the mapping.

11. The computing apparatus of claim 10, wherein:

the one or more processors for creating the new signature of the person'"'"'s face by creating a multi-dimensional vector for pixels in the visual image of the person at the position of the person'"'"'s face.

12. The computing apparatus of claim 10, wherein:

the sensor system includes a depth camera and visual image camera.

13. The computing apparatus of claim 10, further comprising:

a display, the display is updated based on recognizing the person'"'"'s face.

14. The computing apparatus of claim 10, wherein:

the one or more processors for mapping the person to an object associated with the computer based gaming application based on recognizing the person'"'"'s face.

15. One or more processor readable nonvolatile storage devices having processor readable code embodied on said one or more processor readable non-transitory storage devices, said processor readable code for programming one or more processors to perform a method comprising:

executing a computer based gaming application, the computer based gaming application including interaction with a subset of a set of previously enrolled players;

determining that a person has become detectable in a physical play space of the computer based gaming application, the person not being one of the subset of the set of previously enrolled players currently interacting with the computer based gaming application;

receiving a depth image of the person obtained from a depth image sensor, the depth image comprises a two dimensional arrangements of pixels, each pixel of the depth image stores a number identifying a distance from the depth image sensor to a body of the person;

receiving a visual image of the person from a visual image sensor, the visual image being different than the depth image;

correlating the depth image to the visual image so that pixels in the visual image are identified to correspond to pixels in the depth image;

identifying foreground pixels and background pixels in the two dimensional arrangement of pixels in the depth image;

assigning each of the foreground pixels to a different part of the body of the person;

using the different parts of the body to construct a model;

utilizing the model to estimate a position of a person'"'"'s face in the visual image that corresponds to the position of the person'"'"'s face in the depth image by using one or more criteria to locate eyes and nose of the person;

determining a bounding box for the depth image that surrounds the person'"'"'s face;

identifying pixels in the visual image that correspond to pixels in the depth image inside the bounding box;

determining a new signature for an appearance of the person'"'"'s face representing pixels in the visual image that correspond to the pixels in the depth image inside the bounding box;

identifying the person as a specific previously enrolled player of the set of previously enrolled players based on comparing the new signature to stored signatures of the set of previously enrolled players; and

mapping the person to the determined specific enrolled player and enabling the person to interact with the computer based gaming application based on the mapping.

16. One or more processor readable nonvolatile storage devices according to claim 15, wherein:

the method further comprises mapping the person to an object associated with a computer based gaming application based on the signature of the person'"'"'s face.