Method and apparatus for tracking of a plurality of subjects in a video game

• US 9,272,202 B2
• Filed: 02/10/2013
• Issued: 03/01/2016
• Est. Priority Date: 02/08/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Edge Technologies, Inc
• Inventors: El Dokor, Tarek, King, Joshua E., Holmes, James E., Gigliotto, Justin R., Glomski, William E.
• Primary Examiner(s): Laneau, Ronald
• Application Number:
  US13/763,721
• Publication Number:
  US 20130150160A1
• Time in Patent Office: 1,115 Days
• Field of Search: 463 30- 33, 345/158, 345/181, 345/424, 348/46
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 2300/1018 Calibration; Key and button assignment
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/208 for storing personal settings or data of the player
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: E05Y 2900/132 Doors
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering

Abstract

Method, computer program and system for tracking movement of a subject. The method includes receiving data from a distributed network of camera sensors employing one or more emitted light sources associated with one or more of the one or more camera sensors to generate a volumetric three-dimensional representation of the subject, identifying a plurality of clusters within the volumetric three-dimensional representation that correspond to motion features indicative of movement of the motion features of the subject, presenting one or more objects on one or more three dimensional display screens, and using the plurality of fixed position sensors to track motion of the motion features of the subject and track manipulation of the motion features of the volumetric three-dimensional representation to determine interaction of one or more of the motion features of the subject and one or more of the one or more objects on the three dimensional display.

Claims

1. A system for tracking movement of motion features of a plurality of subjects for operating a video game, comprising:

a plurality of fixed position sensors comprising a distributed network of camera sensors including one or more emitted light sources associated with one or more of the one or more camera sensors, the plurality of fixed position sensors capturing data and generating a volumetric three-dimensional representation for a first subject and a second subject;

a processor for identifying one or more motion features of each of the first and second subjects within the three-dimensional representation indicative of motion of one or more motion features of the first subject relative to the fixed position sensors, one or more other portions of the first subject, and one or more portions of the second subject;

one or more three dimensional displays for presenting the generated volumetric three-dimensional representation for the first subject and for the second subject; and

the processor further using the plurality of fixed position sensors to track motion of the one or more motion features of the first subject to determine interaction of the one or more motion features of the first subject and one or more portions of the second subject on one or more of the one or more three dimensional displays.

2. The system of claim 1, wherein the one or more three dimensional displays presents one or more objects;

and

wherein the plurality of fixed position sensors track motion of the one or more motion features of the first subject to determine interaction of the one or more motion features of the first subject and the one or more objects on one or more of the one or more three dimensional displays.

3. The system of claim 1, the processor further identifying one or more recognized events within the video game corresponding to the tracked motion of the first subject.

4. The system of claim 3, wherein identifying one or more events further comprises matching the motion of the first subject with an event stored in an event library associated with the video game.

5. The system of claim 4, wherein the processor further generates new events for the library associated with the video game taking into account one or more of the anatomical constraints of the first subject, sensor resolution, pixel depth and a total number of sensors.

6. The system of claim 1, wherein the processor further:

acquires an image of the face of the first and second subjects playing the video game; and

integrates a non-photo-real avatar resembling the acquired image of the first and second subjects into first and second characters in the video game.

7. The system of claim 6, wherein the processor further warps each of the non-photo-real avatars to best match a non-photo-real color space associated with the video game.

8. The system of claim 6, wherein each of the non-photo-real avatars changes according to one or more game conditions.

9. The system of claim 8, wherein the one or more game conditions comprises a change in skill level of a character.

10. The system of claim 8, wherein the changing of the non-photo-real avatar represents a nonlinear transformation.

11. The system of claim 10, wherein the nonlinear transformation comprises eigen faces fusion between the avatar and the subject.

12. The system of claim 10, wherein the nonlinear transformation comprises landmarking between the avatar and the subject.

13. The system of claim 1, wherein the one or more three dimensional displays comprise autostereoscopic displays.

14. The system of claim 1, wherein the one or more three dimensional displays employ a three dimensional viewing device.

15. The system of claim 14, wherein the three dimensional viewing device comprises three dimensional viewing glasses.

16. The system of claim 1, wherein the fixed position sensors are time-of-flight sensors.

17. A system for tracking motion of a subject during operation of a video game, comprising:

a distributed network of camera sensors employing one or more emitted light sources associated with one or more of the one or more camera sensors, at least data received from the camera sensors being used to generate a volumetric three-dimensional representation of each of a first and second subject;

a processor for identifying one or more motion features of each of the first and second subjects within the three-dimensional representation indicative of motion of one or more motion features of the first subject relative to the fixed position sensors, one or more other portions of the first subject and one or more portions of the second subject;

one or more three dimensional displays for representing the generated volumetric three-dimensional representations for each of the first and second subjects; and

wherein the plurality of fixed position sensors are used to track motion of the one or more motion features of the first subject to determine interaction of the one or more motion features of the first subject and one or more portions of the second subject on one or more of the one or more three dimensional displays.

18. The system of claim 17, wherein the plurality of fixed position sensors comprises a distributed grid of sensors.

19. The system of claim 18, wherein the distributed grid of sensors further comprises a plurality of light sources.

20. The system of claim 17, wherein a representation of the volumetric three-dimensional representation for each of the first and second subjects is displayed on one or more of the three dimensional displays, wherein each of the volumetric three-dimensional representations move according to instructions generated by a software development kit that links the motion features to a skeletal structure.

21. The system of claim 20, wherein the one or more displays display interaction between the representations of the volumetric three-dimensional representations and one or more objects within the video game.

22. The system of claim 17, wherein the processor further identifies one or more recognized events within the video game corresponding to the tracked motion of the first subject.

23. The system of claim 22, wherein identifying one or more events by the processor further comprises matching the motion of the first subject with an event stored in an event library associated with the video game.

24. The system of claim 23, wherein the processor generates new events for the library associated with the video game taking into account one or more of the anatomical constraints of the subject, sensor resolution, pixel depth, location of light source, and a total number of sensors.

25. The system of claim 17, wherein the processor acquires an image of the face of the first and second subjects playing the video game in accordance with at least data received from the camera sensor, and integrates a non-photo-real avatar resembling the acquired image of the first and second subjects into a character in the video game.

26. The system of claim 25, wherein the non-photo-real avatars are warped by the processor to best match a non-photo-real color space associated with the video game.

27. The system of claim 25, wherein the processor changes the non-photo-real avatars according to one or more game conditions.

28. The system of claim 27, wherein the one or more game conditions comprises a change in skill level of a character controlled by one of the first and second subjects.

29. The system of claim 27, wherein the changing of the non-photo-real avatar represents a nonlinear transformation.

30. The system of claim 29, wherein the nonlinear transformation comprises eigen faces fusion between the avatar and the corresponding one of the first and second subjects.

31. The system of claim 29, wherein the nonlinear transformation comprises landmarking between the avatar and corresponding ones of the first and the second subjects.

32. The system of claim 17, wherein the one or more three dimensional displays comprise autostereoscopic displays.

33. The system of claim 17, wherein the one or more three dimensional displays employ a three dimensional viewing device.

34. The system of claim 33, wherein the three dimensional viewing device comprises three dimensional viewing glasses.

35. The system of claim 17, wherein the fixed position sensors are time-of-flight sensors.

36. A non-transitory computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions when executed by a computer coupled to a plurality of fixed position sensors for receiving data and one or more three dimensional displays for presenting one or more objects thereon, causes the computer to track movement of motion features of a subject operating a video game by performing the steps of:

receiving data from the plurality of fixed position sensors comprising a distributed network of camera sensors employing one or more emitted light sources associated with one or more of the one or more camera sensors to generate a volumetric three-dimensional representation for a first subject and a second subject;

identifying one or more motion features of each of the first and second subjects within the three-dimensional representation indicative of motion of one or more motion features of the first subject relative to the fixed position sensors one or more other portions of the first subject, and one or more portions of the second subject;

presenting the generated volumetric three-dimensional representation for the first subject and for the second subjects on the one or more three dimensional displays; and

using the plurality of fixed position sensors to track motion of the one or more motion features of the first subject to determine interaction of the one or more motion features of the first subject and one or more portions of the second subject on one or more of the one or more three dimensional displays.

37. The computer-readable medium of claim 36, having stored thereon additional instructions, the additional instructions when executed by the computer coupled to the plurality of fixed position sensors for receiving data and one or more three dimensional displays for presenting one or more objects thereon, cause the computer to further perform the steps of:

presenting one or more objects on the one or more three dimensional displays; and

using the plurality of fixed position sensors to track motion of the one or more motion features of the first subject to determine interaction of the one or more motion features of the first subject and the one or more objects on one or more of the one or more three dimensional displays.

38. The computer-readable medium of claim 36, having stored thereon additional instructions, the additional instructions when executed by a computer coupled to the plurality of fixed position sensors for receiving data and one or more three dimensional displays for presenting one or more objects thereon, cause the computer to further perform the step of identifying one or more recognized events within the video game corresponding to the tracked motion of the first subject.

39. The computer-readable medium of claim 38, wherein identifying one or more events further comprises matching the motion of the first subject with an event stored in an event library associated with the video game.

40. The computer-readable medium of claim 39, having stored thereon additional instructions, the additional instructions when executed by a computer coupled to the plurality of fixed position sensors for receiving data and one or more three dimensional displays for presenting one or more objects thereon, cause the computer to further perform the step of generating new events for the library associated with the video game taking into account anatomical constraints of the first subject, sensor resolution, pixel depth and a total number of sensors.

41. The computer-readable medium of claim 36, having stored thereon additional instructions, the additional instructions when executed by a computer coupled to the plurality of fixed position sensors for receiving data and one or more three dimensional displays for presenting one or more objects thereon, cause the computer to further perform the steps of:

acquiring an image of the face of the first and second subjects playing the video game; and

integrating a non-photo-real avatar resembling the acquired image of the first and second subjects into first and second characters in the video game.

42. The computer-readable medium of claim 41, wherein each of the non-photo-real avatars is warped to best match a non-photo-real color space associated with the video game.

43. The computer-readable medium of claim 41, wherein each of the non-photo-real avatars changes according to one or more game conditions.

44. The computer-readable medium of claim 43, wherein the one or more game conditions comprises a change in skill level of a character.

45. The computer-readable medium of claim 43, wherein the changing of the non-photo-real avatar represents a nonlinear transformation.

46. The computer-readable medium of claim 45, wherein the nonlinear transformation comprises eigen faces fusion between the avatar and the subject.

47. The computer-readable medium of claim 45, wherein the nonlinear transformation comprises landmarking between the avatar and the subject.

48. The computer-readable medium of claim 36, wherein the one or more three dimensional displays comprise autostereoscopic displays.

49. The computer-readable medium of claim 36, wherein the one or more three dimensional displays employ a three dimensional viewing device.

50. The computer-readable medium of claim 49, wherein the three dimensional viewing device comprises three dimensional viewing glasses.

51. The computer-readable medium of claim 36, wherein the fixed position sensors are time-of-flight sensors.