Method and system for vision-based interaction in a virtual environment

• US 8,223,147 B1
• Filed: 03/05/2012
• Issued: 07/17/2012
• Est. Priority Date: 02/08/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Edge 3 Technologies, LLC
• Inventors: El Dokor, Tarek, King, Joshua E, Holmes, James E, Gigliotti, Justin R, Glomski, William E
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Chin, Michelle
• Application Number:
  US13/411,657
• Publication Number:
  US 20120196660A1
• Time in Patent Office: 134 Days
• Field of Search: None
• US Class Current: 345/424
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 2300/1018 Calibration; Key and button assignment
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/208 for storing personal settings or data of the player
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: E05Y 2900/132 Doors
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering

Abstract

Method, computer program and system for tracking movement of a subject within a video game. The method includes receiving data from a plurality of fixed position sensors comprising a distributed network of time of flight camera sensors to generate a volumetric three-dimensional representation of the subject, identifying a plurality of clusters within the volumetric three-dimensional representation that correspond to features indicative of movement of the subject relative to the fixed position sensors and the subject, presenting one or more objects as the subject of a video game on one or more three dimensional display screens, and using the plurality of fixed position sensors to track motion of the features of the subject to determine interaction of one or more of the features of the subject and one or more of the one or more objects on one or more of the one or more the three dimensional display screens.

Claims

1. A computer-implemented method for tracking movement of a subject within a video game, comprising:

receiving data from a plurality of fixed position sensors comprising a distributed network of time of flight camera sensors to generate a volumetric three-dimensional representation of the subject;

identifying a plurality of clusters within the volumetric three-dimensional representation that correspond to features indicative of movement of the subject relative to the fixed position sensors and one or more other portions of the subject;

presenting one or more objects as the subject of a video game on one or more three dimensional display screens; and

using the plurality of fixed position sensors to track motion of the features of the subject to determine interaction of one or more of the features of the subject and one or more of the one or more objects on one or more of the one or more the three dimensional display screens.

2. The computer-implemented method of claim 1, further comprising identifying one or more events corresponding to the tracked motion of the subject.

3. The computer-implemented method of claim 2, wherein identifying one or more events further comprises matching the motion of the subject with an event stored in an event library.

4. The computer-implemented method of claim 3, wherein one or more of the one or more events comprises a user punching in the video game.

5. The computer-implemented method of claim 4, wherein data associated with the motion associated with the punching comprises a best fit path.

6. The computer-implemented method of claim 3, wherein one or more of the one or more events comprises a user kicking in a video game.

7. The computer-implemented method of claim 3, wherein the one or more events takes the place of an event traditionally invoked on a game controller.

8. The computer-implemented method of claim 3, wherein the one or more events may be used in conjunction with a traditional video game controller.

9. The computer-implemented method of claim 1, wherein the object presented on one or more of the one or more three dimensional display screens comprises a volumetric three-dimensional representation of a second subject.

10. The computer-implemented method of claim 9, wherein the interaction between the subject and the second subject takes place in a multi-player video game.

11. The computer-implemented method of claim 1, further comprising the steps of:

acquiring an image of the face of the user; and

integrating a non-photoreal avatar resembling the acquired image of the user into the video game.

12. The computer-implemented method of claim 11, wherein the nonphotoreal avatar may include one or more user-populated features selected from the group of characters, moods and facial expressions.

13. The computer-implemented method of claim 1, wherein the one or more of the features of the subject include one or more of hands, feet, knees, elbows, head and shoulders.

14. The computer-implemented method of claim 13, wherein the movement of one or more of the features of the subject is interpreted to control a video game.

15. The computer-implemented method of claim 14, wherein a position of the movement relative to a location of one or more of the one or more three-dimensional display screens is employed to interpret the movement.

16. A non-transitory computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions when executed by a computer, cause the computer to perform the steps of:

receiving data from a plurality of fixed position sensors comprising a distributed network of time of flight camera sensors to generate a volumetric three-dimensional representation of the subject;

identifying a plurality of clusters within the volumetric three-dimensional representation that correspond to features indicative of movement of the subject relative to the fixed position sensors and one or more other portions of the subject;

presenting one or more objects as the subject of a video game on one or more three dimensional display screens; and

using the plurality of fixed position sensors to track motion of the features of the subject to determine interaction of one or more of the features of the subject and one or more of the one or more objects on one or more of the one or more the three dimensional display screens.

17. The computer-readable medium of claim 16, having stored thereon additional instructions, the additional instructions when executed by a computer, cause the computer to further perform the step of identifying one or more events corresponding to the tracked motion of the subject.

18. The computer-readable medium of claim 17, having stored thereon additional instructions, the additional instructions when executed by a computer, cause the computer to further perform the step of matching the motion of the subject with an event stored in an event library.

19. The computer-readable medium of claim 18, wherein one or more of the one or more events comprises a user punching in the video game.

20. The computer-readable medium of claim 19, wherein data associated with the motion associated with the punching comprises a best fit path.

21. The computer-readable medium of claim 18, wherein one or more of the one or more events comprises a user kicking in a video game.

22. The computer-readable medium of claim 16, having stored thereon additional instructions, the additional instructions when executed by a computer, cause the computer to further perform the step of displaying the volumetric three-dimensional representation.

23. The computer-readable medium of claim 22, wherein the volumetric three-dimensional representation moves according to instructions generated by a software development kit that links the motion features to a skeletal structure.

24. The computer-readable medium of claim 16, wherein the tracked motion is used to operate the video game.

25. A video game system for tracking motion of a subject for use with a video game, comprising:

a plurality of fixed position sensors comprising a distributed network of time of flight camera sensors, at least data received from the plurality of fixed position sensors being used to generate a volumetric three-dimensional representation of the subject;

a processor comprising a portion of the video game system for identifying a plurality of clusters within the volumetric three-dimensional representation that correspond to features indicative of motion of the subject relative to the fixed position sensors and one or more other portions of the subject; and

one or more three dimensional display screens for presenting one or more objects thereon corresponding to one or more objects in the video game; and

wherein the plurality of fixed position sensors is used to track motion of the features of the subject to manipulate the volumetric three-dimensional representation, and the processor is used to determine interaction of one or more of the features of the subject and one or more of the one or more objects on the one or more three dimensional display screens indicative of an interaction in the video game.

26. The system of claim 25, wherein each of the one or more three dimensional display screens displays three dimensional information without the use of special eyeware by the subject.

27. The system of claim 25, wherein each of the one or more three dimensional screens displays three dimensional information employing special eyeware by the subject.

28. The system of claim 25, wherein the tracked motion is identified as corresponding to one or more events.

29. The system of claim 25, wherein one or more of the plurality of fixed position sensors is mounted in a bezel of one or more of the one or more three dimensional displays.

30. The system of claim 25, wherein one or more of the plurality of fixed position sensors is mounted adjacent one or more of the one or more three dimensional displays.

31. The system of claim 30, wherein one or more of the plurality of fixed position sensors is mounted in a wall.

32. The system of claim 25, wherein one or more of the plurality of fixed position sensors is positioned within a discrete unit separated from each of the one or more three dimensional displays.

33. The system of claim 32, wherein the discrete unit is associated with a video game console.

34. The system of claim 32, wherein the discrete unit is mounted adjacent one or more of the one or more three dimensional display screens.

35. The video game system of claim 25, wherein the video game system is a portable system.

36. The video game system of claim 35, wherein the fixed position sensors are fixed relative to the one or more three dimensional display screens.

37. The video game system of claim 35, wherein the fixed position sensors are fixed independent of the one or more three dimensional display screens.

38. The video game system of claim 25, wherein the video game system is a self-contained portable video game system.

39. The system of claim 25, wherein the plurality of fixed position sensors comprises a distributed grid of sensors.

40. The system of claim 39, wherein the grid of sensors is synchronous.

41. The system of claim 39, wherein the grid of sensors is asynchronous.