Method and System for Vision-Based Interaction in a Virtual Environment

• US 20080225041A1
• Filed: 02/08/2008
• Published: 09/18/2008
• Est. Priority Date: 02/08/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Edge 3 Technologies, LLC
• Inventors: Holmes, James E., King, Joshua E., Gigliotti, Justin R., El Dokor, Tarek, Glomski, William E.
• Granted Patent: US 8,144,148 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 2300/1018 Calibration; Key and button assignment
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/208 for storing personal settings or data of the player
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: E05Y 2900/132 Doors
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering

Abstract

A method and system for vision-based interaction in a virtual environment is disclosed. According to one embodiment, a computer-implemented method comprises receiving data from a plurality of sensors to generate a meshed volumetric three-dimensional representation of a subject. A plurality of clusters is identified within the meshed volumetric three-dimensional representation that corresponds to motion features. The motion features include hands, feet, knees, elbows, head, and shoulders. The plurality of sensors is used to track motion of the subject and manipulate the motion features of the meshed volumetric three-dimensional representation.

Claims

1. A computer-implemented method, comprising:

receiving data from a plurality of sensors to generate a meshed volumetric three-dimensional representation of a subject;

identifying a plurality of clusters within the meshed volumetric three-dimensional representation that correspond to motion features, the motion features including hands, feet, knees, elbows, head and shoulders; and

using the plurality of sensors to track motion of the subject and manipulate the motion features of the meshed volumetric three-dimensional representation.

2. The computer-implemented method of claim 1, wherein the plurality of sensors include a distributed network of time of flight sensors.

3. The computer-implemented method of claim 1, further comprising identifying one or more events corresponding to the motion of the subject.

4. The computer-implemented method of claim 3, wherein identifying one or more events further comprises matching the motion of the subject with an event stored in an event library.

5. The computer-implemented method of claim 4, further comprising adding new events to the event library using a supervised computer-implemented learning process that uses Hidden Markov and Bayesian models.

6. The computer-implemented method of claim 5, further comprising generating the new events taking into account anatomical constraints, sensor resolution, pixel depth and a total number of sensors.

7. The computer-implemented method of claim 4, further comprising displaying the meshed volumetric three-dimensional representation, wherein the meshed volumetric three-dimensional representation moves according to instructions generated by a software development kit that links the motion features to a skeletal structure.

8. The computer-implemented method of claim 7, wherein the software development kit provides a graphical representation of the skeletal structure having a plurality of points corresponding to the motion features, the plurality of points used to select a desired motion feature.

9. A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions when executed by a computer, cause the computer to perform:

receiving data from a plurality of sensors to generate a meshed volumetric three-dimensional representation of a subject;

identifying a plurality of clusters within the meshed volumetric three-dimensional representation that correspond to motion features, the motion features including hands, feet, knees, elbows, head, and shoulders; and

using the plurality of sensors to track motion of the subject and manipulate the motion features of the meshed volumetric three-dimensional representation.

10. The computer-readable medium of claim 9, wherein the plurality of sensors include a distributed network of time of flight sensors.

11. The computer-readable medium of claim 9, having stored thereon additional instructions, the additional instructions when executed by a computer, cause the computer to further perform identifying one or more events corresponding to the motion of the subject.

12. The computer-readable medium of claim 11, having stored thereon additional instructions, the additional instructions when executed by a computer for identifying one or more events, cause the computer to further perform matching the motion of the subject with an event stored in an event library.

13. The computer-readable medium of claim 12, having stored thereon additional instructions, the additional instructions when executed by a computer, cause the computer to further perform adding new events to the event library using a supervised computer-implemented learning process that uses Hidden Markov and Bayesian models.

14. The computer-readable medium of claim 13, having stored thereon additional instructions, the additional instructions when executed by a computer, cause the computer to further perform generating the new events taking into account anatomical constraints, sensor resolution, pixel depth and a total number of sensors.

15. The computer-readable medium of claim 12, having stored thereon additional instructions, the additional instructions when executed by a computer, cause the computer to further perform displaying the meshed volumetric three-dimensional representation, wherein the meshed volumetric three-dimensional representation moves according to instructions generated by a software development kit that links the motion features to a skeletal structure.

16. The computer-readable medium of claim 15, wherein the software development kit provides a graphical representation of the skeletal structure having a plurality of points corresponding to the motion features, the plurality of points used to select a desired motion feature.