Mapping a natural input device to a legacy system

• US 8,448,094 B2
• Filed: 03/25/2009
• Issued: 05/21/2013
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kipman, Alex, Polzin, R. Stephen, Tsunoda, Kudo, Bennett, Darren, Latta, Stephen, Finocchio, Mark, Snook, Gregory G., Markovic, Relja
• Primary Examiner(s): Sax, Steven
• Assistant Examiner(s): VARGA, WILSON V
• Application Number:
  US12/411,276
• Publication Number:
  US 20100199229A1
• Time in Patent Office: 1,518 Days
• Field of Search: 345156-184, 715/863
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods for mapping natural input devices to legacy system inputs are disclosed. One example system may include a computing device having an algorithmic preprocessing module configured to receive input data containing a natural user input and to identify the natural user input in the input data. The computing device may further include a gesture module coupled to the algorithmic preprocessing module, the gesture module being configured to associate the natural user input to a gesture in a gesture library. The computing device may also include a mapping module to map the gesture to a legacy controller input, and to send the legacy controller input to a legacy system in response to the natural user input.

Claims

1. A method for using a natural input device to control a legacy gaming system, the method comprising:

receiving a natural user input from the natural input device, the natural input device distinct from a legacy controller device of the legacy gaming system, the natural user input simulating a game motion that corresponds to a motion a legacy game input represents,

wherein receiving the natural user input includes receiving a depth image from an image capture device including a depth camera;

selecting a gesture library from a plurality of gesture libraries based on a legacy gaming program running on the legacy gaming system;

associating the natural user input with a first gesture from the gesture library selected;

mapping the associated first gesture to the corresponding input of the legacy controller device;

sending the corresponding input to the legacy gaming system in response to receiving the natural user input; and

adding a second gesture to a gesture library in the plurality of gesture libraries.

2. The method of claim 1, wherein mapping the first gesture to the corresponding input of the legacy controller device includes:

mapping the first gesture to a legacy controller state using a look up table; and

generating the corresponding input of the legacy controller device via a legacy controller emulator.

3. The method of claim 1, wherein the depth image includes depth values for objects in a field of view of the depth camera, the method further comprising processing the depth image to identify the natural user input.

4. The method of claim 3, wherein processing the depth image to identify the natural user input comprises:

removing at least one background element from the depth image;

segmenting at least one player of the legacy gaming system within the depth image;

mapping a skeleton to the at least one segmented player;

associating a change in the skeleton with the first gesture; and

identifying the natural user input based on the first gesture.

5. The method of claim 4, wherein each of the plurality of gesture libraries corresponds to a different game genre.

6. The method of claim 1, wherein the corresponding input of the legacy controller device is a first corresponding input, the method further comprising:

mapping the second gesture to a second corresponding input of the legacy controller device; and

sending the second corresponding input to the legacy gaming system in response to receiving natural user input associated with the second gesture.

7. The method of claim 1 wherein the legacy gaming program is configured to work with the legacy controller device and not designed to be compatible with the natural input device.

8. A computing device for using a natural input device to control a legacy gaming system, the device comprising:

an algorithmic preprocessing module configured to receive data from a natural input device distinct from a legacy controller device of the legacy gaming system, and to identify in the data a natural user input, the natural user input simulating a game motion that corresponds to a motion a legacy game input represents,

wherein the data includes a depth image from an image capture device including a depth camera;

a gesture module configured to select a gesture library from a plurality of gesture libraries based on a legacy gaming program running on the legacy gaming system, to associate the natural user input with a first gesture from the selected gesture library, and to add a second gesture to a gesture library in the plurality of gesture libraries;

a mapping module configured to map the associated first gesture to the corresponding input of the legacy controller device, and to send the corresponding input to the legacy gaming system in response to receiving the natural user input.

9. The computing device of claim 8, wherein the mapping module includes:

a mapper configured to map the first gesture to a legacy controller state using a look up table, and

a legacy controller emulator configured to receive the legacy controller state from the mapper and to generate the corresponding input of the legacy controller device based on the legacy controller state.

10. The computing device of claim 8, wherein the depth image includes depth values for objects in a field of view of the image capture device.

11. The computing device of claim 9, wherein the algorithmic preprocessing module includes:

a background removal service configured to remove at least one background element from the depth image;

a player segmenting service in communication with the background removal service, the player segmenting service configured to segment at least one player of the legacy gaming system within the depth image;

a player mapping service configured to map a skeleton to the at least one segmented player, wherein the algorithmic preprocessing module associates a change in the skeleton with the first gesture and identifies the natural user input based on the first gesture.

12. The computing device of claim 8, wherein each of the plurality of gesture libraries corresponds to a different computer program genre.

13. The computing device of claim 12, further comprising a genre update module configured to securely update one of the plurality of gesture libraries in the gesture module.

14. A computer-readable storage device containing instructions executable by a computing device to enable a natural input device to control a legacy gaming system, the instructions being executable to perform a method comprising:

receiving a depth image from a depth camera of a natural input device distinct from a legacy controller device of the legacy gaming system, the depth image including depth values for objects in a field of view of the depth camera;

processing the depth image to identify a natural user input, the natural user input simulating a game motion that corresponds to a motion a legacy game input represents;

selecting a gesture library from a plurality of gesture libraries based on a legacy gaming program on the legacy gaming system;

associating the natural user input with a first gesture from the gesture library selected;

mapping the associated first gesture to the corresponding input of the legacy controller device;

sending the corresponding input of the legacy controller device to the legacy gaming system in response to receiving the natural user input; and

adding a second gesture to a gesture library in the plurality of gesture libraries.

15. The computer-readable storage device of claim 14, wherein mapping the associated first gesture to the corresponding input of the legacy controller device includes:

mapping the first gesture to a legacy controller state using a look up table, and

generating the corresponding input of the legacy controller via a legacy controller emulator.

16. The computer-readable storage device of claim 14, further comprising instructions executable for:

removing at least one background element from the depth image;

segmenting at least one player of the legacy gaming system within the depth image;

mapping a skeleton to the at least one segmented player;

associating a change in the skeleton with the first gesture; and

identifying the natural user input based on the first gesture.

17. The computer-readable storage device of claim 14, wherein the corresponding input of the legacy controller device is a first corresponding input, the medium further comprising instructions executable for:

mapping the second gesture to a second corresponding input of the legacy controller device; and

sending the second corresponding input to the legacy gaming system in response to receiving a natural user input associated with the second gesture.

18. The computer-readable storage device of claim 14, wherein each of the plurality of gesture libraries corresponds to a different game genre.

19. The computer-readable storage device of claim 14, further comprising instructions executable for:

operating a legacy game on the legacy gaming system; and

providing the corresponding input of the legacy controller device to the legacy game in response to receiving the natural user input.