Motion-based tracking

• US 8,144,118 B2
• Filed: 01/23/2006
• Issued: 03/27/2012
• Est. Priority Date: 01/21/2005
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan
• Primary Examiner(s): CUTLER, ALBERT H
• Application Number:
  US11/337,090
• Publication Number:
  US 20060192782A1
• Time in Patent Office: 2,255 Days
• Field of Search: 463/2, 463/3, 463/4, 463/5
• US Class Current: 345/156
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 2300/1018 Calibration; Key and button assignment
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

In one implementation, a first captured image is accessed. The first captured image includes (1) a first display produced at a first point in time, and (2) a user interacting with the first display and not part of the first display. A second captured image is accessed. The second captured image includes (1) a second display produced at a second point in time, and (2) the user interacting with the second display and not part of the second display. The first captured image and the second captured image are compared. The motion of the user is determined based on a result of the comparing of the first captured image and the second captured image. The determined motion of the user is related to a portion of one or more of the first and second captured images.

Claims

1. A computer-implemented method comprising:

accessing a first captured image that includes (1) one or more projected objects in a first display produced at a first point in time by a set of instructions executed by a computing device, and (2) a user interacting with the first display and not part of the first display;

accessing a second captured image that includes (1) one or more of the projected objects in a second display produced at a second point in time by the set of instructions executed by the computing device, and (2) the user interacting with the second display and not part of the second display;

determining a display image motion map based on the first display and the second display, the display image motion map describing motion of the one or more projected objects;

determining a captured image motion map based on the first captured image and the second captured image, the captured image motion map describing motion of both the user and the one or more projected objects;

comparing the display image motion map and the captured image motion map to segment motion of the user from motion of the one or more projected objects;

determining motion of the user based on a result of the comparing of the display image motion map and the captured image motion map; and

relating the determined motion of the user to a portion of one or more of the first and second captured images.

2. The method of claim 1 further comprising providing an indication of the determined motion of the user and the portion to the computing device.

3. The method of claim 1 further comprising determining an input for the set of instructions based on the determined motion and the portion.

4. The method of claim 1 wherein comparing the display image motion map and the captured image motion map comprises performing an absolute difference operation on the display image motion map and the captured image motion map.

5. The method of claim 1 wherein comparing the display image motion map and the captured image motion map comprises performing an optical flow operation on the display image motion map and the captured image motion map.

6. The method of claim 1 wherein comparing the display image motion map and the captured image motion map comprises producing the result, and the result is an image indicating motion of the user and motion associated with one or more of the projected objects in the first and second displays.

7. The method of claim 6 further comprising performing a geometric transform operation on the result.

8. The method of claim 6 wherein determining motion of the user, based on the result of the comparing of the display image motion map and the captured image motion map comprises isolating, in the result, the motion of the user from the motion associated with one or more of the projected objects in the first and second displays.

9. A system comprising:

a captured image motion detection module configured to;

access a first captured image that includes (1) one or more projected objects in a first display produced at a first point in time by a set of instructions executed by a computing device, and (2) a user interacting with the first display and not part of the first display;

access a second captured image that includes (1) one or more of the projected objects in a second display produced at a second point in time by the set of instructions executed by the computing device, and (2) the user interacting with the second display and not part of the second display; and

determine a captured image motion map based on the first captured image and the second captured image, the captured image motion map describing motion of both the user and the one or more projected objects;

a display image motion detection module configured to determine a display image motion map based on the first display and the second display, the display image motion map describing motion of the one or more projected objects; and

a comparison module configured to;

compare the display image motion map and the captured image motion map to segment motion of the user from motion of the one or more projected objects;

determine motion of the user based on a result of the comparing of the display image motion map and the captured image motion map; and

relate the determined motion of the user to a portion of one or more of the first and second captured images.

10. The system of claim 9 further comprising:

a camera configured (1) to capture the first captured image and provide the first captured image to the captured image motion detection module, and (2) to capture the second captured image and provide the second captured image to the captured image motion detection module;

a display to display images; and

a computing device configured to execute a set of instructions to produce the first and second displays.

11. The system of claim 9 further comprising a latency compensation module configured to perform a latency compensation operation on the display image motion map.

12. The system of claim 9 further comprising a geometric transform module configured to perform a geometric transform operation on the captured image motion map.

13. An apparatus comprising a non-transitory computer-readable medium including instructions that when executed by a processing device result in at least the following:

accessing a first captured image that includes (1) one or more projected objects in a first display produced at a first point in time by a set of instructions executed by a computing device, and (2) a user interacting with the first display and not part of the first display;

accessing a second captured image that includes (1) one or more of the projected objects in a second display produced at a second point in time by the set of instructions executed by the computing device, and (2) the user interacting with the second display and not part of the second display;

determining a display image motion map based on the first display and the second display, the display image motion map describing motion of the one or more projected objects;

determining a captured image motion map based on the first captured image and the second captured image, the captured image motion map describing motion of both the user and the one or more projected objects;

comparing the display image motion map and the captured image motion map to segment motion of the user from motion of the one or more projected objects;

determining motion of the user based on a result of the comparing of the display image motion map and the captured image motion map; and

relating the determined motion of the user to a portion of one or more of the first and second captured images.

14. A computer-implemented method comprising:

accessing display images that are generated by an application for display on a display screen and that include one or more objects moving in the display images;

determining a display image motion map based on the display images, the display image motion map describing motion of the one or more objects within the display images;

accessing captured images of a user interacting with the display images when displayed on the display screen;

determining a captured image motion map based on the captured images, the captured image motion map describing motion of both the user and the one or more objects within the captured images;

comparing the display image motion map and the captured image motion map to segment motion of the user within the captured images from motion of the one or more objects within the captured images;

determining a user motion map based on the comparison of the display image motion map and the captured image motion map, the user motion map describing motion of the user within the captured images; and

using the user motion map to control the application.

15. The method of claim 14:

wherein the one or more obiects comprise one or more projected obiects,

wherein accessing display images that are generated by the application for display on the display screen and that include one or more objects moving in the display images comprises accessing display images that are generated by a application for display on a display screen and that include the one or more projected objects moving in the display images;

wherein determining the display image motion map based on the display images comprises determining a display image motion map based on the display images, the display image motion map describing motion of the one or more projected objects within the display images;

wherein determining the captured image motion map based on the captured images comprises determining a captured image motion map based on the captured images, the captured image motion map describing motion of both the user and the one or more projected objects within the captured images; and

wherein comparing the display image motion map and the captured image motion map to segment motion of the user within the captured images from motion of the one or more objects within the captured images comprises comparing the display image motion map and the captured image motion map to segment motion of the user within the captured images from motion of the one or more projected objects within the captured images.

16. The method of claim 14, further comprising performing a latency compensation operation on the display image motion map.

17. The method of claim 14, further comprising performing a geometric transform operation on the captured image motion map.

18. The method of claim 14, further comprising:

performing a latency compensation operation on the display image motion map to produce a compensated motion map; and

performing a geometric transform operation on the captured image motion map to produce a transformed motion map,

wherein comparing the display image motion map and the captured image motion map to segment motion of the user within the captured images from motion of the one or more objects within the captured images comprises comparing the compensated motion map and the transformed motion map to segment motion of the user within the captured images from motion of the one or more objects within the captured images.

19. The method of claim 14 wherein determining the user motion map based on the comparison of the display image motion map and the captured image motion map comprises isolating motion of the user from motion associated with the one or more objects within the captured images such that the user motion map only describes motion of the user within the captured images without describing motion of any of the one or more objects within the captured images.

20. An apparatus comprising a non-transitory computer-readable medium including instructions that when executed by a processing device result in at least the following:

accessing display images that are generated by an application for display on a display screen and that include one or more objects moving in the display images;

determining a display image motion map based on the display images, the display image motion map describing motion of the one or more objects within the display images;

accessing captured images of a user interacting with the display images when displayed on the display screen;

determining a captured image motion map based on the captured images, the captured image motion map describing motion of both the user and the one or more objects within the captured images;

comparing the display image motion map and the captured image motion map to segment motion of the user within the captured images from motion of the one or more objects within the captured images;

determining a user motion map based on the comparison of the display image motion map and the captured image motion map, the user motion map describing motion of the user within the captured images; and

using the user motion map to control the application.