Manipulation of virtual objects using enhanced interactive system

• US 8,559,676 B2
• Filed: 12/27/2007
• Issued: 10/15/2013
• Est. Priority Date: 12/29/2006
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US11/965,332
• Publication Number:
  US 20080166022A1
• Time in Patent Office: 2,119 Days
• Field of Search: 382/106, 382/107, 382/110, 345/156, 345/426, 345/423
• US Class Current: 382/107
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The detection of motion of a user via a camera and the generation of a dynamic virtual representation of a user on a display, where the user'"'"'s detected motion causes the dynamic virtual representation to interact with virtual objects on the display. The magnitude and direction of the user'"'"'s detected motion is calculated to determine the magnitude and direction of a force applied by the dynamic virtual representation to the virtual object. Further arrangements include water or smoke fluid simulations, in order to enhance the user experience.

Claims

1. A computer-implemented method comprising:

receiving at least a first image of an object;

detecting the object in the first image;

generating an object data map based on detecting the object in the first image, the object data map indicating, for each pixel in the first image, whether the object is detected at the each pixel;

generating a motion history indicating, for each pixel included in the object data map, an indication of time since the object has been detected at the each pixel; and

controlling an application based on the motion history,

wherein controlling an application based on the motion history includes;

determining whether the object is interacting with a virtual object rendered in a display image associated with the application, based on at least a part of the motion history within a region representing the virtual object;

calculating a velocity for the virtual object based on the motion history; and

controlling the application based on the determined velocity for the virtual object;

wherein the object in the first image and the virtual object rendered in the display image are distinct from each other.

2. The method of claim 1 wherein generating a motion history indicating, for each pixel included in the object data map, an indication of time since the object has been detected at the each pixel includes generating a motion history indicating, for each pixel included in the object data map, an opacity value corresponding to the time since the object has been detected at the each pixel.

3. The method of claim 1 wherein generating a motion history indicating, for each pixel included in the object data map, an indication of time since the object has been detected at the each pixel includes generating a motion history indicating, for each pixel included in the object data map, a number of frames since the object has been detected at the each pixel.

4. The method of claim 1 wherein generating a motion history indicating, for each pixel included in the object data map, an indication of time since the object has been detected at the each pixel includes generating a motion history indicating, for each pixel included in the object data map, a measurement of time since the object has been detected at the each pixel.

5. The method of claim 1 wherein:

generating a motion history indicating, for each pixel included in the object data map, an indication of time since the object has been detected at the each pixel includes filtering the motion history; and

controlling an application based on the motion history includes controlling an application based on the filtered motion history.

6. The method of claim 5 wherein filtering the motion history includes, for each pixel, computing a value for the each pixel based on a value of the each pixel and values of pixels neighboring the each pixel.

7. The method of claim 1 wherein receiving at least a first image of an object includes receiving at least a first image of a user.

8. The method of claim 1 wherein controlling an application based on the motion history includes displaying a representation of the object in the display image associated with the application.

9. The method of claim 8 wherein controlling an application based on the motion history includes controlling movement of the representation of the object in the display image associated with the application.

10. The method of claim 8 wherein displaying a representation of the object in the display image associated with the application includes displaying the representation of the object in the display image associated with the application as one or more particles of a particle system.

11. The method of claim 8 wherein displaying a representation of the object in the display image associated with the application includes:

generating an outline contour representing motion of the object based on the motion history; and

determining whether the representation of the object in the display image touches the virtual object based on the generated outline contour,

wherein the virtual object is rendered in the display image by the application.

12. The method of claim 8 wherein displaying a representation of the object in the display image associated with the application includes displaying a representation of the object such that pixels corresponding to pixels in the motion history at which the indication of time indicates the object has been most recently detected are displayed with an opacity greater than pixels corresponding to pixels in the motion history at which the indication of time indicates the object has been detected further in the past.

13. The method of claim 12 wherein displaying a representation of the object such that pixels corresponding to pixels in the motion history at which the indication of time indicates the object has been most recently detected are displayed with an opacity greater than pixels corresponding to pixels in the motion history at which the indication of time indicates the object has been detected further in the past includes displaying a representation of the object such that pixels corresponding to pixels in the motion history are displayed with an opacity based on the indication of time associated with the pixel.

14. The method of claim 13 wherein displaying a representation of the object such that pixels corresponding to pixels in the motion history are displayed with an opacity based on the indication of time associated with the pixel includes displaying a representation of the object such that a first pixel associated with an indication of time that indicates that the object has been detected more recently at the first pixel than a second pixel appears with an opacity that is greater than an opacity of the second pixel.

15. The method of claim 8 wherein displaying a representation of the object in the display image associated with the application includes:

generating an alpha channel based on the motion history; and

displaying a representation of the object in the display image based on the alpha channel.

16. The method of claim 8 wherein displaying a representation of the object in the display image associated with the application includes displaying an animated effect image based on an animation or a particle system.

17. The method of claim 8 wherein displaying a representation of the object in the display image associated with the application includes displaying a keyed image of the object.

18. The method of claim 8 wherein displaying a representation of the object in the display image associated with the application includes:

controlling the application based on determination results, wherein the virtual object is rendered in the display image by the application.

19. The method of claim 1 wherein:

calculating a velocity for the virtual object includes calculating a velocity and a direction of motion for the virtual object based on optical flow data associated with the motion history; and

controlling the application based on the determined velocity for the virtual object includes controlling the application based on the determined velocity and the determined direction of motion for the virtual object.

20. The method of claim 18, wherein the virtual object comprises a particle of a particle system.

21. The method of claim 1 wherein detecting the object in the first image includes comparing the first image with a background image.

26. The method of claim 1, wherein controlling an application based on the motion history further includes:

determining a first centroid of a part of the object within a region representing the virtual object having a frame count or time value more recent than a first predetermined threshold;

determining a second centroid of the part of the object within the region having a frame count or time value more recent than a second predetermined threshold, wherein the second threshold includes a longer span of time than the first threshold;

using the relative position of the first centroid to the second centroid as a representative motion of the part; and

using the representative motion of the part to determine a magnitude and direction that is applied to the virtual object.

27. The method of claim 1, wherein determining whether the object is interacting with the virtual object includes determining whether the object is touching the virtual object, based on identifying a part of the object as being coincident to the virtual object.

28. The method of claim 1, further comprising identifying the region representing the virtual object.

29. The method of claim 28, wherein the region represents a boundary shape of the virtual object.

30. The method of claim 28, wherein the region is larger than the virtual object.

31. The method of claim 1, wherein detecting the object in the first image includes classifying the part of the motion history as a part of the object.

32. The method of claim 1, wherein determining whether the object is interacting with the virtual object includes calculating an average motion of the part of the motion history.

33. The method of claim 1, wherein determining whether the object is interacting with the virtual object includes:

calculating a centroid of the part of the motion history; and

comparing the centroid of the part of the motion history with a centroid of the region.

34. The method of claim 1, wherein the first image comprises an image captured by a camera.

35. The method of claim 34, wherein the object comprises a portion of a user or an object manipulated by the user that is within a field of view of the camera.

22. A computer-implemented method comprising controlling an application based upon a motion history that indicates, for each pixel included in an object data map, an indication of time since an object has been detected at each pixel, the object data map indicating, for each pixel in a first image received of the object, whether the object is detected at the each pixel, and further based upon determining whether the object touches a virtual object rendered in a display image associated with the application, wherein determining whether the object touches a virtual object rendered in the display image associated with the application is based on identifying that a part of the motion history within a region representing the virtual object is coincident to the virtual object, calculating a velocity for the virtual object based on the motion history;

and controlling the application based on the determined velocity for the virtual wherein the object in the first image and the virtual object rendered in the display image are distinct from each other.

23. An apparatus comprising:

a camera configured to capture at least a first image of an object;

a processor;

memory storing instructions that, when executed by the processor, cause the apparatus to;

receive the first image of the object from the camera,

detect the object in the first image,

generate an object data map in response to detecting the object in the first image, the object data map indicating, for each pixel in the first image, whether the object is detected at the each pixel,

generate a motion history indicating, for each pixel included in the object data map, an indication of time since the object has been detected at the each pixel, and

control an application based on the motion history,

wherein controlling an application based on the motion history includes;

determining whether the object is interacting with a virtual object rendered in a display image associated with the application, based on at least a part of the motion history within a region representing the virtual object;

calculating a velocity for the virtual object based on the motion history; and

controlling the application based on the determined velocity for the virtual object;

wherein the object in the first image and the virtual object rendered in the display image are distinct from each other; and

a display configured to render the display image associated with the application.

24. A non-transitory computer readable storage medium having stored thereon instructions that, when executed, cause at least one computing device to:

receive at least a first image of an object;

detect the object in the first image;

generate an object data map in response to detecting the object in the first image, the object data map indicating, for each pixel in the first image, whether the object is detected at the each pixel;

generate a motion history indicating, for each pixel included in the object data map, an indication of time since the object has been detected at the each pixel; and

control an application based on the motion history,

wherein controlling an application based on the motion history includes;

determining whether the object is interacting with a virtual object rendered in a display image associated with the application, based on at least a part of the motion history within a region representing the virtual object;

calculating a velocity for the virtual object based on the motion history; and

controlling the application based on the determined velocity for the virtual object;

wherein the object in the first image and the virtual object rendered in the display image are distinct from each other.

25. A system comprising:

means for receiving at least a first image of an object;

means for detecting the object in the first image;

means for generating an object data map in response to detecting the object in the first image, the object data map indicating, for each pixel in the first image, whether the object is detected at the each pixel;

means for generating a motion history indicating, for each pixel included in the object data map, an indication of time since the object has been detected at the each pixel; and

means for controlling an application based on the motion history,

wherein the means for controlling an application based on the motion history includes;

means for determining whether the object is interacting with a virtual object rendered in a display image associated with the application, based on at least a part of the motion history within a region representing the virtual object;

means for calculating a velocity for the virtual object based on the motion history; and

means for controlling the application based on the determined velocity for the virtual object;

wherein the object in the first image and the virtual object rendered in the display image are distinct from each other.