RENDERING AUGMENTED REALITY BASED ON FOREGROUND OBJECT

• US 20140184644A1
• Filed: 01/03/2013
• Published: 07/03/2014
• Est. Priority Date: 01/03/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Sharma, Piyush, Rezaiifar, Ramin
• Granted Patent: US 9,076,257 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A mobile device detects a moveable foreground object in captured images, e.g., a series of video frames without depth information. The object may be one or more of the user'"'"'s fingers. The object may be detected by warping one of a captured image of a scene that includes the object and a reference image of the scene without the object so they have the same view and comparing the captured image and the reference image after warping. A mask may be used to segment the object from the captured image. Pixels are detected in the extracted image of the object and the pixels are used to detect the point of interest on the foreground object. The object may then be tracked in subsequent images. Augmentations may be rendered and interacted with or temporal gestures may be detected and desired actions performed accordingly.

Claims

1. A method comprising:

capturing an image of a scene with a foreground object that is not attached to the scene, the foreground object including a point of interest that is a distinct physical aspect;

warping at least one of the image and a reference image of the scene that does not include the foreground object so the image and the reference image have a same view;

comparing the image to the reference image after warping to detect pixels that belong to the point of interest on the foreground object;

detecting the point of interest on the foreground object using the detected pixels;

displaying the image on a display; and

rendering an augmentation on the display over the image based on the point of interest.

2. The method of claim 1, wherein the foreground object that is not attached to the scene is at least one finger of a user and the point of interest is a tip of the finger.

3. The method of claim 1, detecting the point of interest on the foreground object is performed without pre-determined geometric constraints on the foreground object.

4. The method of claim 1, wherein comparing the image to the reference image after warping to detect pixels that belong to the point of interest on the foreground object comprises:

generating a mask for the foreground object;

segmenting the foreground object from the image using the mask; and

detecting the pixels using the foreground object segmented from the image.

5. The method of claim 1, wherein comparing the image to the reference image after warping to detect pixels that belong to the point of interest on the foreground object comprises:

generating a foreground object image using pixels in the image that are different than corresponding pixels in the reference image; and

detecting the pixels that belong to the point of interest on the foreground object in the foreground object image.

6. The method of claim 1, wherein comparing the image and the reference image after warping comprises:

subtracting pixels in the image from corresponding pixels in the reference image to generate a difference for each pixel after warping; and

comparing the difference for each pixel to a threshold.

7. The method of claim 1, wherein comparing the image and the reference image after warping comprises:

generating ratios for corresponding pixels in the image and the reference image after warping; and

comparing the ratios for corresponding pixels to a threshold.

8. The method of claim 1, wherein warping at least one of the image and the reference image comprises:

generating a pose between the image and the reference image; and

warping one of the image and the reference image based on the pose.

9. The method of claim 1, further comprising:

displaying subsequently captured images on the display; and

altering the augmentation based on the point of interest in the subsequently captured images.

10. The method of claim 1, wherein rendering the augmentation on the display comprises rendering the augmentation to appear as if the augmentation is underneath the foreground object displayed on the display.

11. The method of claim 1, wherein the scene includes multiple foreground objects with points of interest, the method further comprising:

detecting pixels that belong to the points of interest on the multiple foreground objects by comparing the image to the reference image after warping; and

detecting the points of interest on the multiple foreground objects using the detected pixels.

12. The method of claim 1, further comprising:

tracking the point of interest on the foreground object in subsequently captured images;

detecting a temporal gesture based on movement of the point of interest on the foreground object; and

performing an action associated with the temporal gesture.

13. The method of claim 1, further comprising:

comparing a configuration of the point of interest on the foreground object to a library of gesture configurations;

identifying a gesture from the configuration of the point of interest on the foreground object; and

performing an action associated with the gesture.

14. The method of claim 1, wherein the image is captured using a camera with no depth information.

15. The method of claim 1, further comprising tracking the foreground object in subsequently captured images comprises:

for each subsequently captured image, warping at least one of the subsequently captured image and the reference image of the scene;

comparing the subsequently captured image to the reference image after warping to detect pixels that belong to the point of interest on the foreground object; and

detecting the point of interest on the foreground object using the detected pixels in the subsequently captured image.

16. The method of claim 1, further comprising tracking the foreground object in subsequently captured images using optical flow with the pixels that belong to the point of interest.

17. An apparatus comprising:

a camera;

a display; and

a processor coupled to the display and coupled to the camera to receive an image of a scene with a foreground object that is not attached to the scene, the foreground object including a point of interest that is a distinct physical aspect, the processor configured to warp at least one of the image and a reference image of the scene that does not include the foreground object so the image and the reference image have a same view, compare the image to the reference image after warping to detect pixels that belong to the point of interest on the foreground object, detect the point of interest on the foreground object using the detected pixels, display the image on the display, and render an augmentation on the display over the image based on the point of interest.

18. The apparatus of claim 17, wherein the foreground object that is not attached to the scene is at least one finger of a user and the point of interest is a tip of the finger.

19. The apparatus of claim 17, wherein the processor is configured to detect the point of interest on the foreground object without pre-determined geometric constraints on the foreground object.

20. The apparatus of claim 17, wherein the processor is configured to compare the reference image after warping to detect pixels that belong to the point of interest on the foreground object by being configured to:

generate a mask for the foreground object;

segment the foreground object from the image using the mask; and

detect the pixels using the foreground object segmented from the image.

21. The apparatus of claim 17, wherein the processor is configured to compare the reference image after warping to detect pixels that belong to the point of interest on the foreground object by being configured to:

generate a foreground object image using pixels in the image that are different than corresponding pixels in the reference image; and

detect the pixels that belong to the point of interest on the foreground object in the foreground object image.

22. The apparatus of claim 17, wherein the processor is configured to compare the image and the reference image after warping by being configured to subtract pixels in the image from corresponding pixels in the reference image to generate a difference for each pixel after warping;

and compare the difference for each pixel to a threshold.

23. The apparatus of claim 17, wherein the processor is configured to compare the image and the reference image after warping by being configured to generate ratios for corresponding pixels in the image and the reference image after warping, and compare the ratios for corresponding pixels to a threshold.

24. The apparatus of claim 17, wherein the processor is configured to warp the at least one of the image and the reference image by being configured to generate a pose between the image and the reference image, and warp one of the image and the reference image based on the pose.

25. The apparatus of claim 17, wherein the processor is further configured to cause subsequently captured images to be displayed on the display, and alter the augmentation based on based on the point of interest in the subsequently captured images.

26. The apparatus of claim 17, wherein the processor is configured to render the augmentation on the display by being configured to render the augmentation to appear as if the augmentation is underneath the foreground object displayed on the display.

27. The apparatus of claim 17, wherein the scene includes multiple foreground objects with points of interest, wherein the processor is further configured to:

detect pixels that belong to the points of interest on the multiple foreground objects by comparing the image to the reference image after warping; and

detect the points of interest on the multiple foreground objects using the detected pixels.

28. The apparatus of claim 17, wherein the processor is further configured to:

track the point of interest on the foreground object in subsequently captured images;

detect a temporal gesture based on movement of the point of interest on the foreground object; and

perform an action associated with the temporal gesture.

29. The apparatus of claim 17, wherein the processor is further configured to:

compare a configuration of the point of interest on the foreground object to a library of gesture configurations;

identify a gesture from the configuration of the point of interest on the foreground object; and

perform an action associated with the gesture.

30. The apparatus of claim 17, wherein the camera provides no depth information.

31. The apparatus of claim 17, wherein the processor is configured to track the foreground object in subsequently captured images by being configured to:

for each subsequently captured image, warp at least one of the subsequently captured image and the reference image of the scene;

compare the subsequently captured image to the reference image after warping to detect pixels that belong to the point of interest on the foreground object; and

detect the point of interest on the foreground object using the detected pixels in the subsequently captured image.

32. The apparatus of claim 17, wherein the processor is configured to track the foreground object in subsequently captured images by being configured use optical flow with the pixels that belong to the point of interest.

33. An apparatus comprising:

means for capturing an image of a scene with a foreground object that is not attached to the scene, the foreground object including a point of interest that is a distinct physical aspect;

means for warping at least one of the image and a reference image of the scene that does not include the foreground object so the image and the reference image have a same view;

means for comparing the image to the reference image after warping to detect pixels that belong to the point of interest on the foreground object;

means for detecting the point of interest on the foreground object using the detected pixels;

means for displaying the image on a display; and

means for rendering an augmentation on the display over the image based on the point of interest.

34. The apparatus of claim 33, wherein comparing the image to the reference image after warping to detect pixels that belong to the point of interest on the foreground object generates a mask for the foreground object, the apparatus further comprising a means for segmenting the foreground object from the image using the mask.

35. The apparatus of claim 33, the apparatus further comprising a means for extracting the foreground object from the image.

36. The apparatus of claim 33, wherein the means for warping at least one of the image and the reference image comprises:

means for generating a pose between the image and the reference image; and

means for warping one of the image and the reference image based on the pose.

37. The apparatus of claim 33, wherein the means for comparing the image and the reference image after warping subtracts pixels in the image from corresponding pixels in the reference image to generate a difference for each pixel after warping;

and compares the difference for each pixel to a threshold.

38. The apparatus of claim 33, wherein the comparing the image and the reference image after warping generates ratios for corresponding pixels in the image and the reference image after warping;

and compares the ratios for corresponding pixels to a threshold.

39. The apparatus of claim 33, further comprising:

means for displaying subsequently captured images on the display; and

means for altering the augmentation based on the point of interest in the subsequently captured images.

40. The apparatus of claim 33, further comprising:

means for tracking the point of interest on the foreground object in subsequently captured images;

means for detecting a temporal gesture based on movement of the point of interest on the foreground object; and

means for performing an action associated with the temporal gesture.

41. A storage medium including program code stored thereon, comprising:

program code to receive an image of a scene with a foreground object that is not attached to the scene, the foreground object including a point of interest that is a distinct physical aspect;

program code to warp at least one of the image and a reference image of the scene that does not include the foreground object so the image and the reference image have a same view;

program code to compare the image to the reference image after warping to detect pixels that belong to the point of interest on the foreground object;

program code to detect the point of interest on the foreground object using the detected pixels;

program code to display the image on a display; and

program code to render an augmentation on the display over the image based on the point of interest.

42. The storage medium of claim 41, further comprising program code to extract the foreground object from the image.

43. The storage medium of claim 41, wherein the program code to warp at least one of the image and the reference image comprises:

program code to generate a pose between the image and the reference image; and

program code to warp one of the image and the reference image based on the pose.

44. The storage medium of claim 41, wherein the program code to compare the image and the reference image after warping comprises:

program code to subtract pixels in the image from corresponding pixels in the reference image to generate a difference for each pixel after warping; and

program code to compare the difference for each pixel to a threshold.

45. The storage medium of claim 41, wherein the program code to comparing the image and the reference image after warping comprises:

program code to generate ratios for corresponding pixels in the image and the reference image after warping; and

program code to compare the ratios for corresponding pixels to a threshold.

46. The storage medium of claim 41, further comprising:

program code to display subsequently captured images; and

program code to alter the augmentation based on the point of interest in the subsequently captured images.

47. The storage medium of claim 41, further comprising:

program code to track the point of interest on the foreground object in subsequently captured images;

program code to detect a temporal gesture based on movement of the point of interest on the foreground object; and

program code to perform an action associated with the temporal gesture.