SELECTIVE HAND OCCLUSION OVER VIRTUAL PROJECTIONS ONTO PHYSICAL SURFACES USING SKELETAL TRACKING

• US 20120249590A1
• Filed: 03/29/2012
• Published: 10/04/2012
• Est. Priority Date: 03/29/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: MACIOCCI, Giuliano, EVERITT, Andrew J., MABBUTT, Paul, BERRY, David T.
• Granted Patent: US 9,142,062 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 9/3173 wherein the projection device is specially adapted for enhanced portability

Abstract

A head mounted device provides an immersive virtual or augmented reality experience for viewing data and enabling collaboration among multiple users. Rendering images in a virtual or augmented reality system may include performing operations for capturing an image of a scene in which a virtual object is to be displayed, recognizing a body part present in the captured image, and adjusting a display of the virtual object based upon the recognized body part. The rendering operations may also include capturing an image with a body mounted camera, capturing spatial data with a body mounted sensor array, recognizing objects within the captured image, determining distances to the recognized objects within the captured image, and displaying the virtual object on a head mounted display.

Claims

1. A method of rendering images in an augmented reality system, comprising:

capturing an image of a scene in which a virtual object is to be displayed;

recognizing a body part present in the captured image; and

adjusting a display of the virtual object based upon the recognized body part.

2. The method of claim 1, wherein adjusting a display of the virtual object comprises:

identifying a portion of the virtual object that overlaps the recognized body part; and

suppressing the identified portion such that the recognized body part appears to occlude the identified portion of the virtual object.

3. The method of claim 1, wherein adjusting a display of the virtual object comprises:

identifying a portion of the virtual object that overlaps the recognized body part; and

superimposing the virtual object on the recognized body part such that the identified portion of the virtual object appears to occlude the recognized body part.

4. The method of claim 1, wherein adjusting the displayed virtual object comprises adjusting displayed portions of the virtual object overlapping the recognized body part such that the virtual object is viewable through the recognized body part and such that the recognized body part appears semitransparent.

5. The method of claim 1, further comprising:

displaying the virtual object on a body mounted semitransparent display configured such that that an individual wearing the body mounted semitransparent display can view the scene through the display.

6. The method of claim 1, wherein capturing an image of a scene in which a virtual object is to be displayed comprises capturing an image with a body mounted camera worn by a first individual, the method further comprising:

capturing spatial data with a body mounted sensor array worn by the first individual;

recognizing objects within the captured image; and

determining distances to the recognized objects within the captured image.

7. The method of claim 6, further comprising:

displaying the virtual object and the captured image on a head mounted display such that the scene is viewable to an individual wearing the body mounted semitransparent display.

8. The method of claim 1, wherein recognizing a body part present in the captured image comprises:

applying an anatomical analysis to the captured image.

9. The method of claim 8, wherein applying an anatomical analysis to the image comprises:

recognizing objects within the captured image;

comparing the recognized objects to a skeletal model stored in memory, the stored skeletal model including relative position information about key joints, bones, and limbs of a human skeleton; and

determining if any of the recognized objects match a portion of the skeletal model in response to comparing the recognized objects to the skeletal model.

10. The method of claim 8, further comprising:

determining whether the body part belongs to a first individual or to a second individual in response to recognizing a body part present in the captured image;

determining a location and encompassed area of the virtual object; and

determining whether the recognized body part overlaps at least one portion of the area encompassed by the virtual object.

11. The method of claim 10, wherein determining whether the body part belongs to a first individual or to a second individual comprises:

determining whether the body part belongs to the first or second individual based on an orientation of the recognized body part.

12. The method of claim 10, wherein determining whether the body part belongs to a first individual or to a second individual comprises determining that the body part belongs to the first or second individual based on a distance of the recognized body part from the body mounted camera.

13. The method of claim 10, wherein determining whether the body part belongs to a first individual or to a second individual comprises determining that the body part belongs to the first or second individual based on a number of body parts determined to be in the captured image.

14. The method of claim 10, wherein adjusting a display of the virtual object comprises:

displaying the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual; and

displaying the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual.

15. The method of claim 14, wherein displaying the virtual object so that the virtual object appears to be located beneath the recognized body part comprises not rendering portions of the virtual object which overlap the recognized body part.

16. The method of claim 14, further comprising:

displaying an outline of portions of the recognized body part of the second individual that overlap a display area encompassed by the virtual object such that the outline is visible over the virtual object in the display.

17. A computing device, comprising:

a memory; and

a processor coupled to the memory, wherein the processor is configured with processor-executable instructions to perform operations comprising;

capturing an image of a scene in which a virtual object is to be displayed;

recognizing a body part present in the captured image; and

adjusting a display of the virtual object based upon the recognized body part.

18. The computing device of claim 17, wherein adjusting a display of the virtual object comprises:

identifying a portion of the virtual object that overlaps the recognized body part; and

suppressing the identified portion such that the recognized body part appears to occlude the identified portion of the virtual object.

19. The computing device of claim 17, wherein the processor is configured with processor-executable instructions such that adjusting a display of the virtual object comprises:

identifying a portion of the virtual object that overlaps the recognized body part; and

superimposing the virtual object on the recognized body part such that the identified portion of the virtual object appears to occlude the recognized body part.

20. The computing device of claim 17, wherein the processor is configured with processor-executable instructions such that adjusting the displayed virtual object comprises adjusting displayed portions of the virtual object overlapping the recognized body part such that the virtual object is viewable through the recognized body part and such that the recognized body part appears semitransparent.

21. The computing device of claim 17, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

displaying the virtual object on a body mounted semitransparent display configured such that that an individual wearing the body mounted semitransparent display can view the scene through the display.

22. The computing device of claim 17, wherein the processor is configured with processor-executable instructions such that capturing an image of a scene in which a virtual object is to be displayed comprises capturing an image with a body mounted camera worn by a first individual, and wherein the processor is configured with processor-executable instructions to perform operations further comprising:

capturing spatial data with a body mounted sensor array worn by the first individual;

recognizing objects within the captured image; and

determining distances to the recognized objects within the captured image.

23. The computing device of claim 22, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

displaying the virtual object and the captured image on a head mounted display such that the scene is viewable to an individual wearing the body mounted semitransparent display.

24. The computing device of claim 17, wherein the processor is configured with processor-executable instructions such that recognizing a body part present in the captured image comprises applying an anatomical analysis to the captured image.

25. The computing device of claim 24, wherein the processor is configured with processor-executable instructions such that applying an anatomical analysis to the image comprises:

recognizing objects within the captured image;

comparing the recognized objects to a skeletal model stored in memory, the stored skeletal model including relative position information about key joints, bones, and limbs of a human skeleton; and

determining if any of the recognized objects match a portion of the skeletal model in response to comparing the recognized objects to the skeletal model.

26. The computing device of claim 25, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

determining whether the body part belongs to a first individual or to a second individual in response to recognizing a body part present in the captured image;

determining a location and encompassed area of the virtual object; and

determining whether the recognized body part overlaps at least one portion of the area encompassed by the virtual object.

27. The computing device of claim 26, wherein the processor is configured with processor-executable instructions such that determining whether the body part belongs to a first individual or to a second individual comprises:

determining whether the body part belongs to the first or second individual based on an orientation of the recognized body part.

28. The computing device of claim 26, wherein the processor is configured with processor-executable instructions such that determining whether the body part belongs to a first individual or to a second individual comprises determining that the body part belongs to the first or second individual based on a distance of the recognized body part from the body mounted camera.

29. The computing device of claim 26, wherein the processor is configured with processor-executable instructions such that determining whether the body part belongs to a first individual or to a second individual comprises determining that the body part belongs to the first or second individual based on a number of body parts determined to be in the captured image.

30. The computing device of claim 26, wherein the processor is configured with processor-executable instructions such that adjusting a display of the virtual object comprises:

displaying the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual; and

displaying the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual.

31. The computing device of claim 30, wherein the processor is configured with processor-executable instructions such that displaying the virtual object so that the virtual object appears to be located beneath the recognized body part comprises not rendering portions of the virtual object which overlap the recognized body part.

32. The computing device of claim 30, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

displaying an outline of portions of the recognized body part of the second individual that overlap a display area encompassed by the virtual object such that the outline is visible over the virtual object in the display.

33. A computing device, comprising:

means for capturing an image of a scene in which a virtual object is to be displayed;

means for recognizing a body part present in the captured image; and

means for adjusting a display of the virtual object based upon the recognized body part.

34. The computing device of claim 33, wherein means for adjusting a display of the virtual object comprises:

means for identifying a portion of the virtual object that overlaps the recognized body part; and

means for suppressing the identified portion such that the recognized body part appears to occlude the identified portion of the virtual object.

35. The computing device of claim 33, wherein means for adjusting a display of the virtual object comprises:

means for identifying a portion of the virtual object that overlaps the recognized body part; and

means for superimposing the virtual object on the recognized body part such that the identified portion of the virtual object appears to occlude the recognized body part.

36. The computing device of claim 33, wherein means for adjusting the displayed virtual object comprises means for adjusting displayed portions of the virtual object overlapping the recognized body part such that the virtual object is viewable through the recognized body part and such that the recognized body part appears semitransparent.

37. The computing device of claim 33, further comprising:

means for displaying the virtual object on a body mounted semitransparent display configured such that that an individual wearing the body mounted semitransparent display can view the scene through the display.

38. The computing device of claim 33, wherein means for capturing an image of a scene in which a virtual object is to be displayed comprises means for capturing an image with a body mounted camera worn by a first individual, the computing device further comprising:

means for capturing spatial data with a body mounted sensor array worn by the first individual;

means for recognizing objects within the captured image; and

means for determining distances to the recognized objects within the captured image.

39. The computing device of claim 38, further comprising:

means for displaying the virtual object and the captured image on a head mounted display such that the scene is viewable to an individual wearing the body mounted semitransparent display.

40. The computing device of claim 33, wherein means for recognizing a body part present in the captured image comprises means for applying an anatomical analysis to the captured image.

41. The computing device of claim 40, wherein means for applying an anatomical analysis to the image comprises:

means for recognizing objects within the captured image;

means for comparing the recognized objects to a skeletal model stored in memory, the stored skeletal model including relative position information about key joints, bones, and limbs of a human skeleton; and

means for determining if any of the recognized objects match a portion of the skeletal model in response to comparing the recognized objects to the skeletal model.

42. The computing device of claim 40, further comprising:

means for determining whether the body part belongs to a first individual or to a second individual in response to recognizing a body part present in the captured image;

means for determining a location and encompassed area of the virtual object; and

means for determining whether the recognized body part overlaps at least one portion of the area encompassed by the virtual object.

43. The computing device of claim 42, wherein means for determining whether the body part belongs to a first individual or to a second individual comprises:

means for determining whether the body part belongs to the first or second individual based on an orientation of the recognized body part.

44. The computing device of claim 42, wherein means for determining whether the body part belongs to a first individual or to a second individual comprises means for determining that the body part belongs to the first or second individual based on a distance of the recognized body part from the body mounted camera.

45. The computing device of claim 42, wherein means for determining whether the body part belongs to a first individual or to a second individual comprises means for determining that the body part belongs to the first or second individual based on a number of body parts determined to be in the captured image.

46. The computing device of claim 42, wherein means for adjusting a display of the virtual object comprises:

means for displaying the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual; and

means for displaying the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual.

47. The computing device of claim 46, wherein means for displaying the virtual object so that the virtual object appears to be located beneath the recognized body part comprises means for not rendering portions of the virtual object which overlap the recognized body part.

48. The computing device of claim 46, further comprising:

means for displaying an outline of portions of the recognized body part of the second individual that overlap a display area encompassed by the virtual object such that the outline is visible over the virtual object in the display.

49. A non-transitory computer readable storage medium having stored thereon processor-executable software instructions configured to cause a processor to perform operations for rendering images in an augmented reality system, the operations comprising:

capturing an image of a scene in which a virtual object is to be displayed;

recognizing a body part present in the captured image; and

adjusting a display of the virtual object based upon the recognized body part.

50. The non-transitory computer readable storage medium of claim 49, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that adjusting a display of the virtual object comprises:

identifying a portion of the virtual object that overlaps the recognized body part; and

suppressing the identified portion such that the recognized body part appears to occlude the identified portion of the virtual object.

51. The non-transitory computer readable storage medium of claim 49, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that adjusting a display of the virtual object comprises:

identifying a portion of the virtual object that overlaps the recognized body part; and

superimposing the virtual object on the recognized body part such that the identified portion of the virtual object appears to occlude the recognized body part.

52. The non-transitory computer readable storage medium of claim 49, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that adjusting the displayed virtual object comprises adjusting displayed portions of the virtual object overlapping the recognized body part such that the virtual object is viewable through the recognized body part and such that the recognized body part appears semitransparent.

53. The non-transitory computer readable storage medium of claim 49, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

displaying the virtual object on a body mounted semitransparent display configured such that that an individual wearing the body mounted semitransparent display can view the scene through the display.

54. The non-transitory computer readable storage medium of claim 49, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that capturing an image of a scene in which a virtual object is to be displayed comprises capturing an image with a body mounted camera worn by a first individual, and wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

capturing spatial data with a body mounted sensor array worn by the first individual;

recognizing objects within the captured image; and

determining distances to the recognized objects within the captured image.

55. The non-transitory computer readable storage medium of claim 54, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

displaying the virtual object and the captured image on a head mounted display such that the scene is viewable to an individual wearing the body mounted semitransparent display.

56. The non-transitory computer readable storage medium of claim 49, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that recognizing a body part present in the captured image comprises applying an anatomical analysis to the captured image.

57. The non-transitory computer readable storage medium of claim 56, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that applying an anatomical analysis to the image comprises:

recognizing objects within the captured image;

comparing the recognized objects to a skeletal model stored in memory, the stored skeletal model including relative position information about key joints, bones, and limbs of a human skeleton; and

determining if any of the recognized objects match a portion of the skeletal model in response to comparing the recognized objects to the skeletal model.

58. The non-transitory computer readable storage medium of claim 56, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

determining whether the body part belongs to a first individual or to a second individual in response to recognizing a body part present in the captured image;

determining a location and encompassed area of the virtual object; and

determining whether the recognized body part overlaps at least one portion of the area encompassed by the virtual object.

59. The non-transitory computer readable storage medium of claim 58, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that determining whether the body part belongs to a first individual or to a second individual comprises:

determining whether the body part belongs to the first or second individual based on an orientation of the recognized body part.

60. The non-transitory computer readable storage medium of claim 58, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that determining whether the body part belongs to a first individual or to a second individual comprises determining that the body part belongs to the first or second individual based on a distance of the recognized body part from the body mounted camera.

61. The non-transitory computer readable storage medium of claim 58, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that determining whether the body part belongs to a first individual or to a second individual comprises determining that the body part belongs to the first or second individual based on a number of body parts determined to be in the captured image.

62. The non-transitory computer readable storage medium of claim 58, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that adjusting a display of the virtual object comprises:

displaying the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual; and

displaying the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual.

63. The non-transitory computer readable storage medium of claim 62, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that displaying the virtual object so that the virtual object appears to be located beneath the recognized body part comprises not rendering portions of the virtual object which overlap the recognized body part.

64. The non-transitory computer readable storage medium of claim 62, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations comprising:

displaying an outline of portions of the recognized body part of the second individual that overlap a display area encompassed by the virtual object such that the outline is visible over the virtual object in the display.