Selective hand occlusion over virtual projections onto physical surfaces using skeletal tracking

• US 9,142,062 B2
• Filed: 03/29/2012
• Issued: 09/22/2015
• Est. Priority Date: 03/29/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Maciocci, Giuliano, Everitt, Andrew J., Mabbutt, Paul, Berry, David T.
• Primary Examiner(s): McDowell, Jr., Maurice L
• Assistant Examiner(s): Ricks, Donna J
• Application Number:
  US13/434,488
• Publication Number:
  US 20120249590A1
• Time in Patent Office: 1,272 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 9/3173 wherein the projection device is specially adapted for enhanced portability

Abstract

A head mounted device provides an immersive virtual or augmented reality experience for viewing data and enabling collaboration among multiple users. Rendering images in a virtual or augmented reality system may include performing operations for capturing an image of a scene in which a virtual object is to be displayed, recognizing a body part present in the captured image, and adjusting a display of the virtual object based upon the recognized body part. The rendering operations may also include capturing an image with a body mounted camera, capturing spatial data with a body mounted sensor array, recognizing objects within the captured image, determining distances to the recognized objects within the captured image, and displaying the virtual object on a head mounted display.

Claims

1. A method of rendering images in an augmented reality system, comprising:

capturing an image of a scene, in which a virtual object is to be displayed;

recognizing, via a computing device, a body part present in the captured image;

determining, via the computing device, and in response to the recognizing, whether the body part belongs to a first individual or a second individual;

displaying, via the computing device, the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual; and

displaying, via the computing device, the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual.

2. The method of claim 1, further comprising:

adjusting, via the computing device, displayed portions of the virtual object overlapping the recognized body part such that the virtual object is viewable through the recognized body part and such that the recognized body part appears semitransparent.

3. The method of claim 1, wherein the display comprises a body mounted semitransparent display configured such that that an individual wearing the body mounted semitransparent display views the scene through the display.

4. The method of claim 1, further comprising:

recognizing, via the computing device, objects within the captured image; and

determining, via the computing device, distances to the recognized objects within the captured image.

5. The method of claim 4, wherein the display comprises a head mounted display such that the scene is viewable to an individual wearing the head mounted display.

6. The method of claim 1, wherein the recognizing the body part present in the captured image comprises:

applying, via the computing device, an anatomical analysis to the captured image.

7. The method of claim 6, wherein the applying the anatomical analysis to the captured image comprises:

recognizing, via the computing device, objects within the captured image;

comparing, via the computing device, the recognized objects to a skeletal model stored in a memory, the skeletal model including relative position information about at least one of key joints, bones, or limbs of a human skeleton; and

determining, via the computing device and in response to the comparing, if any of the recognized objects match a portion of the skeletal model.

8. The method of claim 1, further comprising:

determining, via the computing device, a location and an encompassed area of the virtual object; and

determining, via the computing device, whether the recognized body part overlaps at least one portion of the encompassed area of the virtual object.

9. The method of claim 1, wherein the displaying the virtual object so that the virtual object appears to be located beneath the recognized body part comprises not rendering portions of the virtual object which overlap the recognized body part.

10. The method of claim 1, further comprising:

displaying, via the computing device, an outline of portions of the recognized body part of the second individual that overlap the encompassed area of the virtual object such that the outline is visible over the virtual object in the display.

11. A computing device, comprising:

a memory; and

a processor coupled to the memory, wherein the processor is configured with processor-executable instructions to perform operations comprising;

capturing an image of a scene, in which a virtual object is to be displayed;

recognizing a body part present in the captured image;

determining, via the computing device, and in response to the recognizing, whether the body part belongs to a first individual or a second individual;

displaying, via the computing device, the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual; and

displaying, via the computing device, the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual.

12. The computing device of claim 11, further comprising:

adjusting displayed portions of the virtual object overlapping the recognized body part such that the virtual object is viewable through the recognized body part and such that the recognized body part appears semitransparent.

13. The computing device of claim 11, wherein the display comprises a body mounted semitransparent display configured such that that an individual wearing the body mounted semitransparent display views the scene through the display.

14. The computing device of claim 11, wherein the processor is further configured with processor-executable instructions to perform operations comprising:

recognizing objects within the captured image; and

determining distances to the recognized objects within the captured image.

15. The computing device of claim 14, wherein the

display comprises a head mounted display such that the scene is viewable to an individual wearing the head mounted display.

16. The computing device of claim 11, wherein the processor-executable instructions are such that the recognizing the body part present in the captured image comprises applying an anatomical analysis to the captured image.

17. The computing device of claim 16, wherein the processor-executable instructions are such that the applying the anatomical analysis to the image comprises:

recognizing objects within the captured image;

comparing the recognized objects to a skeletal model stored in a memory, the skeletal model including relative position information about at least one of key joints, bones, or limbs of a human skeleton; and

determining, in response to the comparing, if any of the recognized objects match a portion of the skeletal model.

18. The computing device of claim 11, wherein the processor is further configured with processor-executable instructions to perform operations comprising:

determining a location and an encompassed area of the virtual object; and

determining whether the recognized body part overlaps at least one portion of the encompassed area of the virtual object.

19. The computing device of claim 11, wherein the processor-executable instructions are such that the displaying the virtual object so that the virtual object appears to be located beneath the recognized body part comprises not rendering portions of the virtual object which overlap the recognized body part.

20. The computing device of claim 11, wherein the processor is further configured with processor-executable instructions to perform operations comprising:

displaying an outline of portions of the recognized body part of the second individual that overlap the encompassed area of the virtual object such that the outline is visible over the virtual object in the display.

21. A computing device, comprising:

means for capturing an image of a scene, in which a virtual object is to be displayed

means for recognizing a body part present in the image;

means for determining, in response to the recognizing, whether the body part belongs to a first individual or a second individual;

means for displaying the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual; and

means for displaying the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual.

22. The computing device of claim 21, further comprising:

means for adjusting displayed portions of the virtual object overlapping the body part such that the virtual object is viewable through the body part and such that the body part appears semitransparent.

23. The computing device of claim 21, further comprising:

means for recognizing objects within the image; and

means for determining distances to the objects within the image.

24. The computing device of claim 21, wherein the means for recognizing the body part present in the image comprises means for applying an anatomical analysis to the image.

25. The computing device of claim 24, wherein the means for applying the anatomical analysis to the image comprises:

means for recognizing objects within the image;

means for comparing the objects to a skeletal model stored in a memory, the skeletal model including relative position information about at least one of key joints, bones, or limbs of a human skeleton; and

means for determining if any of the objects match a portion of the skeletal model, the means for determining configured to act in response to a comparison of the objects to the skeletal model.

26. The computing device of claim 21, further comprising:

means for determining a location and an encompassed area of the virtual object; and

means for determining whether the body part overlaps at least one portion of the encompassed area of the virtual object.

27. The computing device of claim 21, wherein the means for displaying the virtual object so that the virtual object appears to be located beneath the body part comprises means for not rendering portions of the virtual object which overlap the body part.

28. The computing device of claim 21, further comprising:

means for displaying an outline of portions of the body part of the second individual that overlap the encompassed area of the virtual object such that the outline is visible over the virtual object in the display.

29. A non-transitory computer readable storage medium having stored thereon processor-executable software instructions configured to cause a processor to perform operations for rendering images in an augmented reality system, the operations comprising:

capturing an image of a scene, in which a virtual object is to be displayed;

recognizing a body part present in the captured image;

determining and in response to the recognizing, whether the body part belongs to a first individual or a second individual;

displaying the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual; and

displaying the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual.

30. The non-transitory computer readable storage medium of claim 29, further comprising:

processor-executable software instructions configured for adjusting the display of the virtual object comprises adjusting displayed portions of the virtual object overlapping the recognized body part such that the virtual object is viewable through the recognized body part and such that the recognized body part appears semitransparent.

31. The non-transitory computer readable storage medium of claim 29, wherein the processor-executable software instructions are further configured to cause the processor to perform operations comprising:

recognizing objects within the captured image; and

determining distances to the recognized objects within the captured image.

32. The non-transitory computer readable storage medium of claim 29, wherein the processor-executable software instructions are configured such that the recognizing the body part present in the captured image comprises applying an anatomical analysis to the captured image.

33. The non-transitory computer readable storage medium of claim 32, wherein the processor-executable software instructions are configured such that the applying the anatomical analysis to the image comprises:

recognizing objects within the captured image;

comparing the recognized objects to a skeletal model stored in a memory, the skeletal model including relative position information about at least one of key joints, bones, or limbs of a human skeleton; and

determining, in response to the comparing, if any of the recognized objects match a portion of the skeletal model.

34. The non-transitory computer readable storage medium of claim 29, wherein the processor-executable software instructions are further configured to cause the processor to perform operations comprising:

determining a location and an encompassed area of the virtual object; and

determining whether the recognized body part overlaps at least one portion of the encompassed area of the virtual object.

35. The non-transitory computer readable storage medium of claim 29, wherein the processor-executable software instructions are configured such that the displaying the virtual object so that the virtual object appears to be located beneath the recognized body part comprises not rendering portions of the virtual object which overlap the recognized body part.

36. The non-transitory computer readable storage medium of claim 29, wherein the processor-executable software instructions are further configured to cause the processor to perform operations comprising:

displaying an outline of portions of the recognized body part of the second individual that overlap the encompassed area of the virtual object such that the outline is visible over the virtual object in the display.

37. A method of rendering images in an augmented reality system, comprising:

capturing, via a computing device, an image of a scene in which a virtual object is to be displayed;

recognizing, via the computing device, a body part present in the captured image;

determining, via the computing device and in response to the recognizing, whether the body part belongs to a first individual or a second individual based on at least one of an orientation of the recognized body part, a distance of the recognized body part from a body mounted camera, or a number of body parts determined to be in the captured image;

displaying, via the computing device, the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual; and

displaying, via the computing device, the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual.

38. A computing device, comprising:

a memory; and

a processor coupled to the memory, wherein the processor is configured with processor-executable instructions to perform operations comprising;

capturing an image of a scene in which a virtual object is to be displayed;

recognizing a body part present in the captured image;

determining, in response to the recognizing, whether the body part belongs to a first individual or a second individual based on at least one of an orientation of the recognized body part, a distance of the recognized body part from a body mounted camera, or a number of body parts determined to be in the captured image;

displaying the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual; and

displaying the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual.

39. A computing device, comprising:

means for capturing an image of a scene in which a virtual object is to be displayed;

means for recognizing a body part present in the image;

means for determining whether the body part belongs to a first individual or a second individual based on at least one of an orientation of the body part, a distance of the body part from a body mounted camera, or a number of body parts determined to be in the image, the means for determining configured to act in response to a recognition of the body part in the image;

means for displaying the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual; and

means for displaying the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual.

40. A non-transitory computer readable storage medium having stored thereon processor-executable software instructions configured to cause a processor to perform operations for rendering images in an augmented reality system, the operations comprising:

capturing an image of a scene in which a virtual object is to be displayed;

recognizing a body part present in the captured image;

determining, in response to the recognizing, whether the body part belongs to a first individual or a second individual based on at least one of an orientation of the recognized body part, a distance of the recognized body part from a body mounted camera, or a number of body parts determined to be in the captured image;

displaying the virtual object so that the virtual object appears to be located beneath the recognized body part when it is determined that the body part belongs to the first individual; and

displaying the virtual object superimposed over the recognized body part of the second individual when it is determined that the body part belongs to the second individual.