AUGMENTED REALITY SURFACE SEGMENTATION

• US 20140063060A1
• Filed: 03/13/2013
• Published: 03/06/2014
• Est. Priority Date: 09/04/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: MACIOCCI, Giuliano, MULHOLLAND, James Joseph
• Granted Patent: US 9,530,232 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Methods, systems, computer-readable media, and apparatuses for providing intuitive, functional, and convenient ways of enabling a user of a head-mounted display unit or another augmented reality enabled device to interact with various user interfaces and other features provided by such a unit or device are presented. In some embodiments, a computing device, such as a head-mounted display unit, may receive camera input of a scene. Subsequently, the computing device may identify at least one reference object in the scene, for example, based on detecting one or more rectangles in the received camera input. The computing device then may receive input that defines a surface segment relative to the at least one reference object. Thereafter, the computing device may render the surface segment.

Claims

1. A method comprising:

receiving camera input of a scene;

identifying at least one reference object in the scene, wherein the at least one reference object is a physical object;

receiving input defining a surface segment relative to the at least one reference object; and

causing the surface segment to be rendered.

2. The method of claim 1, wherein identifying at least one reference object in the scene comprises detecting one or more feature points in the received camera input.

3. The method of claim 1, wherein identifying at least one reference object in the scene comprises receiving a wireless communication from the at least one reference object.

4. The method of claim 1, wherein identifying at least one reference object in the scene comprises detecting one or more rectangles in the received camera input.

5. The method of claim 4, further comprising:

determining, based on the one or more detected rectangles, a current perspective of the scene, wherein the surface segment is rendered relative to the current perspective of the scene.

6. The method of claim 5, further comprising:

determining that the current perspective of the scene has changed; and

dynamically updating the rendered surface segment.

7. The method of claim 1, wherein the input defining the surface segment is a finger movement defining a shape of the surface segment, wherein the finger movement is performed by a user of a device causing the surface segment to be rendered.

8. The method of claim 7, wherein causing the surface segment to be rendered includes:

rendering, in a virtual workspace, the surface segment at a first angle; and

rendering, in the virtual workspace, a second surface segment defined by the user at a second angle different from the first angle.

9. The method of claim 8, further comprising:

determining that the at least one reference object has been removed from the scene; and

closing the virtual workspace.

10. The method of claim 9, further comprising:

determining that the at least one reference object has been reintroduced into the scene or another scene; and

opening the virtual workspace.

11. The method of claim 8, further comprising:

detecting a second reference object different from the at least one reference object; and

opening a second virtual workspace different from the virtual workspace.

12. The method of claim 1, wherein the identifying the at least one reference object is based on an image displayed by the at least one reference object.

13. The method of claim 1, further comprising:

transmitting a data signal comprising a virtual workspace that includes the surface segment to least one other device,

wherein the virtual workspace is defined by a first user that provided the input defining the surface segment, and

wherein the at least one other device is associated with a second user different from the first user.

14. The method of claim 13,

wherein causing the surface segment to be rendered comprises:

rendering the virtual workspace for the first user; and

based on interaction of the second user with one or more virtual objects included in the virtual workspace, dynamically updating the rendering of the virtual workspace for the first user.

15. The method of claim 1, further comprising:

determining a pose of the at least one reference object based at least in part on the camera input; and

determining a virtual surface separate from the at least one reference object based at least in part on the determined pose, wherein the surface segment is rendered on the virtual surface.

16. A system comprising:

an optical sensor configured to detect an image of a scene;

a display;

a processor configured to;

identify at least one reference object in the scene, wherein the at least one reference object is a physical object;

receive input defining a surface segment relative to the at least one reference object; and

render the surface segment in the display.

17. The system of claim 16, wherein the processor is configured to identify the at least one reference object in the scene based on detecting one or more feature points in the image of the scene.

18. The system of claim 16, wherein the processor is configured to identify the at least one reference object in the scene based on receiving a wireless communication from the at least one reference object.

19. The system of claim 16, wherein the processor is configured to identify the at least one reference object in the scene based on detecting one or more rectangles in the image of the scene.

20. The system of claim 19, wherein the processor is further configured to:

determine, based on the one or more detected rectangles, a current perspective of the scene,

wherein the surface segment is rendered relative to the current perspective of the scene.

21. The system of claim 20, wherein the processor is further configured to:

determine that the current perspective of the scene has changed; and

dynamically update the rendered surface segment.

22. The system of claim 16, wherein the input defining the surface segment is a finger movement defining a shape of the surface segment, wherein the finger movement is performed by a user of a device causing the surface segment to be rendered.

23. The system of claim 22, wherein rendering the surface segment includes:

rendering, in a virtual workspace, the surface segment at a first angle; and

rendering, in the virtual workspace, a second surface segment defined by the user at a second angle different from the first angle.

24. The system of claim 23, wherein the processor is further configured to:

determine when the at least one reference object has been removed from the scene; and

close the virtual workspace in response to determining that the at least one reference object has been removed.

25. The system of claim 24, wherein the processor is further configured to:

determine that the at least one reference object has been reintroduced into the scene or another scene; and

open the virtual workspace in response to determining the at least one reference object has been reintroduced into the scene or another scene.

26. The system of claim 23, wherein the processor is further configured to:

detect a second reference object different from the at least one reference object; and

open a second virtual workspace different from the virtual workspace.

27. The system of claim 16, wherein the identifying the at least one reference object is based on an image displayed by the at least one reference object.

28. The system of claim 16, further comprising:

a transmitter configured to transmit a data signal comprising a virtual workspace that includes the surface segment to at least one other device,

wherein the virtual workspace is defined by a first user that provided the input defining the surface segment, and

wherein the at least one other device is associated with a second user different from the first user.

29. The system of claim 28, further comprising:

a receiver configured to receive the data signal; and

wherein rendering the surface segment includes;

rendering the virtual workspace for the first user; and

based on interaction of the second user with one or more virtual objects included in the virtual workspace, dynamically updating the rendering of the virtual workspace for the first user.

30. The system of claim 16, wherein the processor is further configured to determine a pose of the at least one reference object based at least in part on the image of the scene, and determine a virtual surface separate from the at least one reference object based at least in part on the determined pose, wherein the surface segment is rendered on the virtual surface.

31. A system comprising:

a means for optically sensing an image of a scene;

a means for identifying at least one reference object in the scene, wherein the at least one reference object is a physical object;

a means for receiving input defining a surface segment relative to the at least one reference object; and

a means for rendering the surface segment.

32. A non-transient computer readable medium comprising program code, which when executed by a processor is configured to cause the processor to:

receive a camera input of a scene;

identify at least one reference object in the scene, wherein the at least one reference object is a physical object;

receive input defining a surface segment relative to the at least one reference object; and

cause the surface segment to be rendered.