Augmented reality surface segmentation

• US 9,530,232 B2
• Filed: 03/13/2013
• Issued: 12/27/2016
• Est. Priority Date: 09/04/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Maciocci, Giuliano, Mulholland, James Joseph
• Primary Examiner(s): Chu, David H
• Application Number:
  US13/802,194
• Publication Number:
  US 20140063060A1
• Time in Patent Office: 1,385 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Methods, systems, computer-readable media, and apparatuses for providing intuitive, functional, and convenient ways of enabling a user of a head-mounted display unit or another augmented reality enabled device to interact with various user interfaces and other features provided by such a unit or device are presented. In some embodiments, a computing device, such as a head-mounted display unit, may receive camera input of a scene. Subsequently, the computing device may identify at least one reference object in the scene, for example, based on detecting one or more rectangles in the received camera input. The computing device then may receive input that defines a surface segment relative to the at least one reference object. Thereafter, the computing device may render the surface segment.

Claims

1. A method comprising:

receiving a first camera input of a scene;

identifying at least one reference object in the scene, wherein the at least one reference object is a physical object;

receiving a second camera input comprising user input from a first user defining a surface segment relative to the at least one reference object, wherein the user input defining the surface segment is a finger movement defining a shape of the surface segment; and

causing the surface segment to be rendered in the first user'"'"'s field of vision as though on a surface on which the physical object is located.

2. The method of claim 1, wherein identifying at least one reference object in the scene comprises detecting one or more feature points in the first camera input.

3. The method of claim 1, wherein identifying at least one reference object in the scene comprises receiving a wireless communication from the at least one reference object.

4. The method of claim 1, wherein identifying at least one reference object in the scene comprises detecting one or more rectangles in the first camera input.

5. The method of claim 4, further comprising:

determining, based on the one or more detected rectangles, a current perspective of the scene, wherein the surface segment is rendered relative to the current perspective of the scene.

6. The method of claim 5, further comprising:

determining that the current perspective of the scene has changed; and

dynamically updating the rendered surface segment.

7. The method of claim 1, wherein the finger movement is performed by the first user of a device causing the surface segment to be rendered.

8. The method of claim 7, wherein causing the surface segment to be rendered includes:

rendering, in a virtual workspace, the surface segment at a first angle; and

rendering, in the virtual workspace, a second surface segment defined by the first user at a second angle different from the first angle.

9. The method of claim 8, further comprising:

determining that the at least one reference object has been removed from the scene; and

closing the virtual workspace.

10. The method of claim 9, further comprising:

determining that the at least one reference object has been reintroduced into the scene or another scene; and

opening the virtual workspace.

11. The method of claim 8, further comprising:

detecting a second reference object different from the at least one reference object; and

opening a second virtual workspace different from the virtual workspace.

12. The method of claim 1, wherein the identifying the at least one reference object is based on an image displayed by the at least one reference object.

13. The method of claim 1, further comprising:

transmitting a data signal comprising a virtual workspace that includes the surface segment to an at least one other device,

wherein the virtual workspace is defined by the first user that provided the input defining the surface segment, and

wherein the at least one other device is associated with a second user different from the first user.

14. The method of claim 13,

wherein causing the surface segment to be rendered comprises:

rendering the virtual workspace for the first user; and

based on interaction of the second user with one or more virtual objects included in the virtual workspace, dynamically updating the rendering of the virtual workspace for the first user.

15. The method of claim 1, further comprising:

determining a pose of the at least one reference object based at least in part on the camera input; and

determining a virtual surface separate from the at least one reference object based at least in part on the determined pose, wherein the surface segment is rendered on the virtual surface.

16. The method of claim 1, wherein the first camera input and the second camera input are different signals.

17. A system comprising:

an optical sensor configured to transmit a first input signal comprising an image of a scene;

a display;

a processor configured to;

identify at least one reference object in the scene, wherein the at least one reference object is a physical object;

receive a second input signal from the optical sensor, the second input signal comprising user input from a first user defining a surface segment relative to the at least one reference object, wherein the user input defining the surface segment is a finger movement defining a shape of the surface segment; and

render the surface segment in the display to appear in the first user'"'"'s field of vision as though on a surface on which the physical object is located.

18. The system of claim 17, wherein the processor is configured to identify the at least one reference object in the scene based on detecting one or more feature points in the image of the scene.

19. The system of claim 17, wherein the processor is configured to identify the at least one reference object in the scene based on receiving a wireless communication from the at least one reference object.

20. The system of claim 17, wherein the processor is configured to identify the at least one reference object in the scene based on detecting one or more rectangles in the image of the scene.

21. The system of claim 20, wherein the processor is further configured to:

determine, based on the one or more detected rectangles, a current perspective of the scene,

wherein the surface segment is rendered relative to the current perspective of the scene.

22. The system of claim 21, wherein the processor is further configured to:

determine that the current perspective of the scene has changed; and

dynamically update the rendered surface segment.

23. The system of claim 17, wherein the finger movement is performed by the first user of a device causing the surface segment to be rendered.

24. The system of claim 23, wherein rendering the surface segment includes:

rendering, in a virtual workspace, the surface segment at a first angle; and

rendering, in the virtual workspace, a second surface segment defined by the first user at a second angle different from the first angle.

25. The system of claim 24, wherein the processor is further configured to:

determine when the at least one reference object has been removed from the scene; and

close the virtual workspace in response to determining that the at least one reference object has been removed.

26. The system of claim 25, wherein the processor is further configured to:

determine that the at least one reference object has been reintroduced into the scene or another scene; and

open the virtual workspace in response to determining the at least one reference object has been reintroduced into the scene or another scene.

27. The system of claim 24, wherein the processor is further configured to:

detect a second reference object different from the at least one reference object; and

open a second virtual workspace different from the virtual workspace.

28. The system of claim 17, wherein the identifying the at least one reference object is based on an image displayed by the at least one reference object.

29. The system of claim 17, further comprising:

a transmitter configured to transmit a data signal comprising a virtual workspace that includes the surface segment to at least one other device,

wherein the virtual workspace is defined by the first user that provided the input defining the surface segment, and

wherein the at least one other device is associated with a second user different from the first user.

30. The system of claim 29, further comprising:

a receiver configured to receive the data signal; and

wherein rendering the surface segment includes;

rendering the virtual workspace for the first user; and

based on interaction of the second user with one or more virtual objects included in the virtual workspace, dynamically updating the rendering of the virtual workspace for the first user.

31. The system of claim 17, wherein the processor is further configured to determine a pose of the at least one reference object based at least in part on the image of the scene, and determine a virtual surface separate from the at least one reference object based at least in part on the determined pose, wherein the surface segment is rendered on the virtual surface.

32. The system of claim 17, wherein the first input signal and the second input signal are different signals.

33. A system comprising:

a means for optically sensing configured to transmit a first input signal comprising an image of a scene;

a means for identifying at least one reference object in the scene, wherein the at least one reference object is a physical object;

a means for receiving a second input signal from the means for optically sensing, the second input signal comprising user input defining a surface segment relative to the at least one reference object, wherein the user input defining the surface segment is a finger movement defining a shape of the surface segment; and

a means for rendering the surface segment to appear in a user'"'"'s field of vision as though on a surface on which the physical object is located.

34. The system of claim 33, wherein the first input signal and the second input signal are different signals.

35. A non-transient computer readable medium comprising program code, which when executed by a processor is configured to cause the processor to:

receive a first camera input of a scene;

identify at least one reference object in the scene, wherein the at least one reference object is a physical object;

receive a second camera input, the second camera input defining a surface segment relative to the at least one reference object, wherein the second camera input comprises user input defining the surface segment, wherein the user input defining the surface segment is a finger movement defining a shape of the surface segment; and

cause the surface segment to be rendered in a user'"'"'s field of vision as though on a surface on which the physical object is located.

36. The non-transient computer readable medium of claim 35, wherein the first camera input and the second camera input are different signals.