Augmented reality object manipulation

• US 10,593,116 B2
• Filed: 04/28/2017
• Issued: 03/17/2020
• Est. Priority Date: 10/24/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Egri, Ozi, Ezra, David Ben, McPhee, Andrew James, Pan, Qi, Zak, Eyal
• Primary Examiner(s): Ma, Tize
• Application Number:
  US15/581,994
• Publication Number:
  US 20180114365A1
• Time in Patent Office: 1,054 Days
• Field of Search: None
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts

Abstract

Among other things, embodiments of the present disclosure improve the functionality of computer imaging software and systems by facilitating the manipulation of virtual content displayed in conjunction with images of real-world objects and environments. Embodiments of the present disclosure allow different virtual objects to be moved onto different physical surfaces, as well as manipulated in other ways.

Claims

1. A system comprising:

a processor;

a user interface coupled to the processor, the user interface comprising an input device, a camera, and a display screen; and

memory coupled to the processor and storing instructions that, when executed by the processor, cause the system to perform operations comprising;

causing an image comprising a real-world scene captured via the camera to be displayed on the display screen;

causing a virtual object to be displayed on the display screen as being anchored to a primary physical element within the image;

tracking the primary physical element within the image;

detecting an interaction by a secondary physical element with the virtual object;

in response to detecting the interaction by the secondary physical element with the virtual object,

switching the virtual object from being anchored to the primary physical element to being anchored to the secondary physical element, and

causing the virtual object to be displayed on the display screen as being anchored to the secondary physical element within the image;

detecting removal of the secondary physical element, wherein detecting removal of the secondary physical element includes detecting that the secondary physical element is not within the image; and

in response to detecting removal of the secondary physical element,

switching the virtual object from being anchored to the secondary physical element to being anchored to the primary physical element, and

causing the virtual object to be displayed on the display screen as being anchored to the primary physical element within the image.

2. The system of claim 1, wherein the memory further stores instructions for causing the system to perform operations comprising:

in response to detecting the interaction by the secondary physical element with the virtual object,

switching from tracking the primary physical element to tracking the secondary physical element.

3. The system of claim 2, wherein the memory further stores instructions for causing the system to perform operations comprising:

in response to detecting the interaction by the secondary physical element with the virtual object,

determining whether the secondary physical element surpasses a predetermined tolerance limit; and

switching from tracking the primary physical element to tracking the secondary physical element in response to determining that the secondary physical element surpasses the predetermined tolerance limit.

4. The system of claim 1, wherein the memory further stores instructions for causing the system to perform operations comprising:

in response to detecting removal of the secondary physical element, resuming tracking of the primary physical element.

5. The system of claim 1, wherein the memory further stores instructions for causing the system to perform operations comprising:

in response to detecting removal of the secondary physical element, anchoring the virtual object to a closest physical entity in the image.

6. The system of claim 5, wherein the closest physical entity is one or more of:

a human, an animal, a surface, and an object.

7. The system of claim 1, wherein the memory further stores instructions for causing the system to perform operations comprising triggering a behavior of the virtual object.

8. The system of claim 1, wherein tracking the primary physical element includes tracking the primary physical element as the system is moved, and wherein the movement of the system includes one or more of:

a movement from a first position to a second position, and a rotation.

9. A computer-implemented method comprising:

causing, by a processor, an image comprising a real-world scene captured via a camera to be displayed on a display screen;

causing a virtual object to be displayed on the display screen as being anchored to a primary physical element within the image;

tracking the primary physical element within the image;

detecting an interaction by a secondary physical element with the virtual object;

in response to detecting the interaction by the secondary physical element with the virtual object,

switching the virtual object from being anchored to the primary physical element to being anchored to the secondary physical element, and

causing the virtual object to be displayed on the display screen as being anchored to the secondary physical element within the image;

detecting removal of the secondary physical element, wherein detecting removal of the secondary physical element includes detecting that the secondary physical element is not within the image; and

in response to detecting removal of the secondary physical element,

switching the virtual object from being anchored to the secondary physical element to being anchored to the primary physical element, and

causing the virtual object to be displayed on the display screen as being anchored to the primary physical element within the image.

10. The method of claim 9, further comprising:

in response to detecting the interaction by the secondary physical element with the virtual object,

switching from tracking the primary physical element to tracking the secondary physical element.

11. The method of claim 10, further comprising:

in response to detecting the interaction by the secondary physical element with the virtual object,

determining whether the secondary physical element surpasses a predetermined tolerance limit; and

switching from tracking the primary physical element to tracking the secondary physical element in response to determining that the secondary physical element surpasses the predetermined tolerance limit.

12. The method of claim 9, further comprising:

in response to detecting removal of the secondary physical element, resuming tracking of the primary physical element.

13. The method of claim 9, further comprising:

in response to detecting removal of the secondary physical element, anchoring the virtual object to a closest physical entity in the image.

14. The method of claim 13, wherein the closest physical entity is one or more of:

a human, an animal, a surface, and an object.

15. The method of claim 9, further comprising:

triggering a behavior of the virtual object.

16. The method of claim 9, wherein tracking the primary physical element includes tracking the primary physical element as the system is moved, and wherein the movement of the system includes one or more of:

a movement from a first position to a second position, and a rotation.

17. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

causing an image comprising a real-world scene captured via a camera to be displayed on a display screen;

causing a virtual object to be displayed on the display screen as being anchored to a primary physical element within the image;

tracking the primary physical element within the image;

detecting an interaction by a secondary physical element with the virtual object;

in response to detecting the interaction by the secondary physical element with the virtual object,

switching the virtual object from being anchored to the primary physical element to being anchored to the secondary physical element, and

causing the virtual object to be displayed on the display screen as being anchored to the secondary physical element within the image;

detecting removal of the secondary physical element, wherein detecting removal of the secondary physical element includes detecting that the secondary physical element is not within the image; and

in response to detecting removal of the secondary physical element,

switching the virtual object from being anchored to the secondary physical element to being anchored to the primary physical element, and

causing the virtual object to be displayed on the display screen as being anchored to the primary physical element within the image.

18. The computer-readable medium of claim 17, wherein the computer system to perform operations further comprising:

in response to detecting the interaction by the secondary physical element with the virtual object,

switching from tracking the primary physical element to tracking the secondary physical element.