Data manipulation based on real world object manipulation

• US 9,639,984 B2
• Filed: 06/03/2013
• Issued: 05/02/2017
• Est. Priority Date: 06/03/2013
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian
• Primary Examiner(s): HAJNIK, DANIEL F
• Application Number:
  US13/909,046
• Publication Number:
  US 20140354686A1
• Time in Patent Office: 1,429 Days
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

A system and method for data manipulation based on real world object manipulation is described. A device captures an image of a physical object. The image is communicated via a network to a remote server. The remote server includes virtual object data associated with the image and a communication notification for a user of the computing device. The device receives the virtual object data and displays the virtual image in a virtual landscape using the virtual object data. In response to relative movement between the computing device and the physical object caused by the user, the virtual image is modified.

Claims

1. A method comprising:

receiving, by a computing device, a primary content dataset from a remote server, the primary content dataset including a subset of known visual references stored by the remote server that were determined to be most popular by the remote server, wherein each known visual reference corresponds to a virtual object model;

storing, by the computing device, the primary content dataset in a local memory of the computing device;

after receiving the primary content dataset, capturing, by an optical sensor of the computing device, an image of a physical object, the image including a captured visual reference associated with the physical object;

searching, by the computing device, the primary content dataset stored in the local memory of the computing device based on the captured visual reference to identify a virtual object model corresponding to the captured visual reference associated with the physical object;

in response to determining that the captured visual reference is not included in the primary content dataset in the local memory of the computing device, transmitting, by the computing device, the image to the remote server, causing the remote server to search the known visual references stored by the remote server based on the captured visual reference to identify the visual object model corresponding to the captured visual reference;

receiving, from the remote server, the virtual object model corresponding to the captured visual reference associated with the physical object;

displaying, based on the virtual object model, a virtual image over a live image of the physical object;

in response to detecting a movement of the computing device relative to the physical object caused by the user, modifying the virtual image;

detecting a physical manipulation of the physical object;

identifying a virtual function assigned to the physical manipulation of the physical object; and

performing the identified virtual function, the virtual function controlling a physical device electrically unconnected to the physical object.

2. The method of claim 1, further comprising:

receiving a virtual functionality dataset comprising virtual functionalities associated with physical manipulations of the physical object.

3. The method of claim 1, further comprising:

receiving first notification data from a first account associated with an identification of a first physical object; and

receiving second notification data from a second account associated with an identification of a second physical object, the first notification data and the second notification data comprising an email notification, a calendar reminder notification, or a social network notification.

4. The method of claim 3, further comprising:

accessing reference identifiers associated with the first notification data;

accessing an augmented reality visualization associated with the reference identifiers associated with the first notification data; and

generating a virtual object engaged with a reference identifier in the image of the physical object using the augmented reality visualization associated with the reference identifier in the image of the physical object.

5. The method of claim 4, further comprising:

changing a feature of the virtual object, or replacing the virtual object with another virtual object based on the augmented reality visualization associated with the reference identifier in the image of the physical object.

6. The method of claim 2, further comprising:

generating the virtual function corresponding to the identified physical manipulation of the physical object.

7. The method of claim 6, further comprising:

generating a virtual user interface corresponding to the identified physical manipulation of the physical object.

8. The method of claim 6, further comprising:

activating a function on the computing device in response to the identified physical manipulation of the physical object.

9. The method of claim 6, further comprising:

generating a command for the virtual function to control another device in response to the identified physical manipulation of the physical object, the other device having no electronic communication with the physical object.

10. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a computing device, cause the computing device to:

receive a primary content dataset from a remote server, the primary content dataset including a subset of known visual references stored by the remote server that were determined to be most popular by the remote server, wherein each known visual reference corresponds to a virtual object model;

store the primary content dataset in a local memory of the computing device;

after receiving the primary content dataset, capture an image of a physical object, the image including a captured visual reference associated with the physical object;

search the primary content dataset stored in the local memory of the computing device based on the captured visual reference to identify a virtual object model corresponding to the captured visual reference associated with the physical object;

in response to determining that the captured visual reference is not included in the primary content dataset in the local memory of the computing device, transmit the image to the remote server, causing the remote server to search the known visual references stored by the remote server based on the captured visual reference to identify the visual object model corresponding to the captured visual reference;

receive, from the remote server, the virtual object model corresponding to the captured visual reference associated with the physical object;

display, based on the virtual object model, a virtual image over a live image of the physical object;

in response to detecting a movement of the computing device relative to the physical object caused by the user, modify the virtual image;

detect a physical manipulation of the physical object;

identify a virtual function assigned to the physical manipulation of the physical object; and

perform the identified virtual function, the virtual function controlling a physical device electrically unconnected to the physical object.

11. A computing device comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the computing device to;

receive a primary content dataset from a remote server, the primary content dataset including a subset of known visual references stored by the remote server that were determined to be most popular by the remote server, wherein each known visual reference corresponds to a virtual object model;

store the primary content dataset in a local memory of the computing device;

after receiving the primary content dataset, capture an image of a physical object, the image including a captured visual reference associated with the physical object;

search the primary content dataset stored in the local memory of the computing device based on the captured visual reference to identify a virtual object model corresponding to the captured visual reference associated with the physical object;

in response to determining that the captured visual reference is not included in the primary content dataset in the local memory of the computing device, transmit the image to the remote server, causing the remote server to search the known visual references stored by the remote server based on the captured visual reference to identify the visual object model corresponding to the captured visual reference;

receive, from the remote server, the virtual object model corresponding to the captured visual reference associated with the physical object;

display, based on the virtual object model, a virtual image over a live image of the physical object;

in response to detecting a movement of the computing device relative to the physical object caused by the user, modify the virtual image;

detect a physical manipulation of the physical object;

identify a virtual function assigned to the physical manipulation of the physical object; and

perform the identified virtual function, the virtual function controlling a physical device electrically unconnected to the physical object.

12. The computing device of claim 11, wherein the instructions further cause the computing device to:

receive a virtual functionality dataset comprising virtual functionalities associated with physical manipulations of the physical object.

13. The computing device of claim 11, wherein the instructions further cause the computing device to:

receive first notification data from a first account associated with an identification of a first physical object; and

receive second notification data from a second account associated with an identification of a second physical object, the first notification data and the second notification data comprising an email notification, a calendar reminder notification, or a social network notification.

14. The computing device of claim 13, wherein the instructions further cause the computing device to:

access reference identifiers associated with the first notification data;

access an augmented reality visualization associated with the reference identifiers associated with the first notification data; and

generate a virtual object engaged with a reference identifier in the image of the physical object using the augmented reality visualization associated with the reference identifier in the image of the physical object.

15. The computing device of claim 14, wherein the instructions further cause the computing device to:

change a feature of the virtual object; and

replace the virtual object with another virtual object based on the augmented reality visualization associated with the reference identifier in the image of the physical object.

16. The computing device of claim 12, wherein the instructions further cause the computing device to:

identify the physical object and virtual functionalities associated with physical manipulations of the physical object, the physical object including a non-electric physical object;

identify the physical manipulation of the physical object; and

generate the virtual function corresponding to the identified physical manipulation of the physical object.

17. The computing device of claim 16, wherein the instructions further cause the computing device to:

generate a virtual user interface corresponding to the identified physical manipulation of the physical object.

18. The computing device of claim 16, wherein the instructions further cause the computing device to:

activate a function on the device in response to the identified physical manipulation of the physical object.

19. The computing device of claim 16, wherein the instructions further cause the computing device to:

generate a command for the virtual function to control another device in response to the identified physical manipulation of the physical object, the other device having no electronic communication with the physical object.

20. The computing device of claim 16, wherein the physical manipulation of the physical object comprises moving or rotating the physical object.