Augmented reality display apparatus and related methods using database record data

• US 9,007,364 B2
• Filed: 08/24/2012
• Issued: 04/14/2015
• Est. Priority Date: 10/12/2011
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Bailey, Samuel W.
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US13/593,940
• Publication Number:
  US 20130093759A1
• Time in Patent Office: 963 Days
• Field of Search: 345/419, 345/473, 345/474, 345/173, 705/26.1, 707/705, 726/3, 455/414.1
• US Class Current: 345/419
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

Disclosed are examples of methods, apparatus, systems, and computer program products for providing an augmented reality display of an image with record data. In one example, image data is received at one or more processors. A request message is sent requesting record data associated with the image data from one or more of a plurality of records stored in a database system. In some implementations, when the requested record data is received, a graphical display of the record data in combination with the image can be provided on a display device.

Claims

1. A computer implemented method for providing an augmented reality display of an image with database record data, the method comprising:

receiving, at one or more processors, data of the image;

sending a request message requesting record data associated with the image data from one or more of a plurality of Customer Relationship Management (CRM) records stored in a database system, the database system being associated with a CRM service;

receiving the requested record data;

generating a graphical display of the image augmented with the record data based at least in part on one or both of a position or an orientation of the image, the graphical display comprising a selectable link providing one or more of;

access to a CRM record associated with the record data, or communication with a person associated with the image and associated with the record data; and

providing the graphical display of the record data in combination with the image on a display device.

2. The computer implemented method of claim 1, wherein the record data is overlaid on the image in the graphical display.

3. The computer implemented method of claim 1, wherein the record data is stored in a plurality of fields of the one or more records.

4. The computer implemented method of claim 1, wherein the one or more records include a user profile in an online social network, and the record data includes information associated with the user profile.

5. The computer implemented method of claim 1, wherein the graphical display includes a three dimensional (3D) representation of the record data.

6. The computer implemented method of claim 5, wherein the 3D representation of the record data is altered based on a position or orientation of a marker in the image.

7. The computer implemented method of claim 1, wherein the image data includes a marker, the method further comprising:

matching the marker with the record data.

8. The computer implemented method of claim 7, wherein the marker includes one or more of:

a pattern, a string of characters, writing, electronic text, and printed text.

9. The computer implemented method of claim 7, wherein the marker is an object represented in the image.

10. The computer implemented method of claim 7, wherein the marker is associated with an entity type, the method further comprising:

identifying a 3D representation model based on the entity type, wherein the graphical display includes a 3D representation of the record data, the 3D representation based on the 3D representation model.

11. The computer implemented method of claim 1, wherein the graphical display includes a user interface, the method further comprising:

receiving update data associated with the record data from the user interface; and

updating one or more fields of the one or more records in the database system based on the update data.

12. The computer implemented method of claim 1, wherein the record data is included in one or more of an online social network and a CRM application.

13. The computer implemented method of claim 1, wherein the record is associated with a person, and wherein the graphical display provides a user interface for interacting with the person.

14. The computer implemented method of claim 1, further comprising:

receiving data of a second image;

identifying a marker in the second image data;

identifying an entity type based on the marker; and

creating a new record in the database system based on the second image data, the new record being of the identified entity type.

15. The computer implemented method of claim 1, wherein the display device is associated with a computing device, the computing device further including an image capturing device operable to capture the image.

16. A computing system comprising:

one or more servers; and

system software stored on a non-transitory medium for the one or more servers, the system software operable to cause;

retrieving record data associated with image data received from a user device from one or more of a plurality of Customer Relationship Management (CRM) records stored in a database system, the database system being associated with a CRM service; and

generating information configured to be processed to produce a graphical display of an image corresponding to the image data and augmented with the record data based at least in part on one or both of a position or an orientation of the image, the graphical display comprising a selectable link providing one or more of;

access to a CRM record associated with the record data, or communication with a person associated with the image and associated with the record data; and

providing the generated information to the user device.

17. The computing system of claim 16, wherein the record data is overlaid on the image in the graphical display.

18. The computing system of claim 16, wherein the one or more records include a user profile in an online social network, and the record data includes information associated with the user profile.

19. The computing system of claim 16, wherein the graphical display includes a three dimensional (3D) representation of the record data.

20. The computing system of claim 16, wherein the image data includes a marker and wherein the system software is further operable to cause:

matching the marker with the record data.

21. The computing system of claim 20, wherein the marker is associated with an entity type and wherein the system software is further operable to cause:

identifying a 3D representation model based on the entity type, wherein the graphical display includes a 3D representation of the record data, the 3D representation based on the 3D representation model.

22. The computing system of claim 16, wherein the graphical display includes a user interface and wherein the system software is further operable to cause:

receiving update data associated with the record data from the user interface; and

updating one or more fields of the one or more records in the database system based on the update data.

23. The computing system of claim 16, wherein the record data is included in one or more of an online social network and a CRM application.

24. The computing system of claim 16, wherein the record is associated with a person, and wherein the graphical display provides a user interface for interacting with the person.

25. The computing system of claim 16, wherein the system software is further operable to cause:

receiving data of a second image;

identifying a marker in the second image data;

identifying an entity type based on the marker; and

creating a new record in the database system based on the second image data, the new record being of the identified entity type.

26. The computing system of claim 16, wherein the display device is associated with a computing device, the computing device further including an image capturing device operable to capture the image.

27. A computing device comprising:

a display device;

an image capturing device operable to capture an image; and

a controller configured to;

receive data of the image;

send a request message requesting record data associated with the image data from one or more of a plurality of Customer Relationship Management (CRM) records stored in a database system, the database system being associated with a CRM service;

receive the requested record data; and

generate a graphical display of the image augmented with the record data based at least in part on one or both of a position or an orientation of the image, the graphical display comprising a selectable link providing one or more of;

access to a CRM record associated with the record data, or communication with a person associated with the image and associated with the record data; and

provide the graphical display of the record data in combination with the image on a display device.

28. The computing device of claim 27, wherein the graphical display includes a 3D representation of the record data and wherein the 3D representation of the record data is altered based on a position or orientation of a marker in the image.

29. A non-transitory tangible computer-readable storage medium storing instructions executable by one or more processors of a computing device to cause a method to be performed for providing an augmented reality display of an image with database record data, the method comprising:

receiving, at one or more processors, data of the image;

sending a request message requesting record data associated with the image data from one or more of a plurality of Customer Relationship Management (CRM) records stored in a database system, the database system being associated with a CRM service;

receiving the requested record data;

generating a graphical display of the image augmented with the record data based at least in part on one or both of a position or an orientation of the image, the graphical display comprising a selectable link providing one or more of;

access to a CRM record associated with the record data, or communication with a person associated with the image and associated with the record data; and

providing the graphical display of the record data in combination with the image on a display device.

30. The non-transitory tangible computer-readable storage medium of claim 29, wherein the graphical display includes a 3D representation of the record data and wherein the 3D representation of the record data is altered based on a position or orientation of a marker in the image.

31. The non-transitory tangible computer-readable storage medium of claim 29, wherein the graphical display includes a user interface, the method further comprising:

receiving update data associated with the record data from the user interface; and

updating one or more fields of the one or more records in the database system based on the update data.

32. The non-transitory tangible computer-readable storage medium of claim 29, the method further comprising:

receiving data of a second image;

identifying a marker in the second image data;

identifying an entity type based on the marker; and

creating a new record in the database system based on the second image data, the new record being of the identified entity type.