DETERMINING NETWORKED MOBILE DEVICE POSITION AND ORIENTATION FOR AUGMENTED-REALITY WINDOW SHOPPING

• US 20140100996A1
• Filed: 10/05/2012
• Published: 04/10/2014
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Klein, Udo, Mayerle, Jochen, Hoyer, Timo, Tatzel, Steffen, Koshy, Sanu
• Granted Patent: US 9,367,870 B2
• Time in Patent Office: Days
• US Class Current: 705/27.2
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 1/00 General purpose image data processing

Abstract

A consumer user can perform a number of augmented reality shopping operations using a local client shopping application executable on a mobile device. Features related to implementations of the current subject matter can include, but are not limited to, the ability to view an augmented reality depiction of a shopping display using the mobile device and to execute various “e-commerce” functions accessible via the augmented reality depiction. Visual image processing based on one or more inputs from sensors on the mobile device can be used to compute and efficiently provide the two dimensional projection of the three dimensional model to be presented on the screen of the mobile device. Related methods, systems, articles of manufacture, and the like are described.

Claims

1. A computer program product comprising a machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

narrowing, using data provided by one or more sensors associated with a mobile device executing a local client shopping application, a number of available degrees of freedom relating to a current view of a shopping display being sensed by a camera of the mobile device;

estimating, based on the narrowed available degrees of freedom and using a computer vision algorithm, a pose and position of the camera relative to a three dimensional model of a shopping display;

computing, based on the pose and position of the camera, a two dimensional projection of the three dimensional model; and

rendering a two dimensional picture for presentation via a screen on the mobile device screen, the two dimensional picture comprising an augmented reality marker associated with a physical item in the shopping display.

2. A computer program product as in claim 1, wherein the one or more sensors comprise at least one of a radio signal triangulation sensor, a global positioning transceiver, a compass sensor, and an accelerometer sensor.

3. A computer program product as in claim 1, wherein the operations further comprise:

identifying, based the data provided by the one or more sensors, a most likely shopping display being viewed through a camera associated with the mobile device; and

determining the three dimensional model of the shopping display.

4. A computer program product as in claim 1, wherein the operations further comprise:

identifying the shopping display based on a visual or textual encoding of a location code entered into the mobile device via the camera or a textual interface on the mobile device.

5. A computer program product as in claim 1, wherein the operations further comprise:

presenting, via the screen, the two dimensional picture.

6. A computer program product as in claim 1, wherein the operations further comprise:

receiving, via a user input, a selection of the augmented reality marker; and

presenting, on the screen of the mobile device in response to the received selection, an item page corresponding to the physical item, the item page comprising additional information about at least one of the physical item and a good or service represented by the physical item.

12. A computer program product as in claim 1, wherein the operations further comprise:

receiving, via a user input, a selection of the augmented reality marker; and

presenting, on the screen of the mobile device in response to the received selection, an item page corresponding to the physical item, the item page comprising additional information about at least one of the physical item and a good or service represented by the physical item.

7. A system comprising:

at least one programmable processor; and

a machine-readable medium storing instructions that, when executed by the at least one processor, cause the at least one programmable processor to perform operations comprising;

narrowing, using data provided by one or more sensors associated with a mobile device executing a local client shopping application, a number of available degrees of freedom relating to a current view of a shopping display being sensed by a camera of the mobile device;

estimating, based on the narrowed available degrees of freedom and using a computer vision algorithm, a pose and position of the camera relative to a three dimensional model of a shopping display;

computing, based on the pose and position of the camera, a two dimensional projection of the three dimensional model; and

rendering a two dimensional picture for presentation via a screen on the mobile device screen, the two dimensional picture comprising an augmented reality marker associated with a physical item in the shopping display.

8. A system as in claim 7, wherein the one or more sensors comprise at least one of a radio signal triangulation sensor, a global positioning transceiver, a compass sensor, and an accelerometer sensor.

9. A system as in claim 7, wherein the operations further comprise:

identifying, based the data provided by the one or more sensors, a most likely shopping display being viewed through a camera associated with the mobile device; and

determining the three dimensional model of the shopping display.

10. A system as in claim 7, wherein the operations further comprise:

identifying the shopping display based on a visual or textual encoding of a location code entered into the mobile device via the camera or a textual interface on the mobile device.

11. A system as in claim 7, wherein the operations further comprise:

presenting, via the screen, the two dimensional picture.

13. A computer-implemented method comprising:

narrowing, using data provided by one or more sensors associated with a mobile device executing a local client shopping application, a number of available degrees of freedom relating to a current view of a shopping display being sensed by a camera of the mobile device;

estimating, based on the narrowed available degrees of freedom and using a computer vision algorithm, a pose and position of the camera relative to a three dimensional model of a shopping display;

computing, based on the pose and position of the camera, a two dimensional projection of the three dimensional model; and

rendering a two dimensional picture for presentation via a screen on the mobile device screen, the two dimensional picture comprising an augmented reality marker associated with a physical item in the shopping display.

14. A computer-implemented method as in claim 13, wherein the one or more sensors comprise at least one of a radio signal triangulation sensor, a global positioning transceiver, a compass sensor, and an accelerometer sensor.

15. A computer-implemented method as in claim 13, further comprising:

identifying, based the data provided by the one or more sensors, a most likely shopping display being viewed through a camera associated with the mobile device; and

determining the three dimensional model of the shopping display.

16. A computer-implemented method as in claim 13, further comprising:

identifying the shopping display based on a visual or textual encoding of a location code entered into the mobile device via the camera or a textual interface on the mobile device.

17. A computer-implemented method as in claim 13, wherein the operations further comprise:

presenting, via the screen, the two dimensional picture.

18. A computer-implemented method as in claim 13, further comprising:

receiving, via a user input, a selection of the augmented reality marker; and

presenting, on the screen of the mobile device in response to the received selection, an item page corresponding to the physical item, the item page comprising additional information about at least one of the physical item and a good or service represented by the physical item.

19. A computer-implemented method as in claim 13, wherein at least one of the narrowing, the estimating, the computing, and the rendering is performed by at least one programmable processor.