Graphic representations of planograms

• US 9,886,678 B2
• Filed: 09/25/2013
• Issued: 02/06/2018
• Est. Priority Date: 09/25/2013
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Yan, Sui
• Primary Examiner(s): Zare, Scott
• Assistant Examiner(s): Danzig, Reva R
• Application Number:
  US14/037,121
• Publication Number:
  US 20150088703A1
• Time in Patent Office: 1,595 Days
• Field of Search: 705 28, 707E17001
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders

Abstract

Systems and methods for generating and distributing graphic representations of planograms used for the configuration, maintenance, and compliance confirmation in the display of merchandise and advertisements, are disclosed. Based on predefined merchandizing agreements, planograms are designed to define specific configurations of products and advertising collateral in specific locations. By referencing an identifier associated with a location, a corresponding planogram can be retrieved and used to generate a graphic representation of the planogram. The graphic representation of the planogram can be adjusted and superimposed onto a live or captured image of the location to indicate where objects should be placed or rearranged. Each object specified in the planogram can be visually represented as a picture superimposed in a position over an image or live view of the location captured by a client computing device by a user. The user can reference the graphic representation to stock, rearrange, or maintain the location.

Claims

1. A method comprising:

receiving, by a computer system, an image of a scene corresponding to a physical location captured by a camera on a client computing device and a time identifier indicating a time when the image of the scene of the physical location was captured by the camera on the client computing device;

receiving, by the computer system, information relating to a location and orientation of the client computing device at the time when the image of the scene of the physical location was captured based on data generated by one or more sensor devices on the client computing device;

receiving, by the computer system, an indication of a distance D between the physical location captured in the image and the location of the client computing device when the image was captured, the distance D measured by the one or more sensor devices that are operated on the client computing device;

calculating, by the computer system, the location of the scene of the physical location captured in the image at the time the image was captured based on (1) the distance D between the physical location captured in the image and the location of the client computing device at the time the image was captured and (2) the location and orientation of the client computing device relative to the physical location when the image was captured,

wherein the location of the scene of the physical location captured in the image is calculated by triangulating the location of the client computing device relative to the physical location captured in the image;

retrieving, by the computer system, a planogram associated with the location of the scene of the physical location captured in the image and the time identifier, wherein the planogram defines an arrangement of objects in the physical location during a time range that includes the time identifier and the scene is a section of the planogram;

generating, by the computer system, a graphic representation of the planogram, wherein the graphic representation of the planogram comprises images of the objects positioned relative to the image of the scene of the physical location; and

transmitting, from the computer system, the graphic representation of the planogram to the client computing device for superimposing the section of the planogram onto the image of the scene of the physical location captured by the client computing device, wherein the graphic representation of the section of the planogram that is superimposed over the image of the scene changes as the image of the scene changes to a different section of the planogram.

2. The method of claim 1 further comprising receiving a location identifier by:

receiving the image of the scene of the physical location from the client computing device; and

analyzing the image of the scene of the physical location to determine the location identifier.

3. The method of claim 2 wherein analyzing the image of the scene of the physical location comprises:

locating a visual representation of the location identifier in the image of the scene of the physical location; and

decoding the visual representation of the location identifier to generate the location identifier.

4. The method of claim 1 wherein generating the graphic representation of the planogram comprises:

analyzing the image of the scene of the physical location to determine a section of the physical location captured in the image; and

generating the graphic representation of the section of the planogram based on the section of the physical location.

5. The method of claim 4 wherein analyzing the image of the scene of the physical location comprises analyzing sensor readings associated with image.

6. The method of claim 1 wherein retrieving the planogram comprises retrieving the images of the objects, and wherein generating the graphic representation of the planogram comprises adjusting the size of the images of the objects in response to the image of the scene of the physical location.

7. The method of claim 1 wherein generating the graphic representation of the planogram comprises:

comparing the image of the scene of the physical location to the graphic representation of the planogram to determine discrepancies between image of the scene of the physical location and the planogram; and

generating visual indications of the discrepancies in the graphic representation of the planogram.

8. The method of claim 1 further comprising:

receiving a second image of a sub-section of the scene of the physical location captured by the camera of the client computing device;

determining which part of the planogram corresponds to the sub-section of the scene of the physical location in the second image; and

referencing a portion of the planogram that corresponds to the sub-section of the scene of the physical location captured in the second image to generate a second graphic representation of the planogram.

9. A non-transitory computer readable storage medium comprising computer programmed instructions, which when executed by a computer processor, cause the computer processor to be configured for operations comprising:

receiving an image of a scene corresponding to a physical location captured by a camera on a client computing device and a time identifier indicating a time when the image of the scene of the physical location was captured by the camera on the client computing device;

receiving information relating to a location and orientation of the client computing device at the time when the image of the scene of the physical location was captured based on data generated by one or more sensor devices on the client computing device;

receiving an indication of a distance D between the physical location captured in the image and the location of the client computing device when the image was captured, the distance D measured by the one or more sensor devices that are operated on the client computing device;

calculating the location of the scene of the physical location captured in the image at the time the image was captured based on (1) the distance D between the physical location captured in the image and the location of the client computing device at the time the image was captured and (2) the location and orientation of the client computing device relative to the physical location when the image was captured,

wherein the location of the scene of the physical location captured in the image is calculated by triangulating the location of the client computing device relative to the physical location captured in the image;

retrieving a planogram associated with the location of the scene of the physical location captured in the image and the time identifier, wherein the planogram defines an arrangement of objects in the physical location during a time range that includes the time identifier and the scene is a section of the planogram;

generating a graphic representation of the planogram, wherein the graphic representation of the planogram comprises images of the objects positioned relative to the image of the scene of the physical location; and

transmitting the graphic representation of the planogram to the client computing device for superimposing the section of the planogram onto the image of the scene of the physical location captured by the client computing device, wherein the graphic representation of the section of the planogram that is superimposed over the image of the scene changes as the image of the scene changes to a different section of the planogram.

10. The non-transitory computer readable medium of claim 9 further comprising receiving a location identifier by:

receiving the image of the scene of the physical location from the client computing device; and

analyzing the image of the scene of the physical location to determine the location identifier.

11. The non-transitory computer readable medium of claim 10 wherein analyzing the image of the scene of the physical location comprises:

locating a visual representation of the location identifier in the image of the scene of the physical location; and

decoding the visual representation of the location identifier to generate the location identifier.

12. The non-transitory computer readable medium of claim 9 wherein generating the graphic representation of the planogram comprises:

analyzing the image of the scene of the physical location to determine a section of the physical location captured in the image; and

generating the graphic representation of the section of the planogram based on the section of the physical location.

13. The non-transitory computer readable medium of claim 9 wherein retrieving the planogram comprises retrieving the images of the objects, and wherein generating the graphic representation of the planogram comprises adjusting the size of the images of the objects in response to the image of the scene of the physical location.

14. The non-transitory computer readable medium of claim 9 wherein generating the graphic representation of the planogram comprises:

comparing the image of the scene of the physical location to the graphic representation of the planogram to determine discrepancies between image of the scene of the physical location and the planogram; and

generating visual indications of the discrepancies in the graphic representation of the planogram.

15. A system comprising:

a computer processor; and

a non-transitory computer readable medium coupled to the computer processor and comprising instructions, which when executed by the computer processor, cause the computer processor to be configured to;

receive an image of a scene corresponding to a physical location captured by a camera on a client computing device and a time identifier indicating a time when the image of the scene of the physical location was captured by the camera on the client computing device;

receive information relating to a location and orientation of the client computing device at the time when the image of the scene of the physical location was captured based on data generated by one or more sensor devices on the client computing device;

receive an indication of a distance D between the physical location captured in the image and the location of the client computing device when the image was captured, the distance D measured by the one or more sensor devices that are operated on the client computing device;

calculate the location of the scene of the physical location captured in the image at the time the image was captured based on (1) the distance D between the physical location captured in the image and the location of the client computing device at the time the image was captured and (2) the location and orientation of the client computing device relative to the physical location when the image was captured,

wherein the location of the scene of the physical location captured in the image is calculated by triangulating the location of the client computing device relative to the physical location captured in the image;

retrieve a planogram associated with the location of the scene of the physical location captured in the image and the time identifier, wherein the planogram defines an arrangement of objects in the physical location during a time range that includes the time identifier and the scene is a section of the planogram;

generate a graphic representation of the planogram, wherein the graphic representation of the planogram comprises images of the objects positioned relative to the image of the scene of the physical location; and

transmit the graphic representation of the planogram to the client computing device for superimposing the section of the planogram onto the image of the scene of the physical location captured by the client computing device, wherein the graphic representation of the section of the planogram that is superimposed over the image of the scene changes as the image of the scene changes to a different section of the planogram.

16. The system of claim 15 wherein the computer processor is further configured to receive a location identifier by:

receiving the image of the scene of the physical location from the client computing device; and

analyzing the image of the physical location to determine the location identifier.

17. The system of claim 16 wherein the code that causes the computer processor to be configured to analyze the image of the scene of the physical location further causes the computer processor to be configured to:

locate a visual representation of the location identifier in the image of the scene of the physical location; and

decode the visual representation of the location identifier to generate the location identifier.

18. The system of claim 15 wherein the code that causes the computer processor to be configured to generate the graphic representation of the planogram further causes the computer processor to be configured to:

analyze the image of the scene corresponding to the physical location to determine the section of the physical location captured in the image; and

generate the graphic representation of the section of the planogram based on the section of the physical location.

19. The system of claim 15 wherein the code that causes the computer processor to be configured to generate the graphic representation of the planogram further causes the computer processor to be configured to:

compare the image of the scene of the physical location to the graphic representation of the planogram to determine discrepancies between image of the physical location and the planogram; and

generate visual indications of the discrepancies in the graphic representation of the planogram.

20. A method comprising:

receiving, by a computer system, an image of a scene corresponding to a physical location captured by a camera on a mobile device and a time identifier indicating a time when the image was captured;

receiving, by the computer system, information relating to a location and orientation of the mobile device at the time when the image was captured based on data generated by one or more sensor devices on the mobile device;

receiving, by the computer system, a measurement of a distance between the physical location captured in the image and the location of the mobile device at the time when the image was captured, wherein the measurement of the distance is captured by the mobile device operating the one or more sensor devices to measure the distance, the one or more sensor devices including one or more of an infrared range finder, a near field communication (NFC) transceiver, and a global positioning system (GPS);

calculating, by the computer system, the location of the scene of the physical location captured in the image at the time the image was captured based on (1) the distance D between the physical location captured in the image and the location of the client computing device at the time the image was captured and (2) the location and orientation of the client computing device relative to the physical location when the image was captured,

wherein the location of the scene of the physical location captured in the image is calculated by triangulating the location of the client computing device relative to the physical location captured in the image;

retrieving, by the computer system, a planogram associated with the location of the scene of the physical location captured in the image and the time identifier, wherein the planogram describes an arrangement of objects in the physical location during a time range that includes the time identifier and the scene is a section of the planogram;

generating, by the computer system, a graphic representation of the planogram comprising images of the objects positioned relative to the image of the scene of the physical location; and

transmitting, by the computer system, the graphic representation of the planogram to the mobile device for display of the graphic representation of the section of the planogram over the image of the scene of the physical location captured by the mobile device, wherein the graphic representation of the section of the planogram that is displayed over the image of the scene changes as the image of the scene changes to a different section of the planogram.

21. The method of claim 20 further comprising:

tracking changes in an image stream of the scene of the physical location based on readings from the one or more sensor devices as a user pans or tilts the mobile device camera to scan different sections of the physical location; and

automatically updating the superimposed graphic representation of the planogram changes as portions of the physical location displayed in a viewfinder of the mobile device camera change with user movement of the mobile device camera.

22. The method of claim 20 further comprising associating readings from the one or more sensor devices with images in the image stream of the scene of the physical location.