Image recognition scoring visualization

• US 10,339,690 B2
• Filed: 12/18/2015
• Issued: 07/02/2019
• Est. Priority Date: 12/18/2015
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Findlay, Roland, Griffin, Michael, Romain, Tiffany, Graham, Jamey
• Primary Examiner(s): Lu, Hua
• Application Number:
  US14/975,648
• Publication Number:
  US 20170177195A1
• Time in Patent Office: 1,292 Days
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A system and method for image processing result visualization. The method includes receiving a job, wherein the job includes a panoramic image and image recognition information, and creating a visualization based on the job, wherein the visualization includes the panoramic image with the image recognition information layered over the panoramic image.

Claims

1. A method comprising:

receiving a request to view a job in a graphical user interface;

requesting, using one or more processors, the job by sending a token related to the job to a server storing data associated with the job via a network;

receiving, using the one or more processors, the job related to the token, wherein the job includes a panoramic image and metadata;

processing, using the one or more processors, the metadata to extract image recognition information, the image recognition information comprising features of a plurality of objects depicted within the panoramic image;

creating, using the one or more processors, a visualization of the job, wherein the visualization includes the panoramic image with the image recognition information overlaid on the panoramic image to highlight an object of the plurality of objects;

determining, using the one or more processors, an identity of the object of the plurality of objects depicted within the panoramic image by matching the features of the object of the plurality of objects with a potential candidate in a product database that has a highest recognition score;

determining, using the one or more processors, a correctness of the determined identity of the object of the plurality of objects based on the matching;

updating, using the one or more processors, the visualization of the job by removing portions of the visualization that do not include the plurality of objects; and

displaying in the graphical user interface, the updated visualization of the job and a correctness indicator signaling the correctness of the object of the plurality of objects.

2. The method of claim 1 wherein the object of the plurality of objects is a recognized product and creating the visualization of the job further comprises:

highlighting, using the one or more processors, the recognized product in the panoramic image using the image recognition information; and

displaying, using the one or more processors, a product recognition list in response to a user selecting the recognized product.

3. The method of claim 2, comprising, creating the product recognition list by:

extracting, using the one or more processors, a listing of potential candidates that share similar features with the recognized product from the image recognition information; and

displaying, using the one or more processors, the listing of potential candidates.

4. The method of claim 2, wherein an entry in the product recognition list includes a recognition score for a corresponding potential candidate, the recognition score indicating similarity of a feature shared between the recognized product and the corresponding potential candidate.

5. The method of claim 2, wherein the correctness indicator further comprises:

a correct visual indicator in the visualization proximate to the object of the plurality of objects responsive to determining that the identity of the object of the plurality of objects is correctly identified;

or

an incorrect visual indicator in the visualization proximate to the object of the plurality of objects responsive to determining that the identity of the object of the plurality of objects is incorrectly identified.

6. The method of claim 1, wherein the plurality of objects are recognized products, the method further comprising:

highlighting, using the one or more processors, a portion of the panoramic image based on the recognized products sharing a characteristic.

7. The method of claim 1, wherein the plurality of objects are recognized products, the method further comprising:

displaying, using the one or more processors, a boundary around the recognized products in the panoramic image.

8. The method of claim 1, wherein the visualization is a first visualization of a first job, the method further comprising:

creating, using the one or more processors, a second visualization of a second job;

comparing, using the one or more processors, the first visualization with the second visualization in a single display screen of a user interface; and

highlighting, using the one or more processors, differences between the first visualization and the second visualization.

9. The method of claim 8, wherein comparing the first visualization with the second visualization further comprises displaying the first visualization and the second visualization in an animation, wherein the animation toggles between the first visualization and the second visualization.

10. A non-transitory computer program product comprising a computer usable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive, using one or more processors, a request to view a job in a graphical user interface;

request, using the one or more processors, the job by sending a token related to the job to a server storing data associated with the job via a network;

receive, using the one or more processors, the job related to the token, wherein the job includes a panoramic image and metadata;

process, using the one or more processors, the metadata to extract image recognition information, the image recognition information comprising features of a plurality of objects depicted within the panoramic image;

create, using the one or more processors, a visualization of the job, wherein the visualization includes the panoramic image with the image recognition information overlaid on the panoramic image to highlight an object of the plurality of objects;

determine, using the one or more processors, an identity of the object of the plurality of objects depicted within the panoramic image by matching the features of the object of the plurality of objects with a potential candidate in a product database that has a highest recognition score;

determine a correctness of the determined identity of the object of the plurality of objects based on the matching;

update, using the one or more processors, the visualization of the job by removing portions of the visualization that do not include the plurality of objects; and

display in the graphical user interface, the updated visualization of the job and a correctness indicator signaling the correctness of the object of the plurality of objects.

11. The non-transitory computer program product of claim 10 wherein the object of the plurality of objects is a recognized product and the computer readable program causing the computer to create the visualization further causes the computer to:

highlight, using the one or more processors, the recognized product in the panoramic image using the image recognition information; and

display, using the one or more processors, a product recognition list in response to a user selecting the recognized product.

12. The non-transitory computer program product of claim 11, wherein the computer readable program when executed on a computer further causes the computer to create the product recognition list to:

extract, using the one or more processors, a listing of potential candidates that share similar features with the recognized product from the image recognition information; and

display, using the one or more processors, the listing of potential candidates.

13. The non-transitory computer program product of claim 11, wherein the correctness indicator further comprises:

a correct visual indicator displayed in the visualization proximate to the object of the plurality of objects responsive to determining that the identity of the object of the plurality of objects is correctly identified;

or

an incorrect visual indicator displayed in the visualization proximate to the object of the plurality of objects responsive to determining that the identity of the object of the plurality of objects is incorrectly identified.

14. The non-transitory computer program product of claim 10, wherein the visualization is a first visualization of a first job, wherein the computer readable program when executed on a computer further causes the computer to:

creating, using the one or more processors, a second visualization of a second job;

compare, using the one or more processors, the first visualization with the second visualization in a single display screen of a user interface; and

highlight, using the one or more processors, differences between the first visualization and the second visualization.

15. A system comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the system to;

receive a request to view a job in a graphical user interface;

request the job by sending a token related to the job to a server storing data associated with the job via a network;

receive the job related to the token, wherein the job includes a panoramic image and metadata;

process the metadata to extract image recognition information, the image recognition information comprising features of a plurality of objects depicted within the panoramic image;

create a visualization of the job, wherein the visualization includes the panoramic image with the image recognition information overlaid on the panoramic image to highlight an object of the plurality of objects;

determine an identity of the object of the plurality of objects depicted within the panoramic image by matching the features of the object of the plurality of objects with a potential candidate in a product database that has a highest recommendation score;

determine a correctness of the determined identity of the object of the plurality of objects based on the matching;

update the visualization of the job by removing portions of the visualization that do not include the plurality of objects; and

display, in the graphical user interface, the updated visualization of the job and a correctness indicator signaling the correctness of the object of the plurality of objects.

16. The system of claim 15 wherein the object of the plurality of objects is a recognized product and wherein to create the visualization of the job, the instructions cause the system to:

highlight the recognized product in the panoramic image using the image recognition information; and

display a product recognition list in response to a user selecting the recognized product.

17. The system of claim 16, wherein to create the product recognition list, the instructions cause the system to:

extract a listing of potential candidates that share similar features with the recognized product from the image recognition information; and

display the listing of potential candidates.

18. The system of claim 16, wherein an entry in the product recognition list includes a recognition score for a corresponding potential candidate, the recognition score indicating similarity of a feature shared between the recognized product and the corresponding potential candidate.

19. The system of claim 15, wherein the correctness indicator comprises:

a correct visual indicator displayed in the visualization proximate to the object of the plurality of objects responsive to determining that the identity of the object of the plurality of objects is correctly identified;

or

an incorrect visual indicator displayed in the visualization proximate to the object of the plurality of objects responsive to determining that the identity of the object of the plurality of objects is incorrectly identified.

20. The system of claim 15, wherein the visualization is a first visualization of a first job, and the instructions cause the system to:

create a second visualization of a second job;

compare the first visualization with the second visualization in a single display screen of a user interface; and

highlight differences between the first visualization and the second visualization.