Generation of two-dimensional and three-dimensional images of items for visual recognition in checkout apparatus

• US 10,540,551 B2
• Filed: 11/05/2018
• Issued: 01/21/2020
• Est. Priority Date: 01/30/2018
• Status: Active Grant

• Current Assignee: Mashgin, Inc.
• Original Assignee: Mashgin, Inc.
• Inventors: Srivastava, Abhinai, Dhankhar, Mukul, Li, Yong, Olson, Maxwell William
• Primary Examiner(s): Entezari, Michelle M
• Application Number:
  US16/180,838
• Publication Number:
  US 20190236362A1
• Time in Patent Office: 442 Days
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 20/201 Price look-up processing, e.g. updating
• CPC Class Codes: G06Q 20/208 Input by product or record sensing, e.g. weighing or scanner processing
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables

Abstract

Methods, systems, and computer programs are presented for capturing color images of items for a machine-learning model. In one method, a request, for adding an item to be visually recognized during sales transactions, is detected at a checkout apparatus. Further, the method includes capturing and storing a color image, by each of one or more cameras of the checkout apparatus, of an examination space when the item is present. Further, the capturing and storing of color images of the item is repeated at different poses until at a predetermined number of color images of the item are captured. Further, the method includes generating, by a processor of the checkout apparatus, additional images of the item by modifying the captured color images. A machine-learning model for item recognition is built by training a machine-learning tool with the captured images and the additional images.

Claims

1. A method comprising:

detecting, at a checkout apparatus a request for adding an item to be visually recognized during sales transactions;

capturing and storing a color image, by each of one or more cameras of the checkout apparatus, of an examination space when the item is present in the examination space, wherein a user is collocated with the checkout apparatus;

repeating the capturing and storing of color images of the item at different poses within the examination space until at least a predetermined threshold number of color images of the item are captured;

generating, by a hardware processor of the checkout apparatus, additional color images of the item by modifying the captured color images;

building a machine-learning model for item recognition, at the checkout apparatus, by training a machine-learning tool with the captured color images and the additional color images; and

rebuilding the machine-learning model with transaction data, wherein the transaction data comprises a transaction image of the item during operation, associated with an item identifier and a note, wherein the note comprises information about a purchase state.

2. The method of claim 1, comprising:

storing transaction data of items obtained during sales transactions;

wherein the machine-learning model is rebuilt with the captured color images and the additional color images.

3. The method of claim 1, wherein the transaction data for each sales transaction comprises color images captured by the one or more cameras, a 3D point cloud for the item, and a 3D mesh for the item.

4. The method of claim 1, comprising:

for each pose, generating a 3D point cloud of the item and a 3D mesh for the item; and

for each additional color image, generating a 3D point cloud associated with the additional color image and a 3D mesh associated with the additional color image, wherein training the machine-learning tool comprises utilizing the 3D point clouds and the 3D meshes for the poses and for the additional color images.

5. The method of claim 1, comprising:

configuring the checkout apparatus to recognize, by the machine-learning model, a purchase of the item during a sales transaction at the checkout apparatus based on color images of the examination space that include the item.

6. The method of claim 1, comprising:

before capturing the color image, prompting, on a display of the checkout apparatus, the user to place the item in an examination space of the checkout apparatus.

7. The method of claim 1, wherein generating the additional color images comprises modifying the captured color images by one or more of changing brightness, cropping, darkening, rotating, adding shadows, adding a partial block of the item, adding artificial shine, adding artificial lines, changing colors, or adding noise.

8. The method of claim 1, wherein generating the additional color images comprises modifying the captured color images by changing brightness, cropping, darkening, rotating, adding shadows, adding a partial block of the item, adding artificial shine, adding artificial lines, changing colors, and adding noise.

9. The method of claim 1, wherein the one or more cameras comprises a plurality of color three-dimensional (3D) cameras, and the hardware processor generates a 3D point cloud of the item by combining image data from the plurality of color 3D cameras.

10. The method of claim 1, wherein capturing and storing the color image comprises:

presenting a user interface for interacting with the user to capture the different poses of the item, the user interface comprising;

a message indicating a number of poses recorded;

a message indicating a recommended number of poses left; and

a selector to indicate when the item is ready for taking the pose;

detecting activation of the selector in the user interface; and

in response to the activation of the selector, capturing the color image, by each of one or more cameras of the checkout apparatus, of the examination space.

11. The method of claim 10, comprising:

updating the messages in the user interface after capturing the color image; and

presenting the color image captured by each of the one or more cameras in the user interface.

12. The method of claim 10, wherein the user interface comprises options for accessing control settings for the checkout apparatus, starting checkout transactions, starting training of new items, presenting checkout apparatus information, and performing a camera calibration operation.

13. The method of claim 10, wherein the user interface comprises a screen for presenting the captured color images of the item, the screen comprising an option for selecting an item for which the machine-learning model has already been trained and a window for presenting color images of the item.

14. The method of claim 1, wherein the machine-learning model is used to analyze a plurality of features for recognizing items, the plurality of features including color, size, and shape of the item.

15. The method of claim 14, wherein the plurality of features comprises any combination of color, size, shape, position, transaction history, geometry, and brightness of the item.

16. The method of claim 1, comprising:

presenting a message, on a display, prompting the user to place the item in the examination space in a new pose.

17. The method of claim 1, comprising:

sending, via a network, the captured color images of the item to a central database that stores image information captured by a plurality of checkout apparatus interconnected by the network.

18. The method of claim 1, comprising:

accessing, by another checkout apparatus, the captured color images and the additional color images from the checkout apparatus; and

building, at the other checkout apparatus, a machine-learning model at the other checkout apparatus for item recognition by training a machine-learning tool, at the other checkout apparatus, with the captured color images and the additional color images.

19. The method of claim 1, comprising:

accessing, by another checkout apparatus, the captured color images and the additional color images from the checkout apparatus;

accessing, by the other checkout apparatus, the transaction images of items obtained during sales transactions at the checkout apparatus; and

building, at the other checkout apparatus, a machine-learning model at the other checkout apparatus for item recognition by training a machine-learning tool, at the other checkout apparatus, with the captured color images, the additional color images, and the transaction images.

30. The method of claim 1, wherein the purchase state comprises one of a purchased state and a refunded state;

wherein the machine-learning model is rebuilt with transaction data associated with a purchased state.

20. A checkout apparatus comprising:

a display;

one or more cameras for capturing color images of an examination space of the checkout apparatus;

a memory comprising instructions; and

one or more processors, wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

detecting a request for adding an item to be visually recognized during sales transactions;

capturing and storing a color image, by each of one or more cameras of the checkout apparatus, of the examination space when the item is present in the examination space, wherein a user is collocated with the checkout apparatus;

repeating the capturing and storing of color images of the item at different poses within the examination space until at least a predetermined threshold number of color images of the item are captured;

generating additional color images of the item by modifying the captured color images;

building a machine-learning model for item recognition, at the checkout apparatus, by training a machine-learning tool with the captured color images and the additional color images; and

rebuilding the machine-learning model with transaction data generated during sales transactions, wherein the transaction data comprises a transaction image of the item during operation, associated with an item identifier and a purchase state.

21. The checkout apparatus of claim 20, wherein the instructions cause the one or more processors to perform operations comprising:

storing transaction data of items obtained during sales transactions; and

rebuilding the machine-learning model with the captured color images and the additional color images.

22. The checkout apparatus of claim 21, wherein the transaction data for each sales transaction comprises color images captured by the one or more cameras, a 3D point cloud for the item, and a 3D mesh for the item.

23. The checkout apparatus of claim 20, wherein the instructions cause the one or more processors to perform operations comprising:

for each pose, generating a 3D point cloud of the item and a 3D mesh for the item; and

for each additional color image, generating a 3D point cloud associated with the additional color image and a 3D mesh associated with the additional color image, wherein training the machine-learning tool comprises utilizing the 3D point clouds and the 3D meshes for the poses and for the additional color images.

24. The checkout apparatus of claim 20, comprising receiving an instruction to enter a sales mode where the checkout apparatus recognizes, by the machine-learning model, a purchase of the item during a sales transaction at the checkout apparatus based on color images of the examination space that include the item.

25. The checkout apparatus of claim 20, wherein generating additional color images comprises modifying the captured color images by one or more of changing brightness, cropping, darkening, rotating, adding shadows, adding a partial block of the item, adding artificial shine, adding artificial lines, changing colors, or adding noise.

26. The checkout apparatus of claim 20, wherein generating the additional color images comprises modifying the captured color images by changing brightness, cropping, darkening, rotating, adding shadows, adding a partial block of the item, adding artificial shine, adding artificial lines, changing colors, and adding noise.

27. The checkout apparatus of claim 20, wherein the one or more cameras comprise a plurality of color three-dimensional (3D) cameras, and the one or more processors generate a 3D point cloud of the item by combining image data from the plurality of color 3D cameras.

28. The checkout apparatus of claim 20, wherein the instructions cause the one or more processors to perform operations comprising:

presenting, on a display of the checkout apparatus, a user interface for displaying the captured color images, a number of poses captured of the item, and a selector to indicate when the item is ready for the image capturing.

29. A non-transitory machine-readable storage medium comprising instructions that, when executed by a machine, cause the machine to perform operations comprising:

detecting, at a checkout apparatus, a request for adding an item to be visually recognized during sales transactions;

capturing and storing a color image, by each of one or more cameras of the checkout apparatus, of an examination space when the item is present in the examination space, wherein a user is collocated with the checkout apparatus;

repeating the capturing and storing of color images of the item at different poses within the examination space until at least a predetermined threshold number of color images of the item are captured;

generating, by a hardware processor of the checkout apparatus, additional color images of the item by modifying the captured color images;

building a machine-learning model for item recognition, at the checkout apparatus, by training a machine-learning tool with the captured color images and the additional color images; and

rebuilding the machine-learning model with transaction data generated during sales transactions, wherein the transaction data comprises a transaction image of the item during operation, associated with an item identifier and a purchase state for the respective sales transaction.