Synchronization of image data from multiple three-dimensional cameras for image recognition

• US 10,467,454 B2
• Filed: 04/26/2017
• Issued: 11/05/2019
• Est. Priority Date: 04/26/2017
• Status: Active Grant

• Current Assignee: Mashgin, Inc.
• Original Assignee: Mashgin, Inc.
• Inventors: Srivastava, Abhinai, Dhankhar, Mukul
• Primary Examiner(s): Abaza, Ayman A
• Application Number:
  US15/497,730
• Publication Number:
  US 20180314877A1
• Time in Patent Office: 923 Days
• Field of Search: 348 48
• CPC Class Codes: G01B 11/245 using a plurality of fixed, simultaneously operating transducers G01B11/2408 - G01B11/2425, G01B11/255 take precedence
• CPC Class Codes: G01B 2210/52 Combining or merging partially overlapping images to an overall image
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06Q 20/208 Input by product or record sensing, e.g. weighing or scanner processing
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 2201/121 using special illumination
• CPC Class Codes: G07G 1/0063 with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details

Abstract

Methods, systems, and computer programs are presented for object recognition performed by electronic devices. One method includes an operation for capturing three-dimensional (3D) images of a region over a surface using 3D cameras, the surface having a pattern and each 3D camera defining a respective camera coordinate system. For each camera, the 3D image is analyzed to identify a location of the pattern indicating an origin of a common coordinate system, and a coordinate transformation function is defined to convert data to the common coordinate system. Each 3D camera captures a 3D object image of an object on the surface that includes 3D object data. The 3D object data is transformed to the common coordinate system to obtain transformed 3D object data. The 3D object data is combined to obtain a composite 3D object data, and object recognition of the object is performed based on the composite 3D object data.

Claims

1. A method comprising, during an active session:

capturing, by a plurality of three-dimensional (3D) cameras in a top section of a checkout apparatus, 3D images of a region over a surface of a base in the checkout apparatus, the surface comprising a pattern, and wherein the checkout apparatus comprises a post section connecting the base to the top section in a spaced relationship, each 3D camera from the plurality of 3D cameras defining a camera coordinate system, each 3D image comprising pixel data having three coordinates in the camera coordinate system, wherein the 3D images comprise 3D pixel object data of an object on the surface and the pattern;

for each 3D camera, analyzing, by one or more processors, the 3D image to determine a location of the pattern relative to the respective 3D camera, wherein the pattern is indicates a common 3D coordinate system shared by the 3D cameras;

for each 3D camera, calibrating the respective 3D camera during the active session, comprising determining a coordinate transformation function to convert pixel data in the 3D image from the camera coordinate system of the 3D camera to the common 3D coordinate system shared by the 3D cameras, the coordinate transformation function being determined based on the identified location of the pattern in the respective 3D image and the common 3D coordinate system;

for the captured 3D images, transforming, by the one or more processors, the 3D pixel object data to the common 3D coordinate system, using the coordinate transformation function, to obtain transformed 3D pixel object data, wherein the transformed 3D pixel object data for the plurality of 3D cameras is defined for the same common 3D coordinate system;

combining, by the one or more processors, the transformed 3D pixel object data from the captured 3D images to obtain a composite 3D pixel object data for the object; and

performing, by the one or more processors, object recognition of the object on the surface based on an appearance of the object described in the composite 3D pixel object data.

2. The method as recited in claim 1, wherein the camera coordinate system includes an origin at a position of the 3D camera, x, y, and z axes, the z axis pointing in a direction of a view by the 3D camera, wherein the common 3D coordinate system includes a common origin on a point of the surface and a common z axis perpendicular to the surface.

3. The method as recited in claim 1, wherein the coordinate transformation function includes a coordinate change a location of a pixel in the camera coordinate system to a location of the pixel in the common 3D coordinate system.

4. The method as recited in claim 1, wherein combining the transformed 3D pixel object data further comprises:

combining the transformed 3D object pixel data from the captured 3D images to obtain the composite 3D pixel object data that defines pixel information and 3D location of each pixel for the object, the combining being based on the coordinates of the transformed 3D object pixel data provided by each 3D camera.

5. The method as recited in claim 1, further comprising:

removing, from the composite 3D pixel data, pixels of the surface before performing the object recognition.

6. The method as recited in claim 1, wherein the pattern includes a plurality of circles arranged on a grid pattern.

7. The method as recited in claim 1, wherein the plurality of 3D cameras includes one or more 3D stereo cameras and one or more structured light imaging cameras.

8. The method as recited in claim 1, further comprising:

for each 3D camera, defining error correction for captured images based on the captured 3D image and the location of the pattern.

9. The method as recited in claim 1, wherein the object recognition is performed by a machine learning algorithm based on the composite 3D pixel object data, the machine learning algorithm being trained with 3D models of a plurality of objects, the 3D models having a plurality of features that comprise shape, size, longest dimension, and color.

10. The method as recited in claim 1, wherein the data from the 3D image comprises a plurality of pixels, each pixel having corresponding values for x coordinate, y coordinate, z coordinate, red color value, green color value, and blue color value.

21. The method as recited in claim 1, wherein the object is in use on a cafeteria tray placed on the surface, wherein the 3D images of the object for object recognition are captured while the cafeteria tray covers the pattern.

22. The method as recited in claim 1, wherein determining the coordinate transformation function comprises:

determining a location of each 3D camera with reference to the common 3D coordinate system.

23. The method as recited in claim 1, further comprising:

identifying multiple food items on the surface;

capturing 3D images of the plurality of food items; and

separating the plurality of food items based on 3D pixel information to identify collections of 3D pixels not touching other collections of 3D pixels in the 3D space.

11. A system comprising:

a base having a surface, the surface comprising a pattern;

a top section;

a post section supporting the top section in a spaced relationship relative to the base;

a plurality of three-dimensional (3D) cameras, in the top section, for capturing 3D images of a region over the surface and below the top section, each 3D camera from the plurality of 3D cameras defining a camera coordinate system, wherein each 3D image comprises 3D pixel object data of an object on the surface and the pattern;

a memory comprising instructions; and

one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising, during an active session;

for each 3D camera, analyzing the 3D image to determine a location of the pattern that indicates an origin of a common 3D coordinate system shared by the 3D cameras;

for each 3D camera, calibrating the respective 3D camera during the active session, comprising determining a coordinate transformation function to convert pixel data in the 3D image from the camera coordinate system of the 3D camera to the common 3D coordinate system shared by the 3D cameras, the coordinate transformation function being determined based on a location of the origin relative to the respective 3D camera, wherein the location of the origin is determined based on the identified location of the pattern in the 3D image;

for the captured 3D images, transforming the 3D pixel object data to the common 3D coordinate system, using the coordinate transformation function, to obtain transformed 3D pixel object data, wherein the transformed 3D pixel object data for the plurality of 3D cameras is defined for the same common 3D coordinate system;

combining the transformed 3D pixel object data from the captured 3D images to obtain a composite 3D pixel object data for the object; and

performing object recognition of the object on the surface based on an appearance of the object described in the composite 3D pixel object data.

12. The system as recited in claim 11, wherein the camera coordinate system includes an origin at a position of the 3D camera, x, y, and z axes, the z axis pointing in a direction of a view by the 3D camera, wherein the common 3D coordinate system includes a common origin on a point of the surface and a common z axis perpendicular to the surface, wherein the coordinate transformation function includes a coordinate change of the pixel data from the camera coordinate system to the common 3D coordinate system.

13. The system as recited in claim 11, wherein the coordinate transformation function includes a coordinate change a location of a pixel in the camera coordinate system to a location of the pixel in the common 3D coordinate system.

14. The system as recited in claim 11, wherein combining the transformed 3D pixel object data further comprises:

combining the transformed 3D object pixel data from the captured 3D images to obtain the composite 3D pixel object data that defines pixel information and 3D location of each pixel for the object, the combining being based on the coordinates of the transformed 3D object pixel data provided by each 3D camera.

15. The system as recited in claim 11, wherein the instructions further cause the one or more computer processors to perform operations comprising:

removing, from the composite 3D pixel data, pixels of the surface before performing the object recognition.

16. A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising, during an active session:

capturing, by a plurality of three-dimensional (3D) cameras in a top section of a checkout apparatus, 3D images of a region over a surface of a base in the checkout apparatus, the surface comprising a pattern, wherein the checkout apparatus comprises a post section connecting the base to the top section in a spaced relationship, each 3D camera from the plurality of 3D cameras defining a camera coordinate system, each 3D image comprising pixel data having three coordinates in the camera coordinate system, wherein the 3D images comprise 3D pixel object data of an object on the surface and the pattern;

for each 3D camera, analyzing, by one or more processors, the 3D image to determine a location of the pattern relative to the respective 3D camera, wherein the pattern is associated with a common 3D coordinate system shared by the 3D cameras;

for each 3D camera, calibrating the respective 3D camera during the active session, comprising determining a coordinate transformation function to convert pixel data in the 3D image from the camera coordinate system of the 3D camera to the common 3D coordinate system shared by the 3D cameras, the coordinate transformation function being determined based on the identified pattern in the respective 3D image and the common 3D coordinate system;

for the captured 3D images, transforming, by the one or more processors, the 3D pixel object data to the common 3D coordinate system, using the coordinate transformation function, to obtain transformed 3D pixel object data, wherein the transformed 3D pixel object data for the plurality of 3D cameras is defined for the same common 3D coordinate system;

combining, by the one or more processors, the transformed 3D pixel object data from the captured 3D images to obtain a composite 3D pixel object data for the object; and

performing, by the one or more processors, object recognition of the object on the surface based on an appearance of the object described in the composite 3D pixel object data.

17. The machine-readable storage medium as recited in claim 16, wherein the camera coordinate system includes an origin at a position of the 3D camera, x, y, and z axes, the z axis pointing in a direction of a view by the 3D camera, wherein the common 3D coordinate system includes a common origin on a point of the surface and a common z axis perpendicular to the surface, wherein the coordinate transformation function includes a coordinate change of the pixel data from the camera coordinate system to the common 3D coordinate system.

18. The machine-readable storage medium as recited in claim 16, wherein the coordinate transformation function includes a coordinate change a location of a pixel in the camera coordinate system to a location of the pixel in the common 3D coordinate system.

19. The machine-readable storage medium as recited in claim 16, wherein combining the transformed 3D pixel object data further comprises:

combining the transformed 3D object pixel data from the captured 3D images to obtain the composite 3D pixel object data that defines pixel information and 3D location of each pixel for the object, the combining being based on the coordinates of the transformed 3D object pixel data provided by each 3D camera.

20. The machine-readable storage medium as recited in claim 16, wherein the machine further performs operations comprising:

removing, from the composite 3D pixel data, pixels of the surface before performing the object recognition.