SYNCHRONIZATION OF IMAGE DATA FROM MULTIPLE THREE-DIMENSIONAL CAMERAS FOR IMAGE RECOGNITION

• US 20180314877A1
• Filed: 04/26/2017
• Published: 11/01/2018
• Est. Priority Date: 04/26/2017
• Status: Active Grant

• Current Assignee: Mashgin, Inc.
• Original Assignee: Mashgin, Inc.
• Inventors: Srivastava, Abhinai, Dhankhar, Mukul
• Granted Patent: US 10,467,454 B2
• Time in Patent Office: Days
• CPC Class Codes: G01B 11/245 using a plurality of fixed, simultaneously operating transducers G01B11/2408 - G01B11/2425, G01B11/255 take precedence
• CPC Class Codes: G01B 2210/52 Combining or merging partially overlapping images to an overall image
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06Q 20/208 Input by product or record sensing, e.g. weighing or scanner processing
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 2201/121 using special illumination
• CPC Class Codes: G07G 1/0063 with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details

Abstract

Methods, systems, and computer programs are presented for object recognition performed by electronic devices. One method includes an operation for capturing three-dimensional (3D) images of a region over a surface using 3D cameras, the surface having a pattern and each 3D camera defining a respective camera coordinate system. For each camera, the 3D image is analyzed to identify a location of the pattern indicating an origin of a common coordinate system, and a coordinate transformation function is defined to convert data to the common coordinate system. Each 3D camera captures a 3D object image of an object on the surface that includes 3D object data. Further, the 3D object data is transformed to the common coordinate system to obtain transformed 3D object data. The 3D object data is combined to obtain a composite 3D object data, and object recognition of the object is performed based on the composite 3D object data.

Claims

1. A method comprising:

capturing three-dimensional (3D) images of a region over a surface by a plurality of 3D cameras, the surface having a pattern, each 3D camera from the plurality of 3D cameras defining a camera coordinate system;

for each camera, analyzing, by one or more processors, the 3D image to identify a location of the pattern that indicates an origin of a common coordinate system shared by the 3D cameras;

for each camera, defining a coordinate transformation function to convert data from the 3D image to the common coordinate system;

capturing, by each of the plurality of 3D cameras, a 3D object image of an object on the surface, the 3D object image comprising 3D object data;

for the captured 3D object images, transforming, by the one or more processors, the 3D object data to the common coordinate system to obtain transformed 3D object data;

combining, by the one or more processors, the transformed 3D object data from the captured 3D object images to obtain a composite 3D object data; and

performing, by the one or more processors, object recognition of the object on the surface based on the composite 3D object data.

2. The method as recited in claim 1, wherein the camera coordinate system includes an origin at a position of the camera, x, y, and z axes, the z axis pointing in a direction of a view by the camera, wherein the common coordinate system includes a common origin on a point of the surface and a common z axis perpendicular to the surface.

3. The method as recited in claim 1, wherein the coordinate transformation function transforms a location of a pixel in the camera coordinate system to a location of the pixel in the common coordinate system.

4. The method as recited in claim 1, wherein combining the transformed 3D object data further comprises:

combining pixel data from the captured 3D object images to obtain the composite 3D object data that defines pixel information and location of each pixel for the object.

5. The method as recited in claim 4, wherein combining the transformed 3D object data further comprises:

removing pixels from a background that includes the surface.

6. The method as recited in claim 1, wherein the pattern includes a plurality of circles arranged on a grid pattern.

7. The method as recited in claim 1, wherein the plurality of 3D cameras includes one or more 3D stereo cameras and one or more structured light imaging cameras.

8. The method as recited in claim 1, further comprising:

for each camera, defining error correction for captured images based on the captured 3D image and the location of the pattern.

9. The method as recited in claim 1, wherein the object recognition is performed by a machine learning program based on the composite 3D object data.

10. The method as recited in claim 1, wherein the data from the 3D image comprises a plurality of pixels, each pixel having corresponding values for x coordinate, y coordinate, z coordinate, red color value, green color value, and blue color value.

11. A system comprising:

a surface having a pattern;

a plurality of three-dimensional (3D) cameras for capturing 3D images of a region over the surface, each 3D camera from the plurality of 3D cameras defining a camera coordinate system;

a memory comprising instructions; and

one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising;

for each camera, analyzing the 3D image to identify a location of the pattern that indicates an origin of a common coordinate system shared by the 3D cameras;

for each camera, defining a coordinate transformation function to convert data from the 3D image to the common coordinate system, wherein each of the plurality of 3D cameras are configured for capturing a 3D object image of an object on the surface, the 3D object image comprising 3D object data;

for the captured 3D object images, transforming the 3D object data to the common coordinate system to obtain transformed 3D object data;

combining the transformed 3D object data from the captured 3D object images to obtain a composite 3D object data; and

performing object recognition of the object on the surface based on the composite 3D object data.

12. The system as recited in claim 11, wherein the camera coordinate system includes an origin at a position of the camera, x, y, and z axes, the z axis pointing in a direction of a view by the camera, wherein the common coordinate system includes a common origin on a point of the surface and a common z axis perpendicular to the surface.

13. The system as recited in claim 11, wherein the coordinate transformation function transforms a location of a pixel in the camera coordinate system to a location of the pixel in the common coordinate system.

14. The system as recited in claim 11, wherein combining the transformed 3D object data further comprises:

combining pixel data from the captured 3D object images to obtain the composite 3D object data that defines pixel information and location of each pixel for the object.

15. The system as recited in claim 14, wherein combining the transformed 3D object data further comprises:

removing pixels from a background that includes the surface.

16. A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:

capturing three-dimensional (3D) images of a region over a surface by a plurality of 3D cameras, the surface having a pattern, each 3D camera from the plurality of 3D cameras defining a camera coordinate system;

for each camera, analyzing the 3D image to identify a location of the pattern that indicates an origin of a common coordinate system shared by the 3D cameras;

for each camera, defining a coordinate transformation function to convert data from the 3D image to the common coordinate system;

capturing, by each of the plurality of 3D cameras, a 3D object image of an object on the surface, the 3D object image comprising 3D object data;

for the captured 3D object images, transforming the 3D object data to the common coordinate system to obtain transformed 3D object data;

combining the transformed 3D object data from the captured 3D object images to obtain a composite 3D object data; and

performing object recognition of the object on the surface based on the composite 3D object data.

17. The machine-readable storage medium as recited in claim 16, wherein the camera coordinate system includes an origin at a position of the camera, x, y, and z axes, the z axis pointing in a direction of a view by the camera, wherein the common coordinate system includes a common origin on a point of the surface and a common z axis perpendicular to the surface.

18. The machine-readable storage medium as recited in claim 16, wherein the coordinate transformation function transforms a location of a pixel in the camera coordinate system to a location of the pixel in the common coordinate system.

19. The machine-readable storage medium as recited in claim 16, wherein combining the transformed 3D object data further comprises:

combining pixel data from the captured 3D object images to obtain the composite 3D object data that defines pixel information and location of each pixel for the object.

20. The machine-readable storage medium as recited in claim 19, wherein combining the transformed 3D object data further comprises:

removing pixels from a background that includes the surface.