Systems and methods for tracking items

• US 10,685,197 B2
• Filed: 11/19/2018
• Issued: 06/16/2020
• Est. Priority Date: 11/17/2017
• Status: Active Grant

• Current Assignee: Divine Logic, Inc.
• Original Assignee: Divine Logic, Inc.
• Inventors: Plummer, Stephen, Plummer, Alexander, Lundberg, Brandon
• Primary Examiner(s): Lee, Seung H
• Application Number:
  US16/194,764
• Publication Number:
  US 20190156086A1
• Time in Patent Office: 575 Days
• CPC Class Codes: G06K 7/10722 Photodetector array or CCD scanning
• CPC Class Codes: G06K 7/1413 1D bar codes
• CPC Class Codes: G06K 7/1417 2D bar codes
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

The present invention provides systems and methods for tracking items (e.g., commodities, goods, containers, boxes, packages, etc.) through transportations to multiple locations within a pre-defined space, to allow the position(s) and movement(s) of such items to be accurately tracked and documented, and to allow such items to be quickly identified and located based on tracking records kept within the tracking system. The system may utilize image sensors, image recognition and processes software, position translation software, and a virtual model of the pre-defined space in order to track objects within the defined space and maintain a record of the movement(s) and position(s) of the object within the pre-defined space.

Claims

1. A system for tracking the position of objects within a predefined space, comprising:

a. a plurality of unique static markers, each comprising a machine-readable code, arranged in predetermined locations within the predefined space;

b. a plurality of unique object markers positioned on each of said objects;

c. an electronic image acquisition device having a machine vision system, the machine vision system comprising an image sensor and image capture electronics, for acquiring images of said unique static markers and said object markers; and

d. an image processing system for analyzing pixels in the acquired image to determine an identity of each static marker and object marker in said acquired image, retrieve location data from a database for each static marker in said acquired image, and determining a position of each static marker and each object marker relative to the image acquisition device,

wherein said image processing system is operable to determine a relative position of said image acquisition device to a plurality of static markers by;

determining the position and orientation of said image acquisition device relative to that of a first static marker by performing a pose estimation calculation,

calculating the position and orientation of the image acquisition device relative to the predefined space based on a location data of said first static marker retrieved from said database, a pose estimation calculation for said first static marker, and an estimation of the distance of the first static marker from the image acquisition device based on a size of said first static marker in said image,

determining the position and orientation of said image acquisition device relative to that of a second static marker by performing a second pose estimation calculation,

calculating the position and orientation of the image acquisition device relative to the predefined space based on a location data of said second static marker retrieved from said database, said second pose estimation calculation for said second static marker, and an estimation of the distance of the second static marker from the image acquisition device based on a size of said second static marker in said image, and

performing an averaging calculation to provide multiple calculations of the position of the image acquisition device to provide an averaged estimate of the location.

2. The system of claim 1, further comprising a machine-readable memory for storing a digital model of said predefined space, positions of said static markers, positions of said object markers, and identification data regarding said objects.

3. The system of claim 2, wherein said image processing system is operable to determine orientation of each said static marker relative to the image acquisition device by performing a unwarping process to compensate for skew in a view of said static marker provided in said image and generate a plan view of said static marker, and analyze optical features within said plan view of said static marker to read coding provided in said optical features and retrieve identification data from said coding.

4. The system of claim 3, wherein said image processing system is operable to compare said identification data to records of each of said plurality of static markers stored in said machine-readable memory to identify a matching record for said static marker and retrieve location data of said static marker from a record for said static marker.

5. The system of claim 2, wherein said image processing system is operable to analyze optical features within each said object marker in said image to determine an orientation of said object marker relative to the image acquisition device by performing a unwarping process to compensate for skew in a view of said object marker provided in said image and generate a plan view of said object marker.

6. The system of claim 5, wherein said image processing system is operable to analyze optical features within said plan view of said object marker to read coding provided in said optical features and retrieve identification data from said coding.

7. The system of claim 6, wherein said image processing system is operable to compare said identification data to records of each of said plurality of object markers stored in said machine-readable memory to identify a matching record for said object marker and retrieve location data of said object marker from a record for said object marker.

8. The system of claim 7, wherein said image processing system is operable to perform a pose estimation calculation that includes calculating rotation and translation vectors based on said optical features of said object marker to determine orientation of the object marker relative to the image acquisition device and the relative positions of said image acquisition device and said object marker.

9. The system of claim 8, wherein said image processing system is operable to determine the location of said object marker in said predefined space based on the pose estimation calculation for the object marker, a known position of said image acquisition device within said predefined space, and a perceived size of said object marker in said image.

10. The system of claim 9, wherein said image acquisition device is set as xyz origin within a coordinate system in the step of determining the location of said object marker in said predefined space.

11. The system of claim 1, wherein

a. said image acquisition device is movable through said predefined space and is operable to acquire images of the predefined space at regular intervals;

b. said image processing system is operable to analyze each of said images to identify each static marker and object marker therein, retrieve said location data from said database regarding a position of each said static marker and each said object marker within the predefined space, and provide an updated location of said image acquisition device based on the position of said image acquisition device relative to a position of one or more of said static markers in said image.

12. The system of claim 11, wherein said image processing system is operable to determine a location of said image acquisition device based on the position of said image acquisition device relative to a position of a single static marker in said image.

13. A method for tracking and retrieving an object in a predefined space, comprising:

a. placing a plurality of unique static markers, each comprising a machine-readable code, in said predefined space arranged in predetermined locations;

b. creating a database stored on a machine-readable memory, said database including records for each of the static markers, each of said records including location data of the corresponding static marker in said predefined space;

c. placing a unique object marker on each of a plurality of objects to be positioned in said predefined space;

d. generating a record for each object marker in said database that includes identification data of the object marker and data regarding the object on which it is positioned;

e. placing each said object in a storage position in said predefined space; and

f. recording the storage position in said record for said object marker positioned on said object;

g. capturing an image of a static marker using an image acquisition device, said image acquisition device having a machine vision system that includes an image sensor and image capture electronics, and an image processing system operable to analyze an image to identify static markers and object markers present in the image; and

h. calculating the position and orientation of the image acquisition device within the predefined space based on a pose estimation calculation for said static marker using an affine transformation of the image of the static marker and an estimation of the distance of the static marker from the image acquisition device estimated from a size of said static marker in said image.

14. The method of claim 13, wherein said recording of the storage position of said object includes capturing a digital image of said object marker using an electronic image acquisition device.

15. The method of claim 14, wherein said image acquisition device analyzes optical features within said static marker with the image processing system to determine orientation of said at least one static marker relative to the image acquisition device.

16. The method of claim 15, further comprising generating a plan view of said static marker and analyzing the optical features within said plan view of said static marker to read coding provided in said optical features and retrieve identification data from said coding.

17. The method of claim 16, further comprising comparing said identification data to said records of each of said static markers stored in said machine-readable memory to identify a matching record for said static marker and retrieve location data of said static marker from a record for said static marker.

18. The method of claim 17, wherein said pose estimation calculation includes calculating rotation and translation vectors based on said optical features of said static marker and said static marker is set as an xyz origin within a coordinate system.

19. The method of claim 14, further comprising translating the position and orientation of said image acquisition device into a digital representation of the image acquisition device in a digital model of said predefined space.

20. The method of claim 19, wherein said recording of the storage position of said object further comprises using said image processing system to determine a position of said object within the predefined space based on the determined position of said image acquisition device, a pose estimation for said object marker positioned on said object, and an estimation of the distance of the object marker from the image acquisition device based on a size of said object marker in said image.

21. The method of claim 20, wherein said pose estimation for said object marker to compensate for skew in a view of said object marker provided in said image and generate a plan view of said object marker using said image processing system.

22. The method of claim 21, further comprising analyzing optical features within said object marker to read coding provided in said optical features and retrieve identification data from said coding using said image processing system.

23. The method of claim 21, wherein said pose estimation calculation includes calculating rotation and translation vectors based on said optical features of said object marker to determine orientation of the object marker relative to the image acquisition device and the relative positions of said image acquisition device and said object marker.

24. The method of claim 23, further comprising determining the location of said object marker in said predefined space based on the pose estimation calculation for the object marker, a known position of said image acquisition device within said predefined space, and a perceived size of said object marker in said image, wherein said image acquisition device is set as xyz origin within a coordinate system in the step of determining the location of said object marker in said predefined space.

25. A system for tracking the position of objects within a predefined space, comprising:

a. a plurality of unique static markers, each comprising a machine-readable code, arranged in predetermined locations within the predefined space;

b. a plurality of unique object markers positioned on each of said objects;

c. an electronic image acquisition device having a machine vision system, the machine vision system comprising an image sensor and image capture electronics, for acquiring images of said unique static markers and said object markers; and

d. an image processing system for analyzing pixels in the acquired image to determine an identity of each static marker and object marker in said acquired image, retrieve location data from a database for each static marker in said acquired image, and determining a position of each static marker and each object marker relative to the image acquisition device, wherein said image processing system is operable to calculate the position and orientation of the image acquisition device within the predefined space based on a pose estimation calculation for a static marker using an affine transformation of an image of the static marker, an estimation of a distance of the static marker from the image acquisition device based on a size of said static marker in said image, and a known location of the image acquisition device.

26. The system of claim 25, wherein

a. said image acquisition device is movable through said predefined space and is operable to acquire images of the predefined space at regular intervals;

b. said image processing system is operable to analyze each of said images to identify each static marker and object marker therein, retrieve said location data from said database regarding a position of each said static marker and each said object marker within the predefined space, and provide an updated location of said image acquisition device based on the position of said image acquisition device relative to a position of one or more of said static markers in said image.

27. The system of claim 25, wherein said image processing system is operable to calculate the position and orientation of an object marker within the predefined space relative to the image acquisition device based on a pose estimation calculation of the object marker using an affine transformation of an image of the static marker and an estimation of a distance of the static marker from the image acquisition device based on a size of said static marker in said image, wherein the position of an object within the predefined space is determined based on the calculated position and orientation of the object marker relative to the image acquisition device using the position of the image acquisition device as an xyz axis within the predefined space.

28. The system of claim 27, wherein said image processing system is operable to generate a plan view of said object marker and analyze optical features within said plan view to read coding provided in said optical features and retrieve identification data from said coding.

29. The system of claim 28, wherein said image processing system is operable to compare said identification data to records of each of said plurality of object markers stored in said machine-readable memory to identify a matching record for said object marker and retrieve location data of said object marker from a record for said object marker.

30. The system of claim 27, wherein said pose estimation calculation includes calculating rotation and translation vectors based on said optical features of said object marker to determine orientation of the object marker relative to the image acquisition device and the relative positions of said image acquisition device and said object marker.