High accuracy localization system and method for retail store profiling via product image recognition and its corresponding dimension database

• US 9,928,438 B2
• Filed: 03/10/2016
• Issued: 03/27/2018
• Est. Priority Date: 03/10/2016
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Wu, Wencheng, Rizzolo, Charles D., Venable, Dennis L., Wade, Thomas F.
• Primary Examiner(s): SHAH, UTPAL D
• Application Number:
  US15/066,507
• Publication Number:
  US 20170262724A1
• Time in Patent Office: 747 Days
• Field of Search: None
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/757 Matching configurations of points or features

Abstract

A method for profiling a location of an image capture device in a region of interest. The method comprises acquiring an image captured by an image capture device. The method comprises acquiring a reported position and pose of the image capture device. The method comprises processing the captured image to detect an object in the captured image. The method comprises identifying a set of interest points characterizing the captured object. The method comprises generating a relative position and pose of the interest points based on dimensional information associated with the captured object. The method comprises computing an estimated position and pose of the image capture device to the object using the reported position and pose of the image capture device and the relative position of the interest points. The method comprises computing the estimated position and pose of the mobile imaging device based on the estimated distance. The method comprises updating the reported position and pose of the image capture device to the estimated position.

Claims

1. A profiling system for determining a location of an associated image capture device in a region of interest, the system comprising:

a processor; and

a non-transitory computer readable memory storing instructions that are executable by the processor to;

acquire an image captured by an associated image capture device;

acquire a reported position and pose of the associated image capture device;

process the captured image to detect an object in the captured image;

identify a set of interest points characterizing the captured object;

generate a relative position and pose of the interest points based on dimensional information associated with the captured object;

compute an estimated distance of the associated image capture device to the object using the reported position and pose of the associated image capture device and the relative position and pose of the interest points;

compute an estimated position and pose of the associated image capture device based on the estimated distance; and

update the reported position and pose of the associated image capture device to the estimated position and pose.

2. The system of claim 1, wherein the processor is further adapted to:

transmit a layout of a region of interest to an associated mobile imaging device carrying the associated image capture device; and

transmit a target position and pose to the associated mobile imaging device, wherein the target position and pose is located in the region of interest;

acquire the reported position and pose of the associated image capture device after the associated mobile imaging device moves the associated image capture device to the target position and pose.

3. The system of claim 2, wherein the region of interest is a store and the captured object is a product located on a display in the store.

4. The system of claim 1, wherein the processor is further adapted to:

access a database of stored reference images of reference objects, the database being in communication with the processor and associating dimensional information with each reference object;

search the stored images to match the captured object to the reference object;

in response to finding a match, determine the dimensional information associated with the reference object; and

associate the dimensional information with the captured object.

5. The system of claim 4, wherein the processor is further adapted to:

bound the reference object in a bounding box;

associate each corner of the bounding box with a relative position in real-world coordinates based on the dimensional information;

identify a set of reference markers in the reference image characterizing the reference object;

determine a relative reference position of each reference marker using at least one corner; and

,

compute a relative distance of the each reference marker to the at least one corner.

6. The system of claim 5, wherein the processor is adapted to:

assign a set of interest points as the set of reference markers, where the interest points are points detected on the reference image using a computer vision approach selected from a group consisting of;

Corner detectors;

Harris corner detector;

SIFT;

SURF;

LoG;

DoG;

DoH scale-space interest points;

SUSAN detector; and

a combination of the above.

7. The system of claim 6, wherein the processor is adapted to:

associate a first interest point in the captured image as a reference point; and

describe relative positions of the interest points using the equation;

(y_kl,z_kl)−

(y_k1,z_k1)=(β

_kl,γ

_kl),

wherein for N reference objects, M_l are the interest points corresponding to the captured object, M_l is the reference markers determined for k-th reference object; and

(x_kl, y_kl, z_kl) is a position of l-th interest point from k-th reference object, and the known value (β

_kl, γ

_kl) is derived based on the detected pixel locations of the interest point on the image of the reference object and its dimensional information.

8. The system of claim 6, wherein the processor is adapted to:

compute a relative pixel location of each one of the interest points of the captured object;

compute relative dimensions between the sets of interest points using the relative pixel locations;

compute coordinates of the associated image capture device to the captured object using the relative dimensions; and

,

compute the estimated position and pose of the associated mobile imaging device based on the computed coordinates.

9. The system of claim 8, wherein the processor is adapted to:

compute an expected relative position of each interest point using the equation;

(x_kl,y_kl,z_kl)−

(x_k1,y_k1,z_k1)=H(i_kl,j_kl;

d^E,θ

^E)−

H(i_k1,j_k1;

d^E,θ

^E),

wherein, k=1˜

N, l=2˜

M_k˜

, and wherein H is a mapping that converts pixel coordinates to real-word coordinates.

10. The system of claim 1, wherein the processor is further programmed to determine spatial characteristics of the image capture assembly.

11. A method for profiling a location of an associated image capture device in a region of interest, the method comprising:

acquiring an image captured by an associated image capture device;

acquiring a reported position and pose of the associated image capture device;

processing the captured image to detect an object in the captured image;

identifying a set of interest points characterizing the captured object;

generating a relative position and pose of the interest points based on dimensional information associated with the captured object;

computing an estimated distance of the associated image capture device to the object using the reported position and pose of the associated image capture device and the relative position of the interest points;

computing an estimated position and pose of the associated the image capture device based on the estimated distance; and

updating the reported position and pose of the associated image capture device to the estimated position.

12. The method of claim 11 further comprising:

transmitting a layout of a region of interest to an associated mobile imaging device carrying the associated image capture device; and

transmitting a target position and pose to the associated mobile imaging device, wherein the target position and pose is located in the region of interest;

acquiring the reported position and pose of the associated image capture device after the associated mobile imaging device moves the associated image capture device to the target position and pose.

13. The method of claim 12, wherein the region of interest is a store and the captured object is a product located on a display in the store.

14. The method of claim 11, wherein the processor is further adapted to:

generate a relative position of the interest points, accessing a database of stored reference images of reference objects, the database being in communication with the processor and associating dimensional information with each reference object;

searching the stored images to match the captured object to the reference object;

in response to finding a match, determining the dimensional information associated with the reference object; and

associating the dimensional information with the captured object.

15. The method of claim 14 further comprising:

bounding the reference object in a bounding box;

associating each corner of the bounding box with a relative position in real-world coordinates based on the dimensional information;

identifying a set of reference markers in the reference image characterizing the reference object;

determining a relative reference position of each reference marker using at least one corner; and

,

computing a relative distance of the each reference marker to the at least one corner.

16. The method of claim 15 further comprising:

after identifying the set of interest points, assigning a set of interest points as the set of reference markers, where the interest points are points detected on the reference image using a computer vision approach selected from a group consisting of;

Corner detectors;

Harris corner detector;

SIFT;

SURF;

LoG;

DoG;

DoH scale-space interest points;

SUSAN detector; and

a combination of the above.

17. The method of claim 16, wherein the generating a relative position includes:

associating a first interest point in the captured image as a reference point; and

describing relative positions of the interest points using the equation;

(y_kl,z_kl)−

(y_k1,z_k1)=(β

_kl,γ

_kl),

wherein for N reference objects, M_l are the interest points corresponding to the captured object, M_l is the reference markers determined for k-th reference object; and

(x_kl, y_kl, z_kl) is a position of l-th interest point from k-th reference object, and the known value (β

_kl, γ

_kl) is derived based on the detected pixel locations of the interest point on the image of the reference object and its dimensional information.

18. The method of claim 16, wherein the computing the estimated position and pose of the associated mobile imaging device includes:

computing a relative pixel location of each one of the set of interest points of the captured object;

computing relative dimensions between sets of interest points using the relative pixel locations

computing coordinates of the associated image capture device to the captured object using the relative dimensions; and

,

computing the estimated position and pose of the associated mobile imaging device based on the computed coordinates.

19. The method of claim 18 further comprising, computing an expected relative position of each interest point using the equation:

(x_kl,y_kl,z_kl)−

(x_k1,y_k1,z_k1)=H(i_kl,j_kl;

d^E,θ

^E)−

H(i_k1,j_k1;

d^E,θ

^E), wherein k=1˜

N, l=2˜

M_k˜

, and wherein H is a mapping that converts pixel coordinates to real-word coordinates.

20. The method of claim 11 further comprising:

determining spatial characteristics of the image capture assembly.