METHOD AND SYSTEM FOR FULL PATH ANALYSIS

• US 20130222599A1
• Filed: 09/05/2012
• Published: 08/29/2013
• Est. Priority Date: 02/29/2012
• Status: Active Grant

• Current Assignee: RetailNext, Inc.
• Original Assignee: George Shaw
• Inventors: SHAW, George
• Granted Patent: US 10,163,031 B2
• Time in Patent Office: Days
• US Class Current: 348/159
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A space is modeled using images from a set of cameras in the space. An easy to use tool is provided that allows users to identify a reference location in an image and a corresponding reference location on a floor plan of the space. Based on these correspondences, a model is generated that can map a point in a camera'"'"'s view to a point on the floor plan, or vice-versa. Subjects moving through the space are identified and tracked using the model and attributes associated with a subject. In a specific implementation, the attributes include velocity, color, size, and position. In a specific implementation, the space is a retail store and the subjects are customers browsing through the retail store.

Claims

1. A method comprising:

displaying on an electronic screen a floor plan of a space;

while the floor plan is being displayed, displaying on the electronic screen an image comprising at least portion of the space, the image having been captured by a camera in the space;

detecting on the floor plan user placement of a first plurality of calibration points;

detecting on the image user placement of a second plurality of calibration points, wherein the second plurality of calibration points placed on the image are to correspond to locations on the floor plan identified by the first plurality of calibration points;

generating a model based on the placement of the first and second plurality of calibration points; and

providing an indication of an ability of the model to map a point in the image captured by the camera to a location on the floor plan.

2. The method of claim 1 comprising:

displaying a plurality of coverage markers on at least one of the image or the floor plan, wherein a distribution of the plurality of coverage markers is the indication of the ability of the model to map a point in the image captured by the camera to a location on the floor plan.

3. The method of claim 1 comprising:

displaying a plurality of coverage markers on at least one of the image or the floor plan, wherein a twist of the plurality of coverage markers indicates an inability of the model to map the point in the image to the location on the floor plan.

4. The method of claim 1 comprising:

displaying a plurality of coverage markers on at least one of the image or the floor plan, wherein a folding of the plurality of coverage markers indicates an inability of the model to map the point in the image to the location on the floor plan.

5. The method of claim 1 wherein the detecting user placement of the first and second plurality of calibration points comprises:

prompting a user to select a reference location on the floor plan and to place a first calibration point of the first plurality of calibration points at the reference location;

detecting that the first calibration point of the first plurality of calibration points has been placed; and

after the detecting that the first calibration point of the first plurality of calibration points has been placed, prompting the user to place a first calibration point of the second plurality of calibration points at a location on the image that corresponds to the reference location.

6. The method of claim 1 wherein the detecting user placement of the first and second plurality of calibration points comprises:

prompting a user to select a reference location on the image and to place a first calibration point of the second plurality of calibration points at the reference location;

detecting that the first calibration point of the second plurality of calibration points has been placed; and

after the detecting that the first calibration point of the second plurality of calibration points has been placed, prompting the user to place a first calibration point of the first plurality of calibration points at a location on the image that corresponds to the reference location.

7. The method of claim 1 wherein the generating a model comprises:

determining real coordinates for the first plurality of calibration points;

determining pixel coordinates for the second plurality of calibration points; and

generating correspondence data between the real and pixel coordinates.

8. The method of claim 7 wherein the generating correspondence data comprises computing two-way transformation equations to map any point in a view of the camera onto the floor plan.

9. The method of claim 1 wherein the generating a model comprises:

selecting a transformation type based on user input and a number of calibration points in the first or second plurality of calibration points.

10. The method of claim 9 wherein the transformation type comprises a projective or a 3rd-degree polynomial.

11. The method of claim 1 wherein the floor plan is based on a first coordinate system, the image is based on a second coordinate system, different from the first coordinate system, and the method comprises:

computing a least-squares to fit a linear system that describes a relationship between the first and second coordinate systems based on the placement of the first and second pluralities of calibration points.

12. The method of claim 1 comprising:

displaying the floor plan within a first graphical window pane; and

displaying the image within a second graphical window pane, wherein the first and second graphical window panes are in a side-by-side vertical arrangement.

13. The method of claim 1 comprising:

displaying the floor plan within a first graphical window pane; and

displaying the image within a second graphical window pane, wherein the first and second graphical window panes are in a side-by-side horizontal arrangement.

14. A method comprising:

storing a base track of a subject captured by a first camera in a space;

storing a first candidate track of a first candidate subject captured by a second camera in the space;

storing a second candidate track of a second candidate subject captured by the second camera;

applying a model comprising camera-space to real-space mapping data to project the base, first, and second candidate tracks onto a floor plan of the space;

analyzing a set of attributes associated with the subject, and first and second candidate subjects to determine whether the first or second candidate subject is the same as the subject; and

based on the analysis, joining one of the projected first or second candidate tracks to the projected base track.

15. The method of claim 14 wherein the mapping data is based on a plurality calibration points placed on each view of each camera and a plurality of corresponding calibration points on placed on the floor plan, and each calibration point and corresponding calibration point is placed by a user using a graphical user interface tool.

16. The method of claim 14 wherein the base, first, and second candidate tracks are defined in a camera-space coordinate system and after the applying the model, the projected base, and first, and second candidate tracks are defined in a real-space coordinate system.

17. The method of claim 14 wherein a first attribute of the set of attributes comprises color and the analyzing a set of attributes comprises:

measuring a difference between a color of the subject and a color of the first candidate subject; and

measuring a difference between the color of the subject and a color of the second candidate subject.

18. The method of claim 14 wherein a first attribute of the set of attributes comprises size and the analyzing a set of attributes comprises:

measuring a difference between a size of the subject and a size of the first candidate subject; and

measuring a difference between the size of the subject and a size of the second candidate subject.

19. The method of claim 14 wherein the size comprises a height and a width.

20. The method of claim 14 wherein a first attribute of the set of attributes comprises position and the analyzing a set of attributes comprises:

measuring a difference between a position of the projected base track and a position of the projected first candidate track; and

measuring a difference between the position of the projected base track and the position of the projected second candidate track.

21. The method of claim 14 wherein a first attribute of the set of attributes comprises velocity and the analyzing a set of attributes comprises:

measuring a difference between a velocity of the projected base track and a velocity of the projected first candidate track; and

measuring a difference between the velocity of the projected base track and a velocity of the projected second candidate track.

22. The method of claim 14 wherein the analyzing a set of attributes comprises:

scoring the projected base track and projected first candidate track by;

calculating a first difference in velocity between the subject and first candidate subject and applying a first weight to the first difference;

calculating a second difference in color between the subject and first candidate subject and applying a second weight to the second difference;

calculating a third difference in size between the subject and first candidate subject and applying a third weight to the third difference; and

calculating a fourth difference in position between the subject and first candidate subject and applying a fourth weight to the fourth difference; and

scoring the projected base track and projected second candidate track by;

calculating a fifth difference in velocity between the subject and second candidate subject and applying the first weight to the fifth difference;

calculating a sixth difference in color between the subject and second candidate subject and applying the second weight to the sixth difference;

calculating a seventh difference in size between the subject and second candidate subject and applying the third weight to the seventh difference; and

calculating an eighth difference in position between the subject and second candidate subject and applying the fourth weight to the eighth difference.

23. The method of claim 22 wherein second weight is greater than each of the first, third, and fourth weights.

24. A method comprising:

providing a floor plan of a space comprising a first camera that provides a first image;

specifying a plurality of first control points on one of the floor plan or first image to generate a model;

specifying a plurality of second control points on another of the floor plan or first image to generate the model, wherein the plurality of second control points are to correspond to the plurality of first control points; and

after the specifying a plurality of first and second control points, reviewing a distribution of a plurality of coverage markers shown on at least one of the first image or floor plan to determine an ability of the model to map a point in the first image to a point on the floor plan.

25. The method of claim 24 comprising:

after the reviewing a distribution, specifying an additional first control point on one of the floor plan or first image to generate a new model;

specifying an additional second control point on another of the floor plan or first image to generate the new model; and

after the specifying an additional first and second control points, reviewing a new distribution of the plurality of coverage markers shown on at least one of the first image or floor plan to determine an ability of the new model to map the point in the first image to the point on the floor plan.

26. The method of claim 24 comprising:

after the reviewing a distribution, adjusting a position of a first control point of the plurality of first control points specified on one of the floor plan or first image to generate a new model; and

after the adjusting a position, reviewing a new distribution of the plurality of coverage markers shown on at least one of the first image or floor plan to determine an ability of the new model to map the point in the first image to the point on the floor plan.

27. The method of claim 24 wherein the space comprises a second camera that provides a second image, and the method comprises:

specifying a plurality of third control points on one of the floor plan or second image to generate the model;

specifying a plurality of fourth control points on another of the floor plan or second image to generate the model, wherein the plurality of fourth control points are to correspond to the plurality of third control points; and

after the specifying the plurality of third and fourth control points, reviewing another distribution of another plurality of coverage markers shown on at least one of the second image or floor plan to determine an ability of the model to map a second point in the second image to a second point on the floor plan.

28. A method comprising:

providing a floor plan of a space comprising a first camera that provides a first view of the space and a second camera that provides a second view of the space;

adjusting the first camera so that the first view is partially overlapping with the second view;

placing, using a graphical user interface tool, a plurality of first control points on one of the first view or the floor plan, and a plurality of first corresponding control points on another of the first view or the floor plan to generate a first model; and

reviewing a distribution of coverage markers shown on at least one of the first view or floor plan to determine an ability of the first model to map a point in the first view to a point on the floor plan.

29. The method of claim 28 comprising:

after the reviewing a distribution of coverage markers, adjusting, using the graphical user interface tool, placement of a control point on the first view.

30. The method of claim 28 comprising:

after the reviewing a distribution of coverage markers, adjusting, using the graphical user interface tool, placement of a control point on the floor plan.

31. The method of claim 28 comprising:

after the reviewing a distribution of coverage markers, placing, using the graphical user interface tool, an additional first control point on the first view; and

placing an additional first corresponding control point on the floor plan.

32. The method of claim 28 comprising:

after the reviewing a distribution of coverage markers, placing, using the graphical user interface tool, an additional first control point on the floor plan; and

placing an additional first corresponding control point on the first view.

33. The method of claim 28 wherein a number of the plurality of first control points is equal to a number of the plurality of first corresponding control points.

34. The method of claim 28 comprising installing the first camera in the space.

35. The method of claim 28 comprising not installing the first camera in the space, the first camera being an existing camera in the space.

36. The method of claim 28 comprising:

selecting a first control point that has been placed on one of the first view or floor plan;

reviewing a first corresponding control point that as been placed on another of the first view or floor plan, wherein the first corresponding control point is highlighted; and

repositioning at least one of the first control point or the first corresponding control point.

37. The method of claim 28 comprising:

placing, using the graphical user interface tool, a plurality of second control points on one of the second view or the floor plan, and a plurality of second corresponding control points on another of the second view or the floor plan to generate a second model; and

reviewing a second distribution of coverage markers shown on at least one of the second view or floor plan to determine an ability of the second model to map a point in the second view to a second point on the floor plan.

38. The method of claim 28 wherein a lens of the first camera is different from a lens of the second camera.

39. The method of claim 28 wherein a make of the first camera is different from a make of the second camera.