Method and system for in-store shopper behavior analysis with multi-modal sensor fusion

• US 10,217,120 B1
• Filed: 04/21/2015
• Issued: 02/26/2019
• Est. Priority Date: 04/21/2015
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Shin, Joonhwa, Sharma, Rajeev
• Primary Examiner(s): Koneru, Sujay
• Application Number:
  US14/692,446
• Time in Patent Office: 1,407 Days
• CPC Class Codes: G01S 5/0252 Radio frequency fingerprinting
• CPC Class Codes: G01S 5/0264 at least one of the systems being a non-radio wave positioning system
• CPC Class Codes: G01S 5/0294 Trajectory determination or predictive filtering, e.g. target tracking or Kalman filtering
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 64/006 with additional information processing, e.g. for direction or speed determination

Abstract

The present invention provides a comprehensive method for automatically and unobtrusively analyzing the in-store behavior of people visiting a physical space using a multi-modal fusion based on multiple types of sensors. The types of sensors employed may include cameras for capturing a plurality of images and mobile signal sensors for capturing a plurality of Wi-Fi signals. The present invention integrates the plurality of input sensor measurements to reliably and persistently track the people'"'"'s physical attributes and detect the people'"'"'s interactions with retail elements. The physical and contextual attributes collected from the processed shopper tracks includes the motion dynamics changes triggered by an implicit and explicit interaction to a retail element, comprising the behavior information for the trip of the people. The present invention integrates point-of-sale transaction data with the shopper behavior by finding and associating the transaction data that corresponds to a shopper trajectory and fusing them to generate a complete an intermediate representation of a shopper trip data, called a TripVector. The shopper behavior analyses are carried out based on the extracted TripVector. The analyzed behavior information for the shopper trips yields exemplary behavior analysis comprising map generation as visualization of the behavior, quantitative shopper metric derivation in multiple scales (e.g., store-wide and category-level) including path-to-purchase shopper metrics (e.g., traffic distribution, shopping action distribution, buying action distribution, conversion funnel), category dynamics (e.g., dominant path, category correlation, category sequence). The present invention includes a set of derived methods for different sensor configurations.

Claims

1. A method for automatically and unobtrusively analyzing in-store behavior of people visiting a physical space based on a fusion of a set of mobile signal- and vision-based person trajectories, an association of the set of mobile signal- and vision-based trajectories with a set of transaction data, and automatic recognition of a set of pre-defined shopping actions, using at least a computing machine, a set of mobile signal and vision sensors, and a set of computer vision and mobile signal processing algorithms, comprising:

a. setting-up a plurality of types of vision and mobile signal sensors in an area of interest such as a retail store,

b. tracking a plurality of persons individually using a set of cameras and a set of mobile signal sensors and a set of corresponding computer vision and mobile signal processing algorithms and yielding a set of vision-based trajectories and a set of mobile signal-based trajectories,

c. fusing a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a person, further comprising;

i. retrieving a pool of candidate vision-based trajectories from a database wherein the pool of candidate vision-based trajectories are generated in a similar time frame during which the mobile signal-based trajectory is generated,

ii. identifying a set of vision-based trajectories among the pool of candidate vision-based trajectories by comparing the distance statistics of the set of vision-based trajectories to the mobile signal-based trajectory of the mobile-carrying person and comparing the motion dynamics of the set of vision-based trajectories and the mobile signal-based trajectory, which includes direction and speed,

iii. integrating the set of vision-based trajectories to generate a fused trajectory and to account for a plurality of vision measurements for a same target at a same time instance,

iv. interpolating the missing segments of the fused trajectory by excerpting the missing segments from the mobile signal-based trajectory stored in a database based on a set of point-to-point correspondence information between the set of vision-based trajectories and the mobile signal-based trajectory, and

v. refining the fused trajectory by incorporating a store floor plan and a set of layout information that describes an occupancy map of a set of fixtures and other facilities or equipments where a set of shopper trajectories can not exist,

d. associating a transaction data set among a pool of candidate transaction data to the fused trajectory based on a set of purchased items and the locations of said set of purchased items,

e. extracting an intermediate shopper behavior representation, called a TripVector, from the fused trajectory and the transaction data set associated to said fused trajectory through detecting and recognizing a set of pre-defined shopping actions,

f. generating a set of pre-defined shopper metric measurements and behavior analyses based on the TripVector, wherein the transaction data set includes a set of items purchased in a trip by a shopper.

2. The method according to claim 1, wherein said method further comprises a step of setting-up a plurality of types of vision and mobile-signal sensors in an area of interest, further comprising:

a. deploying a plurality of cameras and a plurality of mobile signal sensors in the area of interest,

b. calibrating a set of predetermined intrinsic and extrinsic camera calibration parameters using a set of camera calibration patterns,

c. calibrating a set of mobile signal sensors by generating a mobile signal map for the area of interest,

wherein said mobile signal map is a received signal strength-to-distance mapping or a radio fingerprint-to-location mapping,

wherein a radio fingerprint is a vector of received signal strength values of a set of nearby access points for a specific location.

3. The method according to claim 1, wherein said method further comprises a step of tracking a plurality of persons using a set of cameras and a set of corresponding computer vision algorithms and yielding a set of vision-based trajectories, further comprising:

a. capturing a set of images by one or more cameras,

b. learning and building a background model,

c. segmenting the foreground of a set of images using the background model,

d. detecting a plurality of objects using an object detection algorithm,

e. tracking the plurality of objects using a stochastic tracker,

f. fusing one or more measurements of a same object from one or more cameras.

4. The method according to claim 1, wherein said method further comprises a step of tracking a set of persons using a set of mobile signal sensors and a set of corresponding mobile signal processing algorithms and yielding a set of mobile signal-based trajectories, further comprising:

a. detecting a set of MAC addresses from a set of received packets,

b. collecting a set of received signal strength values to distance measures and a set of timestamps of the set of received packets,

c. converting the set of received signal strength values to distance measures based on a learned mobile signal map,

d. detecting a measured location of a set of objects using a trilateration method,

e. estimating a true location of the set of objects using a tracking method.

5. The method according to claim 1, wherein said method further comprises a step of tracking a set of persons using a set of mobile signal sensors and a set of corresponding mobile signal processing algorithms and yielding a set of mobile signal-based trajectories, further comprising:

a. detecting a set of MAC addresses from a set of received packets,

b. collecting a set of received signal strength values and a set of timestamps of said set of received packets,

c. converting the set of received signal strength values into a set of distance measures based on a learned mobile signal map,

d. comparing a set of generated radio fingerprints with a learned fingerprint-to-location mapping,

e. detecting a measured location of a set of objects using a trilateration method,

f. estimating a true location of the set of objects using a tracking method.

6. The method according to claim 1, wherein said method further comprises a step of fusing a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a mobile-carrying person,

wherein the method includes a sensing configuration wherein sensing by a set of mobile signal sensors and cameras covers the entirety of a store.

7. The method according to claim 1, wherein said method further comprises a step of fusing a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a mobile-carrying person,

wherein the method includes a sensing configuration wherein sensing by a set of mobile signal sensors covers the entirety of a store and sensing by a set of cameras partially covers the store.

8. The method according to claim 1, wherein said method further comprises a step of fusing a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a mobile-carrying person,

wherein the method includes a sensing configuration wherein sensing by a set of mobile signal sensors partially covers a store and sensing by a set of cameras covers the entirety of a store.

9. The method according to claim 1, wherein said method further comprises a step of fusing a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a mobile-carrying person,

wherein the method includes a sensing configuration wherein sensing by a set of mobile signal sensors and a set of cameras partially cover a store,

whereby individual shopper-based tracking can be carried out for a set of areas of interest that are covered.

10. The method according to claim 1, wherein said method further comprises a step of associating a transaction data set among a pool of candidate transaction data to a fused trajectory based on a set of purchased items and the locations of said set of purchased items, further comprising:

a. creating a pool of candidate transaction data wherein the pool of candidate transaction data occur in a time-frame when the person of the fused trajectory checked out,

b. performing an initial matching between a set of transaction data among the pool of candidate transaction data and the fused trajectory by marking the locations of a set of purchased items in the set of transaction data on a store layout and laying down the fused trajectory on said store layout,

c. estimating the purchase order of the set of purchased items based on a timestamp of a closest shopping fixation in the fused trajectory, wherein the shopping fixation refers to an event whereby a change in a behavior of a shopper occurs and a retail elements triggers said behavior of said shopper,

d. generating a synthetic trajectory wherein said synthetic trajectory passes through the set of purchased items in the estimated purchase order,

e. computing a dissimilarity measure between the synthetic trajectory and the fused trajectory while smoothing the synthetic trajectory with a set of different smoothing factors,

f. finding a minimum dissimilarity measure in the scale space of the synthetic trajectory,

g. finding a set of transaction data with the minimum dissimilarity measure in the scale space from the pool of candidate transaction data,

h. associating the set of transaction data to the fused trajectory.

11. The method according to claim 10, wherein said method further comprises a step of computing a dissimilarity measure between a synthetic trajectory and a fused trajectory,

wherein the computation of the dissimilarity measure uses a probabilistic framework,

wherein the probabilistic framework estimates a likelihood that a person picks up a purchased item along the fused trajectory and a likelihood that a person picks up a set of purchased items along the fused trajectory.

12. The method according to claim 10, wherein said method further comprises a step of computing a dissimilarity measure between a synthetic trajectory and a fused trajectory,

wherein the computation of the dissimilarity measure uses a geometric framework,

wherein the geometric framework represents a likelihood that a person picks up a purchased item along the fused trajectory and a likelihood that a person picks up a set of purchased items along the fused trajectory.

13. The method according to claim 1, wherein said method further comprises a step of associating a transaction data set among a pool of candidate transaction data to the fused trajectory based on a set of purchased items and the locations of said set of purchased items,

wherein the method includes a sensing configuration wherein sensing by a set of mobile signal sensors and a set of cameras covers the entirety of a store.

14. The method according to claim 1, wherein said method further comprises a step of associating a transaction data set among a pool of candidate transaction data to a fused trajectory based on a set of purchased items and the locations of said set of purchased items,

wherein the method includes a sensing configuration wherein sensing by a set of mobile signal sensors covers the entirety of a store and sensing by a set of cameras partially covers the store,

wherein a trajectory-transaction data association process comprises:

a. laying down a fused trajectory on a store floor plan and marking a set of locations of a set of purchased items,

b. associating each purchased item with a timestamp of a closest shopping fixation point in the fused trajectory,

c. creating a synthetic trajectory by connecting the set of locations of the set of purchased items in a time order,

d. computing a similarity measure between the fused trajectory and the synthetic trajectory in the scale space of the synthetic trajectory,

e. finding a minimum distance between the fused trajectory and the synthetic trajectory, wherein the minimum distance functions as a dissimilarity measure between each pair of the fused trajectory and the synthetic trajectory in the scale space of the synthetic trajectory,

f. performing the trajectory-transaction data association process for a set of candidate synthetic trajectories with a set of different smoothing factors,

g. finding a set of transaction data corresponding to the synthetic trajectory with a minimum dissimilarity with the fused trajectory.

15. The method according to claim 1, wherein said method further comprises a step of associating a transaction data set among a pool of candidate transaction data to the fused trajectory based on a set of purchased items and the locations of said set of purchased items,

wherein the method includes a sensing configuration wherein sensing by a set of mobile signal sensors partially covers a store and sensing by a set of cameras covers the entirety of the store,

wherein a trajectory-transaction data association process comprises:

a. constructing and storing a visual feature model of a shopper for each vision trajectory segment a set of associated timestamps and storing a MAC address of a device of a shopper and an associated timestamp,

b. creating the visual feature model of the shopper and collecting a list of the MAC addresses of mobile devices present around a checkout,

c. searching the visual feature model of the shopper through a set of visual feature models whereby the set of visual feature models are already created in a set of areas of interest,

d. finding a set of matched visual feature models in the checkout and the set of areas of interest,

e. listing a set of MAC addresses of a plurality of mobile devices detected in a set of similar time frames in the checkout and the set of areas of interest,

f. finding a single MAC address that appears in the checkout and the set of areas of interest or most of the set of areas of interest,

g. storing all MAC addresses that appear the checkout and the set of areas of interest or most of the set of areas of interest.

16. The method according to claim 1, wherein said method further comprises a step of associating a transaction data set among a pool of candidate transaction data to the fused trajectory based on a set of purchased items and the locations of said set of purchased items,

wherein the method includes a sensing configuration wherein sensing by a set of mobile signal sensors and a set of cameras partially cover a store.

17. The method according to claim 1, wherein said method further comprises a step of extracting a TripVector based on a trajectory of a person and a set of transaction data wherein the set of transaction data is associated with the trajectory, further comprising:

a. assigning a shopper identification to the TripVector,

b. extracting a set of pre-defined physical attributes of the person,

c. extracting a set of pre-defined contextual attributes of the person,

wherein the set of pre-defined physical attributes comprises a set of locations of the person, a set of associated timestamps, and a set of measurements comprising head orientation, gesture, and visual appearance,

wherein the set of pre-defined contextual attributes comprises visual attention of the person, physical approaching or contact to a product or a display, a set of pre-defined shopping actions comprising holding a product, comparing a plurality of different products, and purchasing a product.

18. The method according to claim 1, wherein said method further comprises a step of generating a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector,

wherein the set of pre-defined shopper metric measurements and behavior analyses comprises a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein the path-to-purchase analysis comprises computing a traffic distribution, a shopping action distribution, and a buying action distribution, a layered map generation, and a conversion calculation,

wherein the category dynamics analysis comprises computing a dominant path, a category correlation, and a category sequence,

whereby the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis comprises performing said analysis for a part of a population whereby the part of the population is filtered or segmented based on a pre-determined set of criteria.

19. The method according to claim 1, wherein said method further comprises a step of generating a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector from a configuration wherein the configuration comprises sensing by a set of mobile signal sensors and cameras which covers the entirety of a store,

wherein the set of behavior analyses comprises a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein said method performs the path-to-purchase analysis, the category dynamics analysis, the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis for a plurality of shopper trajectories throughout the entirety of a store.

20. The method according to claim 1, wherein said method further comprises a step of generating a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector, from a configuration wherein the configuration comprises sensing by a set of mobile signal sensors which covers an entirety of a store and by a set of cameras which partially covers the store,

wherein the set of pre-defined shopper metric measurements and behavior analyses comprise a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein said method performs the path-to-purchase analysis, the category dynamics analysis, the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis for a plurality of shopper trajectories throughout an entirety of a store, wherein the path-to-purchase analysis computes a traffic distribution and a buying action distribution for the entirety of the store and computes a partial shopping action distribution for a part of the store wherein the part of the store contains a set of cameras,

wherein the path-to-purchase analysis computes a layered map generation and a conversion calculation which incorporates the traffic distribution and the buying action distribution for the entirety of the store and a partial shopping action distribution.

21. The method according to claim 1, wherein said method further comprises a step of generating a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector, from a configuration wherein the configuration comprises sensing by a set of mobile signal sensors which partially covers a store and by a set of cameras which covers the entirety of the store,

wherein the set of pre-defined shopper metric measurements and behavior analyses comprise a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein said method performs the path-to-purchase analysis and the category dynamics analysis, throughout an entirety of a store,

wherein said method performs the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis for a plurality of shopper trajectories in a set of areas in the store wherein the set of areas contain a set of mobile signal sensors.

22. The method according to claim 1, wherein said method further comprises a step of generating a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector, from a configuration wherein the configuration comprises sensing by a set of mobile signal sensors and the set of cameras which partially cover a store,

wherein the set of behavior analyses comprises a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein said method performs the path-to-purchase analysis, the category dynamics analysis, the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis for a plurality of shopper trajectories in a set of areas in the store wherein the set of areas contain a set of mobile signal sensors and cameras.

36. The apparatus system according to claim 23, wherein said apparatus further comprises a computer that associates a transaction data set among a pool of candidate transaction data to a fused trajectory based on a set of purchased items and the locations of said set of purchased items,

wherein said computer carries out the method according to claim 1 under a sensing configuration wherein sensing by a set of mobile signal sensors covers the entirety of a store and sensing by a set of cameras partially covers the store,

wherein said computer that carries out a trajectory-transaction data association process comprises:

a. the computer that lays down a fused trajectory on a store floor plan and marking a set of locations of a set of purchased items,

b. the computer that associates each purchased item with a timestamp of a closest shopping fixation point in the fused trajectory,

c. the computer that creates a synthetic trajectory by connecting the set of locations of the set of purchased items in a time order,

d. the computer that computes a similarity measure between the fused trajectory and the synthetic trajectory in the scale space of the synthetic trajectory,

e. the computer that finds a minimum distance between the fused trajectory and the synthetic trajectory, wherein the minimum distance functions as a dissimilarity measure between each pair of the fused trajectory and the synthetic trajectory in the scale space of the synthetic trajectory,

f. the computer that performs the trajectory-transaction data association process for a set of candidate synthetic trajectories with a set of different smoothing factors,

g. the computer that finds a set of transaction data corresponding to the synthetic trajectory with a minimum dissimilarity with the fused trajectory.

37. The system according to claim 23, wherein said apparatus further comprises a computer that associates a transaction data set among a pool of candidate transaction data to a fused trajectory based on a set of purchased items and the locations of said set of purchased items,

wherein said computer carries out the method according to claim 1 under a sensing configuration wherein sensing by a set of mobile signal sensors partially covers a store and sensing by a set of cameras covers the entirety of the store,

wherein said computer that carries out a trajectory-transaction data association process comprises:

a. the computer that constructs and stores a visual feature model of a shopper for each vision trajectory segment and a set of associated timestamps and storing a MAC address of a device of a shopper and an associated timestamp,

b. the computer that creates the visual feature model of the shopper and collecting a list of the MAC addresses of a plurality of mobile devices present around a checkout,

c. the computer that searches the visual feature model of the shopper through a set of visual feature models wherein the set of visual feature models are already created in a set of areas of interest,

d. the computer that finds a set of matched visual feature models in the checkout and the set of areas of interest,

e. the computer that lists a set of MAC addresses of a plurality of mobile devices detected in a set of similar time frames in the checkout and the set of areas of interest,

f. the computer that finds a single MAC address that appears in the checkout and the set of areas of interest or most of the set of areas of interest,

g. the computer that stores all MAC addresses that appear the checkout and the set of areas of interest or most of the set of areas of interest.

23. An automatic and unobtrusive system that analyzes in-store behavior of a plurality of people visiting a physical space based on a fusion of a set of mobile signal- and vision-based person trajectories, an association of the set of mobile signal- and vision-based trajectories with a set of transaction data, and automatic recognition of a set of pre-defined shopping actions, using at least a computing machine, a set of mobile signal and vision sensors, and a set of computer vision and mobile signal processing algorithms, comprising:

a. a computer and a set of human operators that set-up a plurality of types of vision and mobile signal sensors in an area of interest such as a retail store,

b. the computer that tracks a plurality of persons individually using a set of cameras and a set of mobile signal sensors and a set of corresponding computer vision and mobile signal processing algorithms and yielding a set of vision-based trajectories and a set of mobile signal-based trajectories,

c. the computer that fuses a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a person, further comprising;

i. retrieving a pool of candidate vision-based trajectories from a database wherein the pool of candidate vision-based trajectories are generated in a similar time frame during which the mobile signal-based trajectory is generated,

ii. identifying a set of vision-based trajectories among the pool of candidate vision-based trajectories by comparing the distance statistics of the set of vision-based trajectories to the mobile signal-based trajectory of the mobile-carrying person and comparing the motion dynamics of the set of vision-based trajectories and the mobile signal-based trajectory, which includes direction and speed,

iii. integrating the set of vision-based trajectories to generate a fused trajectory and to account for a plurality of vision measurements for a same target at a same time instance,

iv. interpolating the missing segments of the fused trajectory by excerpting the missing segments from the mobile signal-based trajectory stored in a database based on a set of point-to-point correspondence information between the set of vision-based trajectories and the mobile signal-based trajectory, and

v. refining the fused trajectory by incorporating a store floor plan and a set of layout information that describes an occupancy map of a set of fixtures and other facilities or equipments where a set of shopper trajectories can not exist,

d. the computer that associates a transaction data set among a pool of candidate transaction data to the fused trajectory based on a set of purchased items and the locations of said set of purchased items,

e. the computer that extracts an intermediate shopper behavior representation, called a TripVector, from the fused trajectory and the transaction data set associated to said fused trajectory through detecting and recognizing a set of pre-defined shopping actions,

f. the computer that generates a set of pre-defined shopper metric measurements and behavior analyses based on the TripVector,

wherein a transaction data set includes a set of items purchased in a trip by a shopper.

24. The system according to claim 23, wherein said apparatus further comprises a computer and a set of human operators that set up a plurality of types of vision and mobile-signal sensors in an area of interest, further comprising:

a. a set of human operators who deploy a plurality of cameras and a plurality of mobile signal sensors in the area of interest,

b. a computer that calibrates a set of predetermined intrinsic and extrinsic camera calibration parameters using a set of camera calibration patterns,

c. a computer that calibrates a set of mobile signal sensors by generating a mobile signal map for the area of interest,

wherein said mobile signal map is a received signal strength-to-distance mapping or a radio fingerprint-to-location mapping,

wherein a radio fingerprint is a vector of received signal strength values of a set of nearby access points for a specific location.

25. The system according to claim 23, wherein said apparatus further comprises a computer that tracks a plurality of persons using a set of cameras and a set of corresponding computer vision algorithms, further comprising:

a. a set of cameras that captures a set of images,

b. a computer that learns and builds a background model,

c. the computer that segments the foreground of a set of images using the background model,

d. the computer that detects a plurality of objects using an object detection algorithm,

e. the computer that tracks the plurality of objects using a stochastic tracker,

f. the computer that fuses one or more measurements of a same object from one or more cameras.

26. The system according to claim 23, wherein said apparatus further comprises a computer that tracks a plurality of persons using a set of mobile signal sensors and a set of corresponding mobile signal processing algorithms, further comprising:

a. a set of mobile signal sensors that detects a set of MAC addresses from a set of received packets,

b. a computer that collects a set of received signal strength values to distance measures and a set of timestamps of the set of received packets,

c. the computer that converts the set of received signal strength values to distance measures based on a learned mobile signal map,

d. the computer that detects a measured location of a set of objects using a trilateration method,

e. the computer that estimates a true location of the set of objects using a tracking method.

27. The system according to claim 23, wherein said apparatus further comprises a computer that tracks a set of persons using a set of mobile signal sensors and a set of corresponding mobile signal processing algorithms, further comprising:

a. a set of mobile signal sensors that detects a set of MAC addresses from a set of received packets,

b. a computer that collects a set of received signal strength values and a set of timestamps of the set of received packets,

c. the computer that converts the set of received signal strength values into a set of distance measures based on a learned mobile signal map,

d. the computer that compares a set of generated radio fingerprints with a learned fingerprint-to-location mapping,

e. the computer that detects a measured location of a set of objects using a trilateration method,

f. the computer that estimates a true location of the set of objects using a tracking method.

28. The system according to claim 23, wherein said apparatus further comprises a computer that fuses a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a person.

29. The system according to claim 23, wherein said apparatus further comprises a computer that fuses a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a person.

30. The system according to claim 23, wherein said apparatus further comprises a computer that fuses a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a person,

wherein said computer carries out the method under a sensing configuration wherein sensing by a set of mobile signal sensors partially covers a store and sensing by a set of cameras covers the entirety of the store.

31. The system according to claim 23, wherein said apparatus further comprises a computer that fuses a mobile signal-based trajectory to a set of corresponding vision-based trajectories through a matching method and generating a fused trajectory for a person,

wherein individual shopper-based tracking occurs for a set of covered areas of interest.

32. The system according to claim 23, wherein said apparatus further comprises a computer that associates a transaction data set among a pool of candidate transaction data to a fused trajectory based on a set of purchased items and the locations of said set of purchased items, further comprising:

a. the computer that creates a pool of candidate transaction data wherein the pool of candidate transaction data occur in a time-frame when the person of the fused trajectory checked out,

b. the computer that performs an initial matching between a set of transaction data among the pool of candidate transaction data and the fused trajectory by marking the locations of a set of purchased items in the set of transaction data on a store layout and laying down the fused trajectory on said store layout,

c. the computer that estimates the purchase order of the set of purchased items based on a timestamp of a closest shopping fixation in the fused trajectory, wherein the shopping fixation refers to an event where a change in a behavior of a shopper occurs and a retail element triggers said behavior of said shopper,

d. the computer that generates a synthetic trajectory wherein said synthetic trajectory passes through the set of purchased items in the estimated purchase order,

e. the computer that computes a dissimilarity measure between the synthetic trajectory and the fused trajectory while smoothing the synthetic trajectory with a set of different smoothing factors,

f. the computer that finds a minimum dissimilarity measure in the scale space of the synthetic trajectory,

g. the computer that finds a set of transaction data with the minimum dissimilarity measure in the scale space from the pool of candidate transaction data,

h. the computer that associates the set of transaction data to the fused trajectory.

33. The system according to claim 32, wherein said apparatus further comprises a computer that computes a dissimilarity measure between a synthetic trajectory and a fused trajectory,

wherein the computation of the dissimilarity measure uses a probabilistic framework,

wherein the probabilistic framework estimates a likelihood that a person picks up a purchased item along the fused trajectory and a likelihood that a person picks up a set of purchased items along the fused trajectory.

34. The system according to claim 32, wherein said apparatus further comprises a computer that computes a dissimilarity measure between a synthetic trajectory and a fused trajectory,

wherein the computation of the dissimilarity measure uses a geometric framework,

wherein the geometric framework represents a likelihood that a person picks up a purchased item along the fused trajectory and a likelihood that a person picks up a set of purchased items along the fused trajectory.

35. The system according to claim 23, wherein said apparatus further comprises a computer that associates a transaction data set among a pool of candidate transaction data to a fused trajectory based on a set of purchased items and the locations of said set of purchased items.

38. The system according to claim 23, wherein said apparatus further comprises a computer that associates a transaction data set among a pool of candidate transaction data to a fused trajectory based on a set of purchased items and the locations of said set of purchased items.

39. The system according to claim 23, wherein said apparatus further comprises a computer that extracts an intermediate shopper behavior representation, called a TripVector, from a fused trajectory and a transaction data set associated to said fused trajectory through detecting and recognizing a set of pre-defined shopping actions, further comprising:

a. the computer that assigns a shopper identification to the TripVector,

b. the computer that extracts a set of pre-defined physical attributes of the person,

c. the computer that extracts a set of pre-defined contextual attributes of the person,

wherein the set of pre-defined physical attributes includes a set of locations of the person, a set of associated timestamps, and a set of measurements comprising head orientation, gesture, and visual appearance,

wherein the set of pre-defined contextual attributes comprises visual attention of the person, physical approaching or contact to a product or a display, a set of pre-defined shopping actions comprising holding a product, comparing different products, and purchasing a product.

40. The system according to claim 23, wherein said apparatus further comprises a computer that generates a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector,

wherein the set of pre-defined shopper metric measurements and behavior analyses comprises a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein the path-to-purchase analysis comprises computing a traffic distribution, a shopping action distribution, and a buying action distribution, a layered map generation, and a conversion calculation,

wherein the category dynamics analysis comprises computing a dominant path, a category correlation, and a category sequence,

whereby the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis comprises performing said analysis for a part of the population wherein the part of the population is filtered or segmented based on a pre-determined set of criteria.

41. The system according to claim 23, wherein said apparatus further comprises a step of generating a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector,

wherein said computer carries out said set of pre-defined shopper metric measurements and behavior analyses from a configuration wherein sensing by a set of mobile signal sensors and cameras covers the entirety of a store,

wherein the set of behavior analyses comprises a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein said computer performs the path-to-purchase analysis, the category dynamics analysis, the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis for a plurality of shopper trajectories throughout an entirety of a store.

42. The system according to claim 23, wherein said apparatus further comprises a step of generating a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector,

wherein said computer carries out said set of pre-defined shopper metric measurements and behavior analyses from a configuration wherein sensing by a set of mobile signal sensors covers the entirety of a store and sensing by a set of cameras partially covers the store,

wherein the set of pre-defined shopper metric measurements and behavior analyses comprise a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein said computer performs the path-to-purchase analysis, the category dynamics analysis, the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis for a plurality of shopper trajectories throughout an entirety of a store, wherein the path-to-purchase analysis computes a traffic distribution and a buying action distribution for the entirety of the store and computes a partial shopping action distribution for a part of the store wherein the part of the store contains a set of cameras,

wherein the path-to-purchase analysis computes a layered map generation and a conversion calculation which incorporates the traffic distribution and the buying action distribution for the entirety of the store and a partial shopping action distribution.

43. The system according to claim 23, wherein said apparatus further comprises a step of generating a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector,

wherein said computer carries out said set of pre-defined shopper metric measurements and behavior analyses from a configuration wherein sensing by a set of mobile signal sensors partially covers a store and sensing by a set of cameras covers the entirety of the store,

wherein the set of pre-defined shopper metric measurements and behavior analyses comprise a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein said computer performs the path-to-purchase analysis and the category dynamics analysis, throughout the entirety of a store,

wherein said computer performs the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis for a plurality of shopper trajectories in a set of areas in the store wherein the set of areas contain a set of mobile signal sensors.

44. The system according to claim 23, wherein said apparatus further comprises a step of generating a set of pre-defined shopper metric measurements and behavior analyses based on a TripVector,

wherein said computer carries out said set of pre-defined shopper metric measurements and behavior analyses from a configuration wherein sensing by a set of mobile signal sensors and the set of cameras partially cover a store,

wherein the set of behavior analyses comprises a path-to-purchase analysis, a category dynamics analysis, a filtering-based shopper metric analysis and a segmentation-based shopper metric analysis,

wherein said computer performs the path-to-purchase analysis, the category dynamics analysis, the filtering-based shopper metric analysis and the segmentation-based shopper metric analysis for a plurality of shopper trajectories in a set of areas in the store wherein the set of areas contain a set of mobile signal sensors and cameras.