BEHAVIORAL EVENT MEASUREMENT SYSTEM AND RELATED METHOD

• US 20150220814A1
• Filed: 02/06/2014
• Published: 08/06/2015
• Est. Priority Date: 02/06/2014
• Status: Active Grant

• Current Assignee: Verto Analytics Oy (Verto Analytics, Inc.)
• Original Assignee: Verto Analytics Oy (Verto Analytics, Inc.)
• Inventors: VERKASALO, Hannu, KAYAL, Subhradeep, KONTTURI, Matias, MALMI, Eric
• Granted Patent: US 9,471,912 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 20/321 using wearable devices
• CPC Class Codes: G06Q 20/322 Aspects of commerce using mobile devices [M-devices]
• CPC Class Codes: G06Q 20/3255 using mobile network messaging services for payment, e.g. SMS
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Electronic system for obtaining data, via one or more digital devices, on user behavior, digital transactions, and exposure relative to digital content and services, or external exposure and associated events between the user and the environment via sensors attached to digital devices, the system being configured to collect data reflecting the content and objects that the user at least potentially perceives as rendered on one or more digital screens attached to smart devices, reconstruct the at least potentially perceived visual landscape based on the collected data, and determine the target and/or level of user attention in view of the reconstruction and associated exposure events detected therein, and to apply locally stored information about rules or fingerprints in the digital object recognition process involving the collected data and validation of the type or identity of user actions, digital content, or external objects, as reflected by the reconstruction recapturing the visual landscape.

Claims

1. Electronic system for obtaining data, via one or more digital devices, on user behavior, digital transactions, and exposure relative to digital content and services, or on external exposure and associated events between the user and the environment through the use of a number of sensors, the system being configured to

collect data reflecting the content and objects that the user at least potentially perceives as rendered on one or more digital screens attached to the devices, through associated digital projections or characteristics specific to digital content or objects, or in the environment,

reconstruct the at least potentially perceived visual landscape based on the collected data, and

determine the target and/or level of user attention in view of the reconstruction and associated exposure events detected therein,

wherein the system is configured to apply at least some locally stored information about rules or fingerprints in the digital object recognition process involving the collected data and validation of the type or identity of user actions, digital content, or external objects, as reflected by the reconstruction recapturing the potential visual landscape.

2. The system of claim 1, configured to collect visual data through screen capturing.

3. The system of claim 1, configured to adapt data sampling, or specifically screen capturing rate for collecting visual data, through utilization of difference imaging, wherein small differences between subsequently captures images turn into reduced rate and vice versa, to reduce processing and memory requirements as well as power consumption.

4. The system of claim 1, configured to collect visual data through one or more cameras integrated or at least functionally connected with at least one digital device.

5. The system of claim 1, configured to utilize said rules and fingerprints in the recognition and landscape reconstruction process regarding at least element selected from the group consisting of:

icon, graphics, image, text string, app, service, payment, purchase, advertisement, menu item, menu indicator, and window element.

6. The system of claim 1, configured to apply user device-originating metadata database for conducting image recognition to reconstruct the visual landscapes, wherein the database is formed or updated by said digital devices configured to identify new objects visible to the user on the screen or in the environment, and provide indications thereof to the metadata database for the purposes of classifying the said object.

7. The system of claim 1, configured to provide, as a feedback to the collected data on user behavior, transactions or exposure, a user of digital device with content, notifications, or other visual information the user has interest in based on the analysis of the data.

8. The system of claim 1, configured to specifically collect information on the user'"'"'s physical condition including the status and movement of eyes to recognize the visual attention, optionally also heart rate, to validate that the user is awake and present in the situation and/or to validate the level of user attention in a given type of exposure.

9. The system of claim 1, configured to enrich visual content with data on the detected audio exposure or signals.

10. The system of claim 1, configured to specifically recognize, based on the collected and reconstructed visual data, one or more objects the user exposes to in the physical world, and the semantic meaning of such objects.

11. The system of claim 1, configured to detect home screen or menu screen from the obtained visual screen capture data.

12. The system of claim 1, configured to detect home screen or menu screen from the obtained visual screen capture data, and to recognize a home or menu screen based on detecting the outlook, number and/or positioning, optionally screen area and/or mutual positioning, of visual on-screen menu elements as defined by the fingerprints from the visual data.

13. The system of claim 1, configured to compress the obtained screen capture data by run-length encoding, reducing the image resolution and/or color quantization.

14. The system of claim 1, configured to track and log exposure to predefined visual content, such as a widget, based on extracting and recognizing content-associated graphical or textual information from the obtained visual screen data through utilization of icon matching and optical character recognition.

15. The system of claim 1, configured to determine entity, such as application, user preference or usage ranking based on detecting the on-screen order of entity-associated icons or other graphical representations.

16. The system of claim 1, configured to subject the obtained visual data to region of interest segmentation to separate different areas, such as foreground and background areas, from each other.

17. The system of claim 1, configured to subject the obtained visual data to text and non-text separation optionally by a block-based algorithm.

18. The system of claim 1, configured to recognize a payment or purchase transaction based on transaction window detection, transaction symbol such as purchase button or icon detection, and/or OCR-based transaction text detection.

19. The system of claim 1, configured to maintain one or more logs of recognized exposure events, user attention, and/or transactions in said digital devices and/or in a functionally connected remote server arrangement of at least one server.

20. The system of claim 1, comprising said one or more digital devices for obtaining and optionally analyzing said visual data.

21. The system of claim 1, comprising said one or more digital devices for obtaining and optionally analyzing said visual data, further comprising a server arrangement of at least one network-accessible, optionally cloud-located, server configured to receive collected and optionally analyzed data at least indicative of said visual data from said digital devices, process the data, store the data, and distribute the data and/or provide responsive data back to the digital devices optionally for visual or audible reproduction.

22. A method for obtaining data, via one or more digital devices, on user behavior, digital transactions, and exposure relative to digital content and services, or on external exposure and associated events between the user and the environment through the use of a number of sensors, optionally wearable technologies, comprising

collecting data reflecting the content and objects that the user at least potentially perceives as rendered on one or more digital screens attached to the devices, through associated digital projections or characteristics specific to digital content or objects, or in the environment,

reconstructing the at least potentially perceived visual landscape based on the collected data, and

determining the target and/or level of user attention in view of the reconstruction and associated exposure events detected therein,

wherein information stored locally at said one or more devices about rules or fingerprints is applied in the digital object recognition process involving the collected data and validation of the type or identity of user actions, digital content, or external objects, as reflected by the reconstruction recapturing the visual landscape.

23. A computer program product embodied on a non-transitory carrier medium comprising instructions causing the computer to collect data reflecting the content and objects that the user at least potentially perceives as rendered on one or more digital screens attached to smart devices, through associated digital projections or characteristics specific to digital content or objects, or in the environment, reconstruct the at least potentially perceived visual landscape based on the collected data, and determine the target and/or level of user attention in view of the reconstruction and associated exposure events detected therein, wherein information stored locally at said one or more devices about rules or fingerprints is applied in the digital object recognition process involving the collected data and validation of the type or identity of user actions, digital content, or external objects, as reflected by the reconstruction recapturing the visual landscape.