Method, computer program product, and system for providing a sensor-based environment

• US 10,482,724 B2
• Filed: 03/16/2015
• Issued: 11/19/2019
• Est. Priority Date: 10/15/2014
• Status: Active Grant

• Current Assignee: Toshiba Global Commerce Solutions Holdings Corporation (Toshiba Corporation)
• Original Assignee: Toshiba Global Commerce Solutions Holdings Corporation (Toshiba Corporation)
• Inventors: Herring, Dean Frederick, Singh, Ankit
• Primary Examiner(s): Sigmond, Bennett M
• Assistant Examiner(s): Anderson, Scott C
• Application Number:
  US14/659,128
• Publication Number:
  US 20160110760A1
• Time in Patent Office: 1,709 Days
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01B 11/02 for measuring length, width or thickness G01B11/08 takes precedence
• CPC Class Codes: G01G 19/40 with provisions for indicating, recording, or computing price or other quantities dependent on the weight indicating means for weighing apparatus G01G23/18; recording means for weighing apparatus G01G23/18; computers in general G06
• CPC Class Codes: G01G 19/4144 for controlling weight of goods in commercial establishments, e.g. supermarket, P.O.S. systems
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 20/18 involving self-service terminals [SST], vending machines, kiosks or multimedia terminals
• CPC Class Codes: G06Q 20/204 comprising interface for record bearing medium or carrier for electronic funds transfer or payment credit
• CPC Class Codes: G06Q 20/208 Input by product or record sensing, e.g. weighing or scanner processing
• CPC Class Codes: G06Q 20/209 Specified transaction journal output feature, e.g. printed receipt or voice output
• CPC Class Codes: G06Q 20/322 Aspects of commerce using mobile devices [M-devices]
• CPC Class Codes: G06Q 30/016 After-sales
• CPC Class Codes: G06Q 30/0224 based on user history
• CPC Class Codes: G06Q 30/0235 constrained by time limit or expiration date
• CPC Class Codes: G06Q 30/0251 : Targeted advertisements
• CPC Class Codes: G06Q 30/0269 : based on user profile or attribute
• CPC Class Codes: G06Q 30/0609 : Buyer or seller confidence or verification
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0629 : for generating comparisons
• CPC Class Codes: G06Q 30/0631 : Item recommendations
• CPC Class Codes: G06Q 30/0633 : Lists, e.g. purchase orders, compilation or processing
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06Q 30/0639 : Item locations
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 7/66 : of image moments or centre of gravity
• CPC Class Codes: G06V 20/68 : Food, e.g. fruit or vegetables
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/179 : metadata assisted face recognition
• CPC Class Codes: G07G 1/0045 : with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
• CPC Class Codes: G07G 1/0054 : with control of supplementary check-parameters, e.g. weight or number of articles
• CPC Class Codes: G07G 1/0063 : with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
• CPC Class Codes: G07G 1/0072 : with means for detecting the weight of the article of which the code is read, for the verification of the registration weighing combined with recording means, e.g. in weighing scale, G01G19/415
• CPC Class Codes: G07G 3/003 : Anti-theft control theft alarms G08B13/00
• CPC Class Codes: H04N 23/00 : Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

Method, computer program product, and system to influence a person within an environment having a plurality of items for selection. The method includes determining, using a first visual sensor disposed within the environment, field of view (FOV) information for the person, and analyzing the FOV information to identify one or more first items within the FOV of the person. The method further includes determining, using a second visual sensor disposed within the environment, behavior information for the person and analyzing the behavior information to classify a behavior of the person into one of a number of predefined behavior types. The method further includes selecting, based on the identified one or more first items and the classified behavior, at least a second item for presentation to the person, and presenting information related to the second item to the person.

Claims

1. A computer-implemented method to influence a person within an environment having a plurality of items, the method comprising:

determining, using a first visual sensor disposed within the environment, field of view (FOV) information for the person, the FOV information relating to one or more images captured using the first visual sensor;

analyzing, at a server comprising one or more computer processors and communicatively coupled with the first visual sensor using a communication network, the FOV information to identify one or more first items of the plurality of items that are disposed within a FOV of the person by;

analyzing a first image of the one or more images, using the one or more computer processors, to identify a body part of the person, the body part comprising at least one of a head of the person, a shoulder of the person, and an eye of the person;

determining, using the one or more computer processors, an angle formed by a position of the body part relative to a reference direction;

analyzing a second image of the one or more images, using the one or more computer processors and based at least in part on the angle, to identify a portion of the second image comprising the FOV of the person; and

identifying, using the one or more computer processors, one or more first items disposed within the portion of the second image;

determining, at the server using a second visual sensor disposed within the environment and communicatively coupled with the server, and based on analyzing a third image captured using the second visual sensor, behavior information for the person relating to one of a number of predefined behavior types;

updating, using the one or more computer processors, an interest score of the person for the one or more first items based on the behavior information of the person;

selecting, based on the identified one or more first items and the behavior information, at least a second item of the plurality of items for presentation to the person;

using the interest score of the person, determining content of information related to the second item that should be subsequently presented to the person; and

transmitting the information related to the second item to a display device communicatively coupled with the one or more computer processors.

2. The computer-implemented method of claim 1, wherein determining FOV information for the person is performed during a first time period, and determining behavior information for the person is performed during a second time period that begins after the first time period begins.

3. The computer-implemented method of claim 1, further comprising:

updating a profile associated with the person based on the behavior information, wherein the profile includes information for correlating the interest score of the person to one or more of the number of predefined behavior types.

4. The computer-implemented method of claim 3, wherein the predefined behavior types include at least one of handling one or more of the first items, reading one or more of the first items, selecting one or more of the first items, returning one or more of the first items, and declining to approach an area within the environment containing the one or more first items.

5. The computer-implemented method of claim 1, wherein the first visual sensor is worn on the body of the person.

6. The computer-implemented method of claim 1, wherein one or more component subscores are included in the interest score of the person, wherein each of the one or more subscores corresponds to a respective behavior type of the predefined behavior types, the method further comprising:

updating a first subscore of the one or more component subscores, wherein the first subscore corresponds to the behavior information.

7. The computer-implemented method of claim 1, wherein determining the content of the information related to the second item comprises:

responsive to determining a decrease in the interest score of the person, performing one of (i) increasing a promotional discount for the second item and (ii) changing formatting information associated with the second item.

8. The computer-implemented method of claim 1, wherein the one or more computer processors are configured to calculate the interest score according to a predefined time-based function.

9. The method of claim 1, wherein analyzing the FOV information to identify one or more first items of the plurality of items that are disposed within the FOV of the person further comprises:

analyzing the first image, using the computer processors, to identify an arm of the person; and

determining, using the computer processors, a second angle formed by a position of the arm relative to the reference direction, wherein identifying the portion of the second image comprising the FOV of the person is further based on the second angle.

10. The method of claim 1, wherein the reference direction is defined based on an item in the environment of the person.

11. The method of claim 10, wherein the item in the environment of the person is at least one of:

a floor, a ceiling, and a shelving unit.

12. A computer program product to influence a person within an environment having a plurality of items, the computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation that includes;

determining, using a first visual sensor disposed within the environment, field of view (FOV) information for the person, the FOV information relating to one or more images captured using the first visual sensor;

analyzing, at a server communicatively coupled with the first visual sensor using a communication network, the FOV information to identify one or more first items of the plurality of items that are disposed within a FOV of the person by;

analyzing, using the server, a first image of the one or more images to identify a body part of the person, the body part comprising at least one of a head of the person, a shoulder of the person, and an eye of the person;

determining, using the server, an angle formed by a position of the body part relative to a reference direction;

analyzing, using the server, a second image of the one or more images, based at least in part on the angle, to identify a portion of the second image comprising the FOV of the person; and

identifying, using the server, one or more first items disposed within the portion of the second image;

determining, at the server using a second visual sensor disposed within the environment and communicatively coupled with the server, and based on analyzing a third image captured using the second visual sensor, behavior information for the person relating to one of a number of predefined behavior types;

updating an interest score of the person for the one or more first items based on the behavior information of the person;

selecting, based on the identified one or more first items and the behavior information, at least a second item of the plurality of items for presentation to the person;

using the interest score of the person, determining content of information related to the second item that should be subsequently presented to the person; and

transmitting the information related to the second item to a display device communicatively coupled with the one or more computer processors.

13. The computer program product of claim 12, wherein determining FOV information for the person is performed during a first time period, and determining behavior information for the person is performed during a second time period that begins after the first time period begins.

14. The computer program product of claim 12, wherein the operation further includes:

updating a profile associated with the person based on the behavior information, wherein the profile includes information for correlating the interest score of the person to one or more of the number of predefined behavior types and wherein the predefined behavior types include at least one of handling one or more of the first items, reading one or more of the first items, selecting one or more of the first items, returning one or more of the first items, and declining to approach an area within the environment containing the one or more first items.

15. The computer program product of claim 12, wherein the first visual sensor is worn on the body of the person.

16. A system to influence a person within an environment having a plurality of items for selection, the system comprising:

a server comprising one or more computer processors;

one or more visual sensors disposed within the environment and communicatively coupled with the server using a communication network;

one or more display devices disposed within the environment and communicatively coupled with the one or more computer processors; and

a memory containing program code which, when executed by the one or more computer processors, performs an operation comprising;

determining, using a first visual sensor of the one or more visual sensors, field of view (FOV) information for the person, the FOV information relating to one or more images captured using the first visual sensor of the one or more visual sensors;

analyzing the FOV information to identify one or more first items of the plurality of items that are disposed within the FOV of the person by;

analyzing a first image of the one or more images to identify a body part of the person, the body part comprising at least one of a head of the person, a shoulder of the person, and an eye of the person;

determining an angle formed by a position of the body part relative to a reference direction;

analyzing a second image of the one or more images, based at least in part on the angle, to identify a portion of the second image comprising the FOV of the person; and

identifying one or more first items disposed within the portion of the second image;

determining, using a second visual sensor of the one or more visual sensors, and based on analyzing a third image captured using the second visual sensor, behavior information for the person relating to one of a number of predefined behavior types;

updating an interest score of the person for the one or more first items based on the behavior information of the person;

selecting, based on the identified one or more first items and the behavior information, at least a second item of the plurality of items for presentation to the person;

using the interest score of the person, determining content of information related to the second item that should be subsequently presented to the person; and

transmitting the information related to the second item to the one or more display devices.

17. The system of claim 16, wherein the one or more visual sensors comprise a visual sensor included in a body-worn computing device of the person, and wherein the FOV information for the person is determined using the visual sensor included in the body-worn computing device.

18. The system of claim 17, wherein the one or more visual sensors further comprises a plurality of visual sensors distributed throughout the environment, and wherein the behavior information for the person is determined using the plurality of visual sensors.

19. The system of claim 17, wherein the one or more display devices comprises a first display device included in one of a handheld computing device of the person and the body-worn computing device of the person.

20. The system of claim 16, wherein the predefined behavior types include at least one of handling one or more of the first items, reading one or more of the first items, selecting one or more of the first items, returning one or more of the first items, and declining to approach an area within the environment containing the one or more first items.