Controlling a system based on user behavioral signals detected from a 3D captured image stream

• US 7,801,332 B2
• Filed: 01/12/2007
• Issued: 09/21/2010
• Est. Priority Date: 01/12/2007
• Status: Expired due to Fees

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Goldman, Steven D., Sessa, Anthony J., Arnold, Kenneth C., Paolini, Michael A., Albertson, Jacob C.
• Primary Examiner(s): KIM, CHONG R
• Application Number:
  US11/622,693
• Publication Number:
  US 20080170749A1
• Time in Patent Office: 1,348 Days
• Field of Search: None
• US Class Current: 382/107
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A computer-implemented method, system, and program includes a behavior processing system for capturing a three-dimensional movement of a user within a particular environment, wherein the three-dimensional movement is determined by using at least one image capture device aimed at the user. The behavior processing system identifies a three-dimensional object properties stream using the captured movement. The behavior processing system identifies a particular defined behavior of the user representing a particular behavioral signal from the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with multiple behavior definitions each representing a separate behavioral signal for directing control of at least one machine. In response to identifying the particular defined behavior, a machine control system generates a control signal triggered by the particular behavioral signal for directing control of the at least one machine.

Claims

1. A computer-implemented method for generating control signals for a machine based on user behavior, comprising:

capturing a three-dimensional movement of a user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the user;

identifying a three-dimensional object properties stream using the captured movement;

identifying a location of the user within a background environment from among a plurality of background environments when the captured movement is captured;

identifying a particular defined behavior of the user representing a particular behavioral signal from the three-dimensional object properties stream and selecting at least one machine from among a plurality of machines by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions specified for the identified location of the user within the background environment and each representing a separate behavioral signal for directing control of the at least one machine from among the plurality of machines; and

in response to identifying the particular defined behavior of the user representing the particular behavioral signal from among the plurality of behavior definitions specified for the location of the user within the background environment, generating a control signal triggered by the particular behavioral signal for directing control of the at least one selected machine.

2. The method according to claim 1, wherein capturing a three-dimensional movement of the user further comprises capturing the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the user.

3. The method according to claim 1, wherein capturing a three-dimensional movement of the user further comprises capturing at least one image frame of a user at a first point in time and comparing at least one additional image frame from a second point in time with the image frame from the first point in time.

4. The method according to claim 1, wherein identifying a particular defined behavior representing a particular behavioral signal from the three-dimensional object properties stream further comprises calculating a percentage probability that the captured three-dimensional movement represents a particular behavior and represents the behavioral signal defined in the particular behavior definition.

5. The method according to claim 1, wherein identifying a particular defined behavior representing a particular behavioral signal from the three-dimensional object properties stream further comprises identifying at least one body movement of the user comprising a fine granularity movement of at least one of a facial expression, an eye movement, a muscle contraction, and change in skin surface characteristics of the user within the three-dimensional object properties stream matching at least one of the plurality of behavior definitions.

6. The method according to claim 1, wherein identifying a particular defined behavior representing a particular behavioral signal from the three-dimensional object properties stream further comprises identifying at least one behavior by the user signaling the user is experiencing a medical emergency, wherein the particular behavioral signal of a medical emergency directs control of the at least one machine for automatically generating a notification to a medical dispatch controller.

7. The method according to claim 1, wherein identifying a particular defined behavior of the user representing a particular behavioral signal from the three-dimensional object properties stream and selecting at least one machine from among a plurality of machines by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions specified for the location of the user within the background environment and each representing a separate behavioral signal for directing control of the at least one machine from among the plurality of machines further comprises:

detecting a current process of the at least one machine from at least one process performed by the at least one machine; and

identifying the particular defined behavior of the user representing a particular behavioral signal from the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the plurality of behavior definitions specified for the current process.

8. The method according to claim 1, wherein generating a control signal triggered by the particular behavioral signal for directing control of at least one machine further comprises:

sending the particular defined behavior to a machine control system for generating the control signal; and

translating, at the machine control system, the particular defined behavior representing the particular behavioral signal into at least one control signal for directing control of the at least one machine from among a plurality of machines controllable by the machine control system.

9. A system for generating control signals for a machine based on user behavior, comprising:

a behavior processing system communicative connected to a network, further comprising;

means for capturing a three-dimensional movement of a user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the user;

means for identifying a three-dimensional object properties stream using the captured movement;

means for identifying a location of the user within a background environment from among a plurality of background environments when the captured movement is captured; and

means for identifying a particular defined behavior of the user representing a particular behavioral signal from the three-dimensional object properties stream and selecting at least one machine from among a plurality of machines by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions specified for the identified location of the user within the background environment and each representing a separate behavioral signal for directing control of the at least one machine from among the plurality of machines;

a machine control system communicatively connected to the behavior processing system via the network for generating a control signal triggered by the particular behavioral signal for directing control of the at least one selected machine.

10. The system according to claim 9, wherein the means for capturing a three-dimensional movement of the user further comprises means for capturing the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the user.

11. The system according to claim 9, wherein the means for identifying a particular defined behavior representing a particular behavioral signal from the three-dimensional object properties stream further comprises means for calculating a percentage probability that the captured three-dimensional movement represents a particular behavior and represents the behavioral signal defined in the particular behavior definition.

12. The system according to claim 9, wherein the means for identifying a particular defined behavior representing a particular behavioral signal from the three-dimensional object properties stream further comprises means for identifying at least one body movement of the user comprising a fine granularity movement of at least one of a facial expression, an eye movement, a muscle contraction, and change in skin surface characteristics of the user within the three-dimensional object properties stream matching at least one of the plurality of behavior definitions.

13. The system according to claim 9, wherein the means for identifying a particular defined behavior representing a particular behavioral signal from the three-dimensional object properties stream further comprises means for identifying at least one behavior by the user signaling the user is experiencing a medical emergency, wherein the particular behavioral signal of a medical emergency directs control of the at least one machine for automatically generating a notification to a medical dispatch controller.

14. The system according to claim 9, wherein the means for identifying a particular defined behavior of the user representing a particular behavioral signal from the three-dimensional object properties stream and selecting at least one machine from among a plurality of machines by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions specified for the location of the user within the background environment and each representing a separate behavioral signal for directing control of the at least one machine from among the plurality of machines further comprises:

means for detecting a current process of the at least one machine from at least one process performed by the at least one machine; and

means for identifying the particular defined behavior of the user representing a particular behavioral signal from the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the plurality of behavior definitions specified for the current process.

15. The system according to claim 9, wherein the machine control system for generating a control signal triggered by the particular behavioral signal for directing control of at least one machine further comprises:

means for translating the particular defined behavior representing the particular behavioral signal into at least one control signal for directing control of the at least one machine from among a plurality of machines controllable by the machine control system.

16. A program product comprising a non-transitory storage-type computer-readable medium including a computer-readable program for generating control signals for a machine based on user behavior, wherein the computer-readable program when executed on a computer cause the computer to:

capture a three-dimensional movement of a user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the user;

identify a three-dimensional object properties stream using the captured movement;

identify a location of the user within a background environment from among a plurality of background environments when the captured movement is captured;

identify a particular defined behavior of the user representing a particular behavioral signal from the three-dimensional object properties stream and selecting at least one machine from among a plurality of machines by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions specified for the identified location of the user within the background environment and each representing a separate behavioral signal for directing control of the at least one machine from among the plurality of machines; and

in response to identifying the particular defined behavior of the user representing the particular behavioral signal from among the plurality of behavior definitions specified for the location of the user within the background environment, generate a control signal triggered by the particular behavioral signal or directing control of of the at least one selected machine.

17. The program product of claim 16, wherein the computer-readable program when executed on the computer further causes the computer to capture the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the user.

18. The program product of claim 16, wherein the computer-readable program when executed on the computer further causes the computer to calculate a percentage probability that the captured three-dimensional movement represents a particular behavior and represents the behavioral signal defined in the particular behavior definition.

19. The program product of claim 16, wherein the computer-readable program when executed on the computer further causes the computer to identify at least one behavior by the user signaling the user is experiencing a medical emergency, wherein the particular behavioral signal of a medical emergency directs control of the at least one machine for automatically generating a notification to a medical dispatch controller.