Controlling a document based on user behavioral signals detected from a 3D captured image stream

• US 7,877,706 B2
• Filed: 01/12/2007
• Issued: 01/25/2011
• Est. Priority Date: 01/12/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Goldman, Steven D., Sessa, Anthony J., Arnold, Kenneth C., Paolini, Michael A., Albertson, Jacob C.
• Primary Examiner(s): BONSHOCK, DENNIS G
• Application Number:
  US11/622,690
• Publication Number:
  US 20080170748A1
• Time in Patent Office: 1,474 Days
• Field of Search: 715/863, 715/865
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A computer-implemented method, system, and program product comprises a behavior processing system for capturing a three-dimensional movement of a user within a particular environment, wherein the three-dimensional movement is determined by using at least one image capture device aimed at the user. The behavior processing system identifies a three-dimensional object properties stream using the captured movement. The behavior processing system identifies a particular defined behavior of the user from the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with multiple behavior definitions each representing a separate behavioral signal for directing control of the document. A document control system selects at least one document element to represent the at least one particular defined behavior and inserts the selected document element into the document.

Claims

1. A computer-implemented method for controlling a document based on user behavior, comprising:

capturing a three-dimensional movement of a user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the user;

identifying a three-dimensional object properties stream using the captured movement;

identifying at least one particular defined behavior of the user from the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions each representing a separate behavioral signal which triggers control of the document;

selecting a position within the document to insert at least one document element based on at least one of a captured time stamp assigned to the captured three-dimensional object properties stream matching a document time stamp textually located within the document and a function performed by an application on the document triggered by the at least one particular defined behavior creating a position for inserting within the document;

creating at least one document element comprising at least one of a graphical element and a textual element to represent a translation of the at least one particular defined behavior into the at least one of the graphical element and the textual element by

transcribing, by a transcript controller, a plurality of web conference streams comprising a plurality of timestamped entries received from a plurality of computer systems over a network for a plurality of participants in a web conference combined into the document comprising a real time transcript of the combined plurality of web conference streams;

receiving, by the transcript controller, from a particular computer system logged into by a particular participant of the plurality of participants, a document control signal comprising the at least one particular defined behavior and captured time stamp;

selecting, by the transcript controller, the position within the document to insert the at least one document element based on the captured time stamp in relation to the plurality of timestamps textually specified in the transcript comprising a plurality of ordered timestamp entries; and

creating the at least one document element comprising a new entry in the transcript comprising a first textual element for the captured time stamp, a second textual element representing an identifier for the particular participant, and a third textual element translating the at least one particular defined behavior into text; and

inserting the selected document element into the document at the position by inserting the new entry in the transcript at the position.

2. The method according to claim 1, wherein capturing a three-dimensional movement of the user further comprises capturing the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the user.

3. The method according to claim 1, wherein capturing a three-dimensional movement of the user further comprises capturing at least one image frame of a user at a first point in time and comparing at least one additional image frame from a second point in time with the image frame from the first point in time.

4. The method according to claim 1, wherein identifying at least one particular defined behavior representing a particular behavioral signal from the three-dimensional object properties stream further comprises calculating a percentage probability that the captured three-dimensional movement represents the at least one particular defined behavior.

5. The method according to claim 1, wherein creating at least one document element comprising at least one of a graphical element and a textual element to represent a translation of the at least one particular defined behavior into the at least one of the graphical element and the textual element further comprises:

translating the at least one particular defined behavior into at least one programming code element, programming code equation, and programming code module.

6. The method according to claim 1, further comprising:

identifying the at least one particular defined behavior of the user from the three-dimensional object properties stream specifying a behavior of the user physically pointing at another person in a physical environment with the user with at least one finger from the plurality of behavior definitions;

identifying the another person physically pointed to by the user by comparing a facial mapping of the another person within the three-dimensional object properties stream with a database of a plurality of facial mappings each assigned a separate user identification;

selecting a particular function of finding an electronic mail address for the another person pointed to by the user;

triggering an electronic mail application to perform the particular function to identify the electronic mail address for the person pointed to;

selecting the position within the document to insert the at least one document element within an address block of a document comprising an electronic mail from among a primary addressee position and a carbon copy addressee position based on a number of the at least one finger detected in the particular defined behavior;

creating the at least one document element comprising the textual element comprising the electronic mail address;

inserting the at least one document element into the electronic mail in the position.

7. A system for controlling a document based on user behavior, comprising:

a behavior processing system comprising a processor connected to a memory communicatively connected to a network, further comprising;

means for capturing a three-dimensional movement of a user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the user;

means for identifying a three-dimensional object properties stream using the captured movement;

means for identifying at least one particular defined behavior of the user from the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions each representing a separate behavioral signal which triggers control of the document; and

a document control system comprising another processor connected to another memory communicatively connected to the behavior processing system via the network for selecting a position within the document to insert at least one document element based on at least one of a captured time stamp assigned to the captured three-dimensional object properties stream matching a document time stamp textually located within the document and a function performed by an application on the document triggered by the at least one particular defined behavior creating a position for inserting within the document;

the document control system comprising a transcript controller for creating at least one document element comprising at least one of a graphical element and a textual element to represent a translation of the at least one particular defined behavior into the at least one of the graphical element and the textual element by;

transcribing a plurality of web conference streams comprising a plurality of timestamped entries received from a plurality of computer systems over a network for a plurality of participants in a web conference combined into the document comprising a real time transcript of the combined plurality of web conference streams;

receiving from a particular computer system logged into by a particular participant of the plurality of participants, a document control signal comprising the at least one particular defined behavior and captured time stamp;

selecting the position within the document to insert the at least one document element based on the captured time stamp in relation to the plurality of timestamps textually specified in the transcript comprising a plurality of ordered timestamp entries; and

creating the at least one document element comprising a new entry in the transcript comprising a first textual element for the captured time stamp, a second textual element representing an identifier for the particular participant, and a third textual element translating the at least one particular defined behavior into text; and

the document control system for inserting the selected document element into the document at the position by inserting the new entry in the transcript at the position.

8. The system according to claim 7, wherein the means for capturing a three-dimensional movement of the user further comprises means for capturing the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the user.

9. The system according to claim 7, wherein the means for capturing a three-dimensional movement of the user further comprises means for capturing at least one image frame of a user at a first point in time and means for comparing at least one additional image frame from a second point in time with the image frame from the first point in time.

10. The system according to claim 7, wherein the means for identifying at least one particular defined behavior representing a particular behavioral signal from the three-dimensional object properties stream further comprises means for calculating a percentage probability that the captured three-dimensional movement represents the at least one particular defined behavior.

11. The system according to claim 7, the document control system further comprising:

means for selecting the at least one document element by translating the at least one particular defined behavior into at least one programming code element, programming code equation, and programming code module.

12. The system according to claim 7, the document control system further comprising:

means for identifying the at least one particular defined behavior of the user from the three-dimensional object properties stream specifying a behavior of the user physically pointing at another person in a physical environment with the user with at least one finger from the plurality of behavior definitions;

means for identifying the another person physically pointed to by the user by comparing a facial mapping of the another person within the three-dimensional object properties stream with a database of a plurality of facial mappings each assigned a separate user identification;

means for selecting a particular function of finding an electronic mail address for the another person pointed to by the user;

means for triggering an electronic mail application to perform the particular function to identify the electronic mail address for the person pointed to;

means for selecting the position within the document to insert the at least one document element within an address block of a document comprising an electronic mail from among a primary addressee position and a carbon copy addressee position based on a number of the at least one finger detected in the particular defined behavior;

means for creating the at least one document element comprising the textual element comprising the electronic mail address;

means for inserting the at least one document element into the electronic mail in the position.

13. A program product comprising a storage-type computer-usable medium including a computer-readable program for controlling a document based on user behavior, wherein the computer-readable program when executed on a computer cause the computer to:

capture a three-dimensional movement of a user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the user;

identify a three-dimensional object properties stream using the captured movement;

identify at least one particular defined behavior of the user from the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of behavior definitions each representing a separate behavioral signal which triggers control of the document;

select a position within the document to insert at least one document element based on at least one of a captured time stamp assigned to the captured three-dimensional object properties stream matching a document time stamp textually located within the document and a function performed by an application on the document triggered by the at least one particular defined behavior creating a position for inserting within the document;

create at least one document element comprising at least one of a graphical element and a textual element to represent a translation of the at least one particular defined behavior into the at least one of the graphical element and the textual element by;

transcribing a plurality of web conference streams comprising a plurality of timestamped entries received from a plurality of computer systems over a network for a plurality of participants in a web conference combined into the document comprising a real time transcript of the combined plurality of web conference streams;

receiving from a particular computer system logged into by a particular participant of the plurality of participants, a document control signal comprising the at least one particular defined behavior and captured time stamp;

selecting the position within the document to insert the at least one document element based on the captured time stamp in relation to the plurality of timestamps textually specified in the transcript comprising a plurality of ordered timestamp entries; and

creating the at least one document element comprising a new entry in the transcript comprising a first textual element for the captured time stamp, a second textual element representing an identifier for the particular participant, and a third textual element translating the at least one particular defined behavior into text; and

insert the selected document element into the document at the position by inserting the new entry in the transcript at the position.

14. The program product of claim 13, wherein said computer-readable program is transmitted over a network.