Translator for infants and toddlers

• US 20020084902A1
• Filed: 12/29/2000
• Published: 07/04/2002
• Est. Priority Date: 12/29/2000
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Zadrozny, Wlodek W., Kanevsky, Dimitri
• Granted Patent: US 7,143,044 B2
• Time in Patent Office: Days
• US Class Current: 340/573.1
• CPC Class Codes: A61B 5/0002 Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network A61B5/07 takes precedence; transmission systems for measured values G08C; transmission H04B; transmission of digital information H04L; wireless communication networks H04W
• CPC Class Codes: A61B 5/1123 Discriminating type of movement, e.g. walking or running A61B5/1116, A61B5/112 take precedence
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/411 Detecting or monitoring allergy or intolerance reactions to an allergenic agent or substance
• CPC Class Codes: A61B 5/7232 involving compression of the physiological signal, e.g. to extend the signal recording period
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system and method of interpreting communicative behaviors including actions, gestures and sounds produced by individuals, the system including a database of stored behavior codes, each behavior code having an associated explanation; means for accessing the database; means for recording a behavior performed by the individual; means for assigning an observed behavior code to the recorded behavior; means for comparing the observed behavior code to the stored behavior codes in the database of stored behavior codes; and means for retrieving the explanation associated with the stored behavior codes which match the observed behavior code. In the preferred embodiment the system further comprises means for recognizing events and correlating recognized events to the individual'"'"'s behavior'"'"'s.

Claims

1. A method for interpreting nonverbal communicative behavior of an individual, the method comprising the steps of:

(a) observing and recording a behavior performed by the individual;

(b) acquiring access to a collection of behavior codes, each behavior code having an associated explanation;

(c) assigning an observed behavior code to the observed behavior;

(d) comparing the observed behavior code to the collection of behavior codes;

(e) retrieving the explanation associated with each behavior code of the collection which matches the observed behavior code.

2. The method according to claim 1, further comprising the step of storing the observed behavior.

3. The method according to claim 2, wherein the recorded behavior is stored in segments.

4. The method according to claim 3, wherein a selected segment is transmitted by a user via a user interface to a computer, wherein the computer assigns the observed behavior code, searches the collection of behavior codes for matches by comparing the observed behavior code to the collection of behavior codes, and retrieves the explanation associated with each behavior code which matches the observed behavior code.

5. The method according to claim 2, wherein the recording is comprised of a string of frames, and the selected segment received by the computer is compressed into a compressed file by sampling the frames of the selected segment, and digitizing and formatting the sampled frames.

6. The method according to claim 5, wherein the recording is further comprised of a string of audio frames and image frames.

7. The method according to claim 5, wherein recorded inactivity is filtered out and not included in the digital compressed file.

8. The method according to claim 1, wherein the step of assigning an observed behavior code to the observed behavior further comprises:

recognizing behavior units which comprise the behavior;

assigning a behavior unit code to each behavior unit;

combining the behavior unit codes to form an observed behavior code.

9. The method according to claim 8, wherein the behavior units comprise gesture units and sound units;

and wherein the behavior unit codes comprise gesture unit codes and sound unit codes.

10. The method according to claim 9, wherein the gesture unit codes and sound unit codes are synchronized to form the observed behavior code.

11. The method according to claim 9, wherein the behavior unit codes are concatenated in chronological order, and each behavior unit code indicating when the associated behavior unit occurred simultaneously with the behavior unit associated with the preceding behavior unit code.

12. The method according to claim 1, wherein when a match is not found for the observed behavior code, the observed behavior code is added to the collection of stored behavior codes as a new code.

13. The method according to claim 12, wherein the collection of stored behavior codes is a database.

14. The method according to claim 13, wherein an ID is associated with a user and an ID is associated with the individual, and the user has the option of allowing the user ID and the individual ID to be entered into the database in association with a new code, when a new code is added, and in association with any matching stored behavior code in the behavior code database.

15. The method according to claim 8, wherein the behavior units comprising the observed behavior are prioritized according to one of:

chronological order, frequency of occurrence in database and degree of dominance.

16. The method according to claim 15, wherein the step of comparing the observed behavior code to the collection of behavior codes further comprises:

searching the collection of behavior codes for behavior codes having behavior units matching the behavior units of the observed behavior code, in order of the priority of the behavior unit codes in the observed behavior code.

17. The method according to claim 16, wherein the step of comparing the observed behavior code to the collection behavior codes further comprises:

synchronizing the comparison of the gesture units and the sound units of the observed behavior to the gesture units and the sound units of the stored behaviors.

19. The method according to claim 18, wherein the collection of behavior codes and the collection of behavior pattern codes are acquired according to past behaviors and behavior patterns performed by the individual.

20. The method according to claim 18, wherein the collection of behavior codes and the collection of behavior pattern codes are acquired according to past behaviors and behavior patterns performed by multiple individuals.

21. The method according to claim 18, wherein the method further comprises the steps of:

recognizing events;

assigning an event code to each recognized event; and

correlating recognized events to one of behaviors and behavior patterns.

22. The method according to claim 21, wherein an event causes an influence upon the behavior of the individual and comprises at least one of:

a) presence of an object, b) movement of an object, c) movement of the individual relative to an object, d) an occurrence in the schedule of the individual, e) an occurrence expected to take place in the schedule of the individual, and f) a detected change in the condition of the individual.

23. The method according to claim 21, wherein the step of correlating recognized events comprises:

associating event data with each behavior code, wherein the event data comprises the event code of each event which has been correlated with the behavior associated with the behavior code, the time of the occurrence of the event, the number of occurrences of the event; and

updating the event data associated with the behavior code in the collection of behavior codes which matches an observed behavior code upon the recognition of an event occurring within a selected time interval preceding and following the performance of the behavior.

24. The method according to claim 23, wherein the step of correlating recognized events further comprises:

analyzing event data for a selected behavior code from the collection of behavior codes, to determine which event occurrences are related to a behaviors represented by the selected behavior code according to the number of occurrences and the time interval between the event and the occurrence of the behavior.

25. The method according to claim 23, wherein the step of correlating recognized events further comprises:

determining which observed behaviors performed by an individual occur within a selected time interval of one another repetitively, for establishing a behavior pattern;

comparing correlated events for each of the observed behaviors included in the behavior pattern, and determining which of the correlated events are the same.

26. The method according to claim 18, wherein the method further comprises the steps of:

recognizing the occurrence of an event which could pose a danger to the individual;

transmitting an alert upon the recognition of a danger posing event.

27. The method according to claim 26, further comprising the steps of:

associating a danger alert to behavior codes in the collection of behavior codes for behavior codes associated with behaviors which indicate that an individual is in danger;

upon comparing observed behavior codes associated with behaviors observed following a danger posing event occurred, retrieving existing danger alerts for each matched behavior code; and

transmitting an alert upon retrieval of a danger alert.

30. The system according to claim 29, wherein the received recording is a file of a series of frames;

and wherein

the means for assigning a recorded behavior code to the recorded behavior comprises;

a Digitize and Sample Module for sampling the frames of the received recording at regular intervals and forming the sampled frames into a digital compressed formatted File of Sampled Frames with each sampled frame having a time stamp indicative of the relative time it was recorded.

31. The system according to claim 30, wherein the Analysis Provider Server further comprises an Individual Personal Data database for storing data about the individual, the data comprising identification, age, body measurements and voice characteristics.

32. The system according to claim 31, wherein the Digitize and Sample Module is in communication with a Scaling Module for applying a scaling factor selected according to the data stored in the Individual Personal Data database to the sampled frames of the digital compressed file for providing a standardized version of the sampled frames to the means for assigning a recorded behavior code to the recorded behavior.

33. The system according to claim 30, wherein the Digitize and Sample Module is in communication with a Noise Elimination Module for eliminating from the sampled frames recorded activity which is not meaningful to behaviors of the individual.

34. The system according to claim 30, wherein the means for assigning a code to the recorded behavior comprises:

a Behavior Coder (BC) Module for recognizing behavior units which comprise the behavior, assigning a behavior unit code to each behavior unit, and combining the behavior unit codes to form a recorded behavior code.

and wherein the known behavior codes are comprised of behavior units codes.

35. The system according to claim 34, wherein the behavior units comprise gesture units and sound units, and wherein the behavior unit codes comprise gesture unit codes and sound unit codes.

36. The system according to claim 35, wherein the BC Module is in communication with a Synchronizer Module for synchronizing the gesture unit codes and the sound unit codes to form the recorded behavior code.

37. The system according to claim 35, wherein the BC Module is in communication with a Behavior Begin/End Detector Module for determining the end of the recorded behavior and the beginning of a subsequent recorded behavior.

38. The system according to claim 35, wherein the BC Module concatenates the behavior unit codes in chronological order, with each behavior unit code indicating when the associated behavior unit occurred simultaneously with the behavior unit associated with the previous behavior unit code.

39. The system according to claim 35, wherein the behavior unit codes are comprised of property unit codes, wherein the property unit codes of a gesture unit code comprise a body part unit code, a movement type unit code, a body position unit code and a duration unit code.

40. The system according to claim 39, wherein the property unit codes of a sound gesture code comprise a sound description unit code, a sound intensity unit code, a sound pitch unit code and a duration period code.

41. The system according to claim 39, wherein the behavior unit codes comprising the recorded behavior code are prioritized according to one of:

chronological order, frequency of occurrence in database and degree of dominance.

42. The system according to claim 41, wherein the means for comparing the recorded behavior code with the known behavior codes stored in the database comprises:

a Behavior Code Matching (BCM) Module for searching the Behavior Code Database for behavior codes having behavior units having property units matching the property units of the observed behavior code.

43. The system according to claim 42, wherein the Behavior Code Database is searched according to the priority of the behavior unit codes and the property unit codes.

44. The system according to claim 41, wherein the means for comparing the recorded behavior code with the known behavior codes stored in the database synchronizes the comparison of the gesture unit codes and the sound unit codes of the recorded behavior code to the gesture unit codes and the sound unit codes of the stored behavior codes.

45. The system according to claim 42, wherein a user enters, via the user interface, a requested series of behavior codes;

and the Analysis Provider Server comprises a Behavior Series Matching (BCM) Module for determining when the individual performs a series of behaviors which correspond to the requested series of behavior codes.

46. The system according to claim 45, wherein the Analysis Provider Server further comprises a Web Server for receiving data from and transmitting data to the user interface.

47. The system according to claim 46, wherein the Analysis Provider Server further comprises a Subscriber Database for storing data relative to the users of the system comprising an identification number, personal data and financial data.

48. The system according to claim 47, wherein the Behavior Code Database comprises a record for each behavior code, each record comprising a field for storing:

the behavior code;

an Identification Number for the behavior code;

a link to the file of the received recording;

a link to the File of the Sampled Frames;

a primary explanation for the behavior code;

a link to a list of similar behavior codes;

a link to a list of related explanations for the behavior code with dates of entry and identification numbers for the submitting users;

recommendations related to the behavior code with dates of entry and identification numbers for the submitting users;

an identification number for the user who first submitted the behavior code;

an identification number for the individual; and

an indicator that the behavior code indicates that the individual is in danger.

49. The system according to claim 48, wherein the Analysis Provider Server further comprises an Individual Behavior Codes Database for storing records for at least one of behavior codes or series of behavior codes performed by the individual, wherein each record comprises a field for storing:

the behavior code;

a key number for the behavior code;

a link to the file of the received recording;

a link to the File of the Sampled Frames;

a primary explanation for the behavior code;

a link to a list of similar behavior codes;

a link to a list of related explanations for the behavior code;

recommendations related to the behavior code; and

an indicator that the behavior code indicates that the individual is in danger.

50. The system according to claim 49, wherein the Analysis Provider Server further comprises an Individual Event Database for storing events relating to the schedule, environment and condition of the individual, each event having an event code and a danger indicator which indicates if the event poses a danger to the individual.

51. The system according to claim 50, wherein the Analysis Provider Server further comprises a Situational Analyzer (SA) Module for correlating events to recorded and stored behavior codes and series of behavior codes;

wherein the events influence the behavior of the individual and comprise at least one of;

a) a presence of an object, b) movement of an object, c) movement of the individual relative to an object d) an occurrence in the schedule of the individual, e) an occurrence expected to take place in the schedule of the individual, or f) a detected change in the condition of the individual.

52. The system according to claim 51, wherein the Situational Analyzer Module comprises:

a General Event Database;

an Event Identifier/Tracker (EI/T) Module for tracking objects and recognizing the occurrence of events in the File of Sampled Frames; and

an Event/Behavior Correlator (E/BC) Module for correlating recognized events with behaviors recorded in the File of Sampled Frames.

53. The system according to claim 52, wherein the EI/T Module recognizes objects in the File of Sampled Frames and identifies the objects by looking them up in the Individual Event Database and the General Event Database.

54. The system according to claim 52, wherein the SA Module further comprises a Labeling Module for identifying events;

and

wherein the EI/T Module queries the Labeling Module when a recognized event is not identified, and upon receiving a successful identification of the event from the Labeling Module, the EI/T adds the event into at least one of the Individual Event Database or General Event Database.

55. The system according to claim 49 wherein each record at least one of the Behavior Code Database and the Individual Behavior Code Database further comprises a field for storing a link to event data, wherein the event data comprises an event code for each event which has been correlated with the behavior code, the time of the occurrence of each correlated event relative to the occurrence of the behavior associated with the behavior code, and the number of occurrences of each correlated event.

56. The system according to claim 52, wherein the E/BC Module correlates recognized events to the behavior recorded in the File of Sampled Frames by determining when a recognized event occurs within a selected time interval preceding and following the performance of the behavior;

and

updates the associated event data of the behavior code associated with the performed behavior in at least one of the Behavior Code Database and the Individual Behavior Code Database.

57. The system according to claim 52, wherein the Individual Event Database and the General Event Database include a field for each event which indicates when the occurrence of the event poses a threat to the individual;

and wherein the SA Module causes an alert to be transmitted to a caretaker of the individual upon recognition of the occurrence of danger posing event.

58. The system according to claim 51, wherein the SA Module further comprises a Behavior Pattern Recognizing (BPR) Module, wherein said BPR Module recognizes the repetitive occurrences of series of behavior codes for establishing behavior patterns and recognizes the occurrence of a behavior pattern performed by the individual.

59. The system according to claim 29, wherein the AP Server provides the user interface with interactive web pages to enable a user

to submit one of a live stream of recordings and a selected recorded segments;

and to enable the user to request that the AP Server recognize at least one of a specific behavior, a specific series of behaviors, a specific event, all behaviors and all repeated behavior pattern.

60. The system according to claim 59, wherein the AP server enables the user to request that the AP Server correlate at least one of a behavior to a specified event, an event to a specified behavior, an event to a specified series of behaviors, and correlation of events to all recognized behaviors.

61. The system according to claim 59, wherein the user specifies a behavior and a series of behaviors by entering, at a Web Page provided, one of the behavior codes for each behavior, the identification number for each behavior and behavior units comprising the behavior.

62. The system according to claim 50, wherein the user enters events into the Individual Event Database, at a Web Page provided, by one of:

entering descriptions of each event;

viewing the recording in a view window on a Web Page, selecting an object displayed in the view window and identifying the object; and

viewing the recording in a view window on a Web Page and entering the event which occurred at the time shown in the view window.

63. The system according to claim 29, wherein the Analysis Provider Server further comprises a Query Module for processing user requests to enter and access data stored in the AP Server.

64. The system according to claim 63, wherein the Analysis Provider Server further comprises a Report Generator Module for transmitting messages, results from submitted requests and alerts to the user interface.

67. The method according to claim 2, further comprising the step of:

submitting the stored observed behavior to a service provider, and wherein the service provider performs steps (b)-(e) for a fee.

18. A method for interpreting nonverbal communicative behavior of an individual, the method comprising the steps of:

observing a behavior pattern comprised of multiple behaviors performed by the individual;

acquiring access to a collection of behavior pattern codes, each behavior pattern code comprised of a string of behavior codes;

assigning a behavior pattern code to the behavior pattern, formed by stringing together the codes associated with the multiple behaviors;

comparing the observed behavior pattern code to the collection of behavior pattern codes;

reporting the results of the comparison.

28. A system for interpreting nonverbal communicative behavior of an individual, the system comprising:

a database of stored behavior codes, each behavior code having an associated explanation;

means for accessing the database;

means for recording a behavior performed by the individual;

means for assigning an observed behavior code to the recorded behavior;

means for comparing the observed behavior code to the stored behavior codes in the database of stored behavior codes; and

means for retrieving the explanation associated with the stored behavior codes which match the observed behavior code.

29. A system for interpreting nonverbal communicative behavior of an individual, the system comprising:

means for recording a behavior performed by the individual;

a user interface in communication with the recording means for receiving recordings; and

an Analysis Provider Server in communication with the user interface, said Analysis Provider Server receiving a recording of a behavior;

said Analysis Provider Server comprising;

means for assigning a recorded behavior code to the recorded behavior;

a Behavior Code Database for storing known behavior codes associated with known behaviors, each known behavior code having an associated explanation associated thereto; and

means for comparing the recorded behavior code with the known behavior codes in the Behavior Code Database and retrieving the explanation associated with each of the known behavior codes which match the recorded behavior code.

65. A system for interpreting nonverbal communicative behaviors of an individual comprising:

a camera for recording behaviors of an individual;

a first computer in communication with the camera for selecting a segment of the recorded behavior;

a second computer in communication with the first computer for prestoring a collection of coded definitions of behaviors, each coded definition having an identification code and an explanation;

receiving the selected segment from the first computer;

converting said segment into a digital format;

sampling said formatted segment at regular intervals;

storing said samples;

converting said samples of said formatted segment into a coded definition of said behavior;

comparing said coded definition of said received segment of the recorded behavior to the pre-stored collection of coded definitions of behaviors;

returning to said first computer the identification code and explanation of each of the pre-stored coded definitions which match the coded definition of the received segment; and

if no matches were found;

adding the definition code of the received segment to the collection of pre-stored coded definitions;

providing the added coded definition with an identification code and a blank explanation code; and

returning the identification code of the added coded definition to the first computer.

66. An article of manufacture comprising:

a computer usable medium having computer readable program code means embodied therein for interpreting a recorded nonverbal communicative behaviors of an individual, the computer readable program code means in said article of manufacture comprising;

computer readable program code means for assigning a label to the recorded behavior;

computer readable program code means for accessing a database of behavior labels, each behavior label having an associated explanation;

comparing the label for the recorded behavior to the labels in the database of behavior labels;

retrieving the explanation associated with each behavior label of the database which matches the recorded label;

updating the database by adding the recorded label to the database if a match was not found;

computer readable program code means for correlating events associated with and included in the recorded behavior to the recorded behavior; and

computer readable program code means for providing Graphic User Interface (GUI) for a user to submit recordings to the computer usable medium and request and receive information from the usable medium.