Translator for infants and toddlers

• US 7,143,044 B2
• Filed: 12/29/2000
• Issued: 11/28/2006
• Est. Priority Date: 12/29/2000
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Zadrozny, Wlodek W., Kanevsky, Dimitri
• Primary Examiner(s): Opsasnick, Michael N.
• Application Number:
  US09/751,504
• Publication Number:
  US 20020084902A1
• Time in Patent Office: 2,160 Days
• Field of Search: 704/270, 704/275, 704/276, 704/277
• US Class Current: 704/275
• CPC Class Codes: A61B 5/0002 Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network A61B5/07 takes precedence; transmission systems for measured values G08C; transmission H04B; transmission of digital information H04L; wireless communication networks H04W
• CPC Class Codes: A61B 5/1123 Discriminating type of movement, e.g. walking or running A61B5/1116, A61B5/112 take precedence
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/16 Devices for psychotechnics using teaching or educational appliances G09B1/00 - G09B7/00; Testing reaction times ; Devices for evaluating the psychological state
• CPC Class Codes: A61B 5/411 Detecting or monitoring allergy or intolerance reactions to an allergenic agent or substance
• CPC Class Codes: A61B 5/7232 involving compression of the physiological signal, e.g. to extend the signal recording period
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system and method of interpreting communicative behaviors including actions, gestures and sounds produced by individuals, the system includes a database of stored behavior codes, each behavior code having an associated explanation; the system can access the database, records a behavior performed by the individual, assigns an observed behavior code to the recorded behavior, compares the observed behavior code to the stored behavior codes in the database of stored behavior codes, and retrieves the explanation associated with the stored behavior codes which match the observed behavior code. The system can also recognize events and correlate recognized events to the individual'"'"'s behavior'"'"'s.

Claims

1. A system for interpreting nonverbal communicative behavior of an individual, the system comprising:

means for recording a behavior performed by the individual;

a user interface in communication with the recording means for receiving recordings; and

an Analysis Provider Server in communication with the user interface, said Analysis Provider Server receiving a recording of a behavior in the form of a file of a series of frames;

said Analysis Provider Server comprising;

means for assigning a recorded behavior code to the recorded behavior, said means for assigning including a Digitize and Sample Module for sampling the frames of the received recording at regular intervals and forming the sampled frames into a digital compressed formatted File of Sampled Frames with each sampled frame having a time stamp indicative of the relative time it was recorded, and a Behavior Coder (BC) Module for recognizing behavior units which comprise the behavior, assigning a behavior unit code to each behavior unit, and combining the behavior unit codes to form a recorded behavior code;

a Behavior Code Database for storing known behavior codes associated with known behaviors, each known behavior code having an associated explanation associated thereto and including the behavior unit codes; and

means for comparing the recorded behavior code with the known behavior codes in the Behavior Code Database and retrieving the explanation associated with each of the known behavior codes which match the recorded behavior code, said means for comparing including a Behavior Code Matching (BCM) Module for searching the Behavior Code Database for behavior codes having behavior units having property units matching the property units of the observed behavior code;

wherein the behavior units comprise gesture units and sound units, the behavior unit codes comprise gesture unit codes, sound unit codes, and property unit codes, and the property unit codes of a gesture unit code comprise a body part unit code, a movement type unit code, a body position unit code and a duration unit code, and the behavior unit codes comprising the recorded behavior code are prioritized according to one of;

chronological order, frequency of occurrence in database and degree of dominance;

and wherein a user enters, via the user interface, a requested series of behavior codes, and the Analysis Provider Server comprises a Behavior Series Matching (BCM) Module for determining when the individual performs a series of behaviors which correspond to the requested series of behavior codes.

2. The system according to claim 1, wherein the Digitize and Sample Module is in communication with a Noise Elimination Module for eliminating from the sampled frames recorded activity which is not meaningful to behaviors of the individual.

3. The system according to claim 1, wherein the BC Module is in communication with a Synchronizer Module for synchronizing the gesture unit codes and the sound unit codes to form the recorded behavior code.

4. The system according to claim 1, wherein the BC Module is in communication with a Behavior Begin/End Detector Module for determining the end of the recorded behavior and the beginning of a subsequent recorded behavior.

5. The system according to claim 1, wherein the BC Module concatenates the behavior unit codes in chronological order, with each behavior unit code indicating when the associated behavior unit occurred simultaneously with the behavior unit associated with the previous behavior unit code.

6. The system according to claim 1, wherein the property unit codes of a sound gesture code comprise a sound description unit code, a sound intensity unit code, a sound pitch unit code and a duration period code.

7. The system according to claim 1, wherein the Behavior Code Database is searched according to the priority of the behavior unit codes and the property unit codes.

8. The system according to claim 1, wherein the means for comparing the recorded behavior code with the known behavior codes stored in the database synchronizes the comparison of the gesture unit codes and the sound unit codes of the recorded behavior code to the gesture unit codes and the sound unit codes of the stored behavior codes.

9. The system according to claim 1, wherein the Analysis Provider Server further comprises an Individual Personal Data database for storing data about the individual, the data comprising identification, age, body measurements and voice characteristics.

10. The system according to claim 9, wherein the Digitize and Sample Module is in communication with a Scaling Module for applying a scaling factor selected according to the data stored in the Individual Personal Data database to the sampled frames of the digital compressed file for providing a standardized version of the sampled frames to the means for assigning a recorded behavior code to the recorded behavior.

11. The system according to claim 1, wherein the Analysis Provider Server further comprises a Query Module for processing user requests to enter and access data stored in the AP Server.

12. The system according to claim 11, wherein the Analysis Provider Server further comprises a Report Generator Module for transmitting messages, results from submitted requests and alerts to the user interface.

13. The system according to claim 1, wherein the AP Server provides the user interface with interactive web pages to enable a user

to submit one of a live stream of recordings and a selected recorded segments;

and to enable the user to request that the AP Server recognize at least one of a specific behavior, a specific series of behaviors, a specific event, all behaviors and all repeated behavior pattern.

14. The system according to claim 13, wherein the AP server enables the user to request that the AP Server correlate at least one of a behavior to a specified event, an event to a specified behavior, an event to a specified series of behaviors, and correlation of events to all recognized behaviors.

15. The system according to claim 13, wherein the user specifies a behavior and a series of behaviors by entering, at a Web Page provided, one of the behavior codes for each behavior, the identification number for each behavior and behavior units comprising the behavior.

16. The system according to claim 1, wherein the Analysis Provider Server further comprises a Web Server for receiving data from and transmitting data to the user interface.

17. The system according to claim 16, wherein the Analysis Provider Server further comprises a Subscriber Database for storing data relative to the users of the system comprising an identification number, personal data and financial data.

18. The system according to claim 17, wherein the Behavior Code Database comprises a record for each behavior code, each record comprising a field for storing:

the behavior code;

an Identification Number for the behavior code;

a link to the file of the received recording;

a link to the File of the Sampled Frames;

a primary explanation for the behavior code;

a link to a list of similar behavior codes;

a link to a list of related explanations for the behavior code with dates of entry and identification numbers for the submitting users;

recommendations related to the behavior code with dates of entry and identification numbers for the submitting users;

an identification number for the user who first submitted the behavior code;

an identification number for the individual; and

an indicator that the behavior code indicates that the individual is in danger.

19. The system according to claim 18, wherein the Analysis Provider Server further comprises an Individual Behavior Codes Database for storing records for at least one of behavior codes or series of behavior codes performed by the individual, wherein each record comprises a field for storing:

the behavior code;

a key number for the behavior code;

a link to the file of the received recording;

a link to the File of the Sampled Frames;

a primary explanation for the behavior code;

a link to a list of similar behavior codes;

a link to a list of related explanations for the behavior code;

recommendations related to the behavior code; and

an indicator that the behavior code indicates that the individual is in danger.

20. The system according to claim 19 wherein each record at least one of the Behavior Code Database and the Individual Behavior Code Database further comprises a field for storing a link to event data, wherein the event data comprises an event code for each event which has been correlated with the behavior code, the time of the occurrence of each correlated event relative to the occurrence of the behavior associated with the behavior code, and the number of occurrences of each correlated event.

21. The system according to claim 19, wherein the Analysis Provider Server further comprises an Individual Event Database for storing events relating to the schedule, environment and condition of the individual, each event having an event code and a danger indicator which indicates if the event poses a danger to the individual.

22. The system according to claim 21, wherein the user enters events into the Individual Event Database, at a Web Page provided, by one of:

entering descriptions of each event;

viewing the recording in a view window on a Web Page, selecting an object displayed in the view window and identifying the object; and

viewing the recording in a view window on a Web Page and entering the event which occurred at the time shown in the view window.

23. The system according to claim 21, wherein the Analysis Provider Server further comprises a Situational Analyzer (SA) Module for correlating events to recorded and stored behavior codes and series of behavior codes;

wherein the events influence the behavior of the individual and comprise at least one of;

a) a presence of an object, b) movement of an object, c) movement of the individual relative to an object d) an occurrence in the schedule of the individual, e) an occurrence expected to take place in the schedule of the individual, or f) a detected change in the condition of the individual.

24. The system according to claim 23, wherein the SA Module further comprises a Behavior Pattern Recognizing (BPR) Module, wherein said BPR Module recognizes the repetitive occurrences of series of behavior codes for establishing behavior patterns and recognizes the occurrence of a behavior pattern performed by the individual.

25. The system according to claim 23, wherein the Situational Analyzer Module comprises:

a General Event Database;

an Event Identifier/Tracker (EI/T) Module for tracking objects and recognizing the occurrence of events in the File of Sampled Frames; and

an Event/Behavior Correlator (E/BC) Module for correlating recognized events with behaviors recorded in the File of Sampled Frames.

26. The system according to claim 25, wherein the EI/T Module recognizes objects in the File of Sampled Frames and identifies the objects by looking them up in the Individual Event Database and the General Event Database.

27. The system according to claim 25, wherein the SA Module further comprises a Labeling Module for identifying events;

and

wherein the EI/T Module queries the Labeling Module when a recognized eyent is not identified, and upon receiving a successful identification of the event from the Labeling Module, the EI/T adds the event into at least one of the Individual Event Database or General Event Database.

28. The system according to claim 25, wherein the E/BC Module correlates recognized events to the behavior recorded in the File of Sampled Frames by determining when a recognized event occurs within a selected time interval preceding and following the performance of the behavior;

and

updates the associated event data of the behavior code associated with the performed behavior in at least one of the Behavior Code Database and the Individual Behavior Code Database.

29. The system according to claim 25, wherein the Individual Event Database and the General Event Database include a field for each event which indicates when the occurrence of the event poses a threat to the individual;

and wherein the SA Module causes an alert to be transmitted to a caretaker of the individual upon recognition of the occurrence of danger posing event.