Infant monitoring system

• US 10,447,972 B2
• Filed: 03/27/2017
• Issued: 10/15/2019
• Est. Priority Date: 07/28/2016
• Status: Active Grant

• Current Assignee: Cradlewise, Inc.
• Original Assignee: Chigru Innovations (OPC) Private Limited
• Inventors: Patil, Radhika
• Primary Examiner(s): Dang, Philip P.
• Application Number:
  US15/469,586
• Publication Number:
  US 20180035082A1
• Time in Patent Office: 932 Days
• CPC Class Codes: A47D 15/00 Accessories for children's furniture, e.g. safety belts or baby-bottle holders
• CPC Class Codes: A61B 2503/04 Babies, e.g. for SIDS detection
• CPC Class Codes: A61B 5/002 Monitoring the patient using a local or closed circuit, e.g. in a room or building A61B5/0017 takes precedence
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/0816 Measuring devices for examining respiratory frequency measuring frequency of electric signals G01R23/00
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/45 For evaluating or diagnosing the musculoskeletal system or teeth A61B5/1036, A61B5/1074, A61B7/006 take precedence
• CPC Class Codes: A61B 5/6892 Mats
• CPC Class Codes: A61B 5/7278 Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: A61B 7/026 comprising more than one sound collector
• CPC Class Codes: A61B 7/04 Electric stethoscopes microphones, acoustic transducers therefor H04R
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30008 Bone
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 2207/30088 Skin; Dermal
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 5/20 : using local operators
• CPC Class Codes: G06T 7/0016 : involving temporal comparison
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/90 : Determination of colour characteristics
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/113 : Recognition of static hand signs
• CPC Class Codes: G06V 40/15 : Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/165 : using facial parts and geometric relationships
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/19 : Sensors therefor
• CPC Class Codes: G06V 40/193 : Preprocessing; Feature extraction
• CPC Class Codes: G08B 21/0202 : Child monitoring systems using a transmitter-receiver system carried by the parent and the child
• CPC Class Codes: G08B 21/0208 : Combination with audio or video communication, e.g. combination with "baby phone" function
• CPC Class Codes: G08B 21/0461 : integrated or attached to an item closely associated with the person but not worn by the person, e.g. chair, walking stick, bed sensor
• CPC Class Codes: G08B 21/0476 : Cameras to detect unsafe condition, e.g. video cameras
• CPC Class Codes: G08B 21/06 : indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: G10L 21/0208 : Noise filtering
• CPC Class Codes: G10L 21/0232 : Processing in the frequency domain
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: H04N 13/20 : Image signal generators
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 2013/0074 : Stereoscopic image analysis
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04R 2430/20 : Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 3/005 : for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/027 : Spatial or constructional arrangements of microphones, e.g. in dummy heads
• CPC Class Codes: Y02A 90/10 : Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Abstract

A monitoring system includes a sensor, a processing block and an alert generator. The sensor is used to generate images of an infant. The processing block processes one or more of the images and identifies a condition with respect to the infant. Then alert generator causes generation of an alert signal if the identified condition warrants external notification. In an embodiment, the sensor is a 3D camera, and the 3D camera, the processing block and the alert generator are part of a unit placed in the vicinity of the infant. The monitoring system further includes microphones and motion sensors to enable detection of sounds and movement.

Claims

1. A monitoring system comprising:

a processing block to process one or more images of an infant generated by a camera, and identify a condition with respect to said infant; and

an alert generator to cause generation of an alert signal if said identified condition warrants an external notification,

said processing block and said alert generator are part of a unit placed in a vicinity of said infant,

wherein said camera comprises;

a 3D (three dimensional) camera to generate said one or more images as one or more 3D (three-dimensional) images of said infant located in a field-of-view (FoV) of said 3D camera,

wherein said infant is placed on a surface within said FoV,

wherein said condition is a posture of said infant, said processing block further operable to;

extract skeletal information of said infant from said one or more 3D images;

determine body parts of interest from said skeletal information; and

determine said posture of said infant from said body parts of interest,

wherein said processing block identifies said posture to be one of;

a belly-up posture, wherein a nose and both eyes of said infant are determined to be visible, and the nose of said infant is determined to point away from said surface;

a side posture, wherein only one ear of said infant is determined to be visible, wherein a level of a head bone of said infant is determined to be below a nearest shoulder and arm bones of said infant; and

a belly-down posture, wherein said nose and eyes of said infant are determined not be visible, or wherein said nose is determined to be pointing towards said surface, or wherein a head of said infant is determined to be at a same level as a torso of said infant, and wherein a hip of the infant is above the torso of said infant, and wherein knees of said infant are lower than or at a same level as a rest of a leg portion of said infant,

wherein said alert generator generates said alert signal if said processing block identifies said posture of said infant to be said belly-down posture.

2. The monitoring system of claim 1, wherein said camera is mounted on a stationary platform, wherein said surface is swingable and is within said FoV even during a motion of said surface, said monitoring system further comprising:

a motion sensor to generate motion sensor signals representative of movements of said surface,

wherein said one or more images contain a common motion, wherein said common motion is a sum of movements of said infant alone and movements of said surface alone,

wherein said processing block is operable to remove from said common motion, components of motion due only to said movements of said surface alone based on said motion sensor signals received from said motion sensor.

3. The monitoring system of claim 1, wherein said extracting skeletal information, said determining body parts of interest and said determining said posture of said infant from said body parts of interest employ machine learning algorithms.

4. The monitoring system of claim 1, further comprising:

stereo microphones to generate audio signals representative of a sound in the vicinity of said infant, wherein said stereo microphones comprise a first microphone and a second microphone, wherein said first microphone and said second microphone are positioned such that the infant lies in a line that passes through each of said first microphone and said second microphone,

wherein said processing block is further operable to;

filter out an ambient noise from said audio signals to generate filtered audio signals;

determine if the eyes of said infant are open by processing a 2D (2-dimensional) image obtained by retaining only intensity components of pixels in one or more of said 3D images;

collate data from said skeletal information, said filtered audio signals, filtered motion signals, and

ascertain said condition from said collated data, wherein at least some operations of an ascertaining employs machine learning algorithms,

wherein said condition is one of;

a first state in which the eyes of said infant are determined to be open for a period of time longer than a first threshold;

a second state in which said infant is crying;

a third state in which movements by said infant are determined to be present for a period of time longer than a second threshold; and

a fourth state in which said infant is determined to be in a standing or a sitting position on said surface,

wherein said alert generator generates said alert signal upon said ascertaining.

5. The monitoring system of claim 1, wherein said processing block is further operable to:

identify, from said posture of said infant, additional body parts exhibiting a breathing motion, wherein said additional body parts comprise the torso of said infant;

filter out motion components in the motion of said torso that are due to a motion of said surface to generate a filtered sequence of images;

determine moving parts in said filtered sequence, wherein said moving parts includes said torso of said infant;

generate a grouped moving parts based on a correlating motion of adjacent parts in said moving parts; and

determine a breath rate of said infant from said grouped moving parts,

wherein said alert generator generates said alert signal if said processing block determines that said breath rate falls outside of a pre-determined range.

6. The monitoring system of claim 1, wherein said processing block is further operable to:

identify, from said posture of said infant, additional body parts exhibiting a breathing motion, wherein said additional body parts comprise a face and a neck of said infant;

filter out motion components in the motion of said face and neck that are due to a motion of said surface to generate a filtered sequence of images; and

determine moving parts in said filtered sequence, wherein said moving parts includes said face and neck of said infant;

generate a grouped moving parts based on a correlating motion of adjacent parts in said moving parts;

process scene-to-scene changes in a set of 2D (2-dimensional) images to detect darkening and lightening of a skin tone of a skin of said infant, wherein said set of 2D images are obtained by retaining only intensity components of pixels in one of more of said 3D images;

filter out frequencies in said scene-to-scene changes that fall outside of a normal range of frequencies of a heart rate of an infant;

weighting and combining data representing said grouped moving parts and data representing said scene-to-scene changes; and

determining a heart rate based on an average motion of said grouped moving parts and said detection of darkening and lightening of skin tone,

wherein said alert generator generates said alert signal if said processing block determines that said heart rate falls outside a normal range.

7. The monitoring system of claim 1, wherein said condition is a presence of an insect in the vicinity of said infant.

8. The monitoring system of claim 1, wherein said processing block is further operable to:

detect, by processing said one or more 3D images, presence of a moving object between said 3D camera and said infant;

filtering out, from said one or more 3D images, objects that have a size that is outside of a normal range of sizes for insects; and

filtering out, from said one or more 3D images, moving objects that have a motion that 5 is determined not to resemble that of an insect,

wherein said alert generator generates said alert signal if said processing block detects presence of said insect.

9. The monitoring system of claim 1, wherein said alert signal is transmitted to a remote computing device.

10. The monitoring system of claim 2, wherein in an event of an alert, said processing block transmits a first set of data and a second set of data to said remote computing device for analysis,

wherein said first set of data are generated by said 3D camera and said motion sensor prior to said alert, and

wherein said second set of data are generated by said 3D camera and said motion sensor after said alert.

11. The monitoring system of claim 1, further comprising stereo microphones to generate audio signals representative of a sound in the vicinity of said infant, wherein said stereo microphones comprise a first microphone and a second microphone, wherein said first microphone and said second microphone are positioned such that said infant lies in a line that passes through each of said first microphone and said second microphone,

wherein said processing block processes said audio signals to determine if said infant is crying.

12. The monitoring system of claim 11, further comprising a rocking mechanism to rock said surface if said processing block determines that said infant is crying.

13. The monitoring system of claim 11, wherein said processing block is operable to initiate one or more actions based on one or more user actions,

wherein said one or more actions include control of lights installed on said unit, change songs played by a music system comprised in said unit, change a volume of music played by said music system, control an operation of said rocking mechanism,

wherein said one or more user actions comprise a gesture of opening a hand, a gesture of hand swiping, a gesture of rotation of a hand, a gesture of placing a hand in front of said 3D camera, and a gesture of to-and-fro hand wave.