Remote biometric monitoring system

• US 10,643,081 B2
• Filed: 10/24/2018
• Issued: 05/05/2020
• Est. Priority Date: 06/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Silberschatz, Paul Joseph, Janssens, David Carl, Shapiro, Evan David, Carlucci, Adam Daniel
• Primary Examiner(s): Jiang, Zaihan
• Application Number:
  US16/169,510
• Publication Number:
  US 20190065862A1
• Time in Patent Office: 559 Days
• Field of Search: 348 77
• CPC Class Codes: A61B 2503/04 Babies, e.g. for SIDS detection
• CPC Class Codes: A61B 2560/0242 adapted to measure environmental factors, e.g. temperature, pollution
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/0816 Measuring devices for examining respiratory frequency measuring frequency of electric signals G01R23/00
• CPC Class Codes: A61B 5/1103 Detecting eye twinkling
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/4806 Sleep evaluation A61B5/4821 takes precedence; devices for inducing sleep A61M21/02
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Remote biometric monitoring systems may include a digital camera having a digital sensor, a processor, and a memory, all enclosed in a common housing. The processor of the camera may locally execute one or more algorithms to perform computer vision analysis of captured images of a sleeping subject, thereby determining an activity state of the subject. The activity state may include a sleep state. One or more environmental control devices may be adjusted automatically by the system based at least in part on the activity state.

Claims

1. A system for remotely monitoring a sleeping subject, the system comprising:

a digital camera configured to capture images of a subject, the digital camera including a digital image sensor, one or more local processors in communication with the digital image sensor, and a memory, wherein the digital image sensor, the one or more local processors, and the memory are enclosed in a same housing, and wherein no part of the system is attached to the subject;

a set of instructions stored in the memory of the digital camera and executable locally by the one or more local processors to;

receive a plurality of time-sequenced images of the subject from the digital image sensor;

determine a position of a torso region of the subject, using an artificial intelligence module trained to determine a presence and a position of the subject in a subset of images of the time-sequenced plurality of images;

define a bounding box that identifies where the subject is located in the image;

determine a respiration pattern of the subject by performing a biometric analysis on the time-sequenced plurality of images using the position of the torso region within the bounding box determined by the artificial intelligence module, wherein;

determining the respiration pattern includes generating a time-varying number corresponding to respiration of the subject, wherein the time-varying number coincides with a depth and a length of each breath of the subject;

cause an indicator corresponding to the time-varying number on a remote notification device to be presented; and

analyze motion outside of the bounding box defined by the artificial intelligence module; and

flag a video clip for review based on greater than a threshold amount of motion being determined by analyzing the motion outside of the bounding box defined by the artificial intelligence module.

2. The system of claim 1, wherein the indicator comprises one or more light emitting diodes (LEDs).

3. The system of claim 2, wherein displaying the indicator comprises displaying a changing light pattern using the one or more LEDs.

4. The system of claim 1, wherein the set of instructions is further executable to identify, based on the biometric analysis, an activity state of the subject.

5. The system of claim 4, wherein the set of instructions is further executable to cause a change in an environmental control device based on the sensed input and the activity state of the subject, wherein the environmental control device is in communication with the one or more local processors of the digital camera, the environmental control device configured to produce a change in the local environment of the subject.

6. The system of claim 5, wherein causing the change in the environmental control device is further based on the respiration pattern of the subject.

7. The system of claim 4, wherein identifying the activity state of the subject includes identifying a sleep state of the subject.

8. The system of claim 4, wherein the remote notification device comprises a dedicated device configured to provide information relating to the activity state of the subject to a user located out of sight of the subject.

9. The system of claim 1, wherein the biometric analysis includes determining whether the subject is moving by performing an actigraphic analysis of the plurality of time-sequenced images of the subject.

10. The system of claim 1, wherein the subset of images comprises a video.

11. A system for remotely monitoring a sleeping person, the system comprising:

a digital camera configured to capture images of a person, the digital camera including a digital image sensor, one or more local processors in communication with the digital image sensor, and a memory, wherein the digital image sensor, the one or more local processors, and the memory are enclosed in a same housing, and wherein no portion of the system is attached to the person;

a remote notification device in communication with a user located out of sight of the person; and

a set of instructions stored in the memory of the digital camera and executable locally by the one or more local processors to;

receive a plurality of time-sequenced images of the person from the digital image sensor;

determine a position of a torso region of the person, using an artificial intelligence module trained to determine a presence and a position of the person, in at least one image of the time-sequenced plurality of images;

define a bounding box that identifies where the subject is located in the image;

determine a respiration pattern of the person by performing a biometric analysis on the plurality of time-sequenced images, wherein determining the respiration pattern includes generating a time-varying number corresponding to respiration of the person, such that the time-varying number coincides with a depth and a length of each breath of the person;

cause an indicator corresponding to the time-varying number on the remote notification device to be displayed;

analyze motion outside of the bounding box defined by the artificial intelligence module; and

flag a video clip for review based on greater than a threshold amount of motion being determined by analyzing the motion outside of the bounding box defined by the artificial intelligence module.

12. The system of claim 11, wherein the set of instructions is further executable to:

identify, using the position of the torso region determined by the artificial intelligence module, an activity state of the person; and

communicate information corresponding to the activity state of the person to the user via the remote notification device.

13. The system of claim 12, wherein the activity state of the person includes a sleep state.

14. The system of claim 12, wherein the set of instructions is further executable to cause a change in an environmental control device local to the person, based on the activity state of the person.

15. The system of claim 12, wherein the remote notification device comprises a dedicated device configured to provide information relating to the activity state of the person.

16. The system of claim 11, wherein the artificial intelligence module comprises a neural network.

17. The system of claim 11, wherein the biometric analysis includes determining a movement state of the person by performing an actigraphic analysis of the plurality of time-sequenced images.

18. The system of claim 11, wherein the respiration pattern comprises a respective duration of each inhalation and each exhalation.

19. The system of claim 11, wherein displaying the indicator comprises displaying a changing light pattern on the remote notification device.

20. The system of claim 19, wherein the indicator comprises one or more light emitting diodes (LEDs).