Remote biometric monitoring system

• US 10,169,662 B2
• Filed: 06/14/2016
• Issued: 01/01/2019
• Est. Priority Date: 06/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Silberschatz, Paul Joseph, Janssens, David Carl, Shapiro, Evan David, Carlucci, Adam Daniel
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): Jiang, Zaihan
• Application Number:
  US15/182,515
• Publication Number:
  US 20160364617A1
• Time in Patent Office: 931 Days
• Field of Search: 348 77
• CPC Class Codes: A61B 2503/04 Babies, e.g. for SIDS detection
• CPC Class Codes: A61B 2560/0242 adapted to measure environmental factors, e.g. temperature, pollution
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/0816 Measuring devices for examining respiratory frequency measuring frequency of electric signals G01R23/00
• CPC Class Codes: A61B 5/1103 Detecting eye twinkling
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/4806 Sleep evaluation A61B5/4821 takes precedence; devices for inducing sleep A61M21/02
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Remote biometric monitoring systems may include a digital camera having a digital sensor, a processor, and a memory, all enclosed in a common housing. The processor of the camera may locally execute one or more algorithms to perform computer vision analysis of captured images of a sleeping subject, thereby determining an activity state of the subject. The activity state may include a sleep state. One or more environmental control devices may be adjusted automatically by the system based at least in part on the activity state.

Claims

1. A system for remotely monitoring a sleeping subject, the system comprising:

a digital camera configured to capture images of a subject, the digital camera including a digital image sensor, one or more local processors in communication with the digital image sensor, and a memory, wherein the digital image sensor, the one or more local processors, and the memory are enclosed in a same housing, and wherein no sensors or markers are attached to the subject;

a tripod support having the digital camera articulatably attached thereto, wherein the tripod support includes a pair of short legs each having a first permanent length, and a long leg having a second permanent length at least fifty percent longer than the first permanent length;

an environmental sensor in communication with the one or more local processors of the digital camera, the environmental sensor configured to detect a physical characteristic of a local environment of the subject;

an environmental control device in communication with the one or more local processors of the digital camera, the environmental control device configured to produce a change in the local environment of the subject; and

a set of instructions stored in the memory of the digital camera and executable locally by the one or more local processors to;

receive a sensed input from the environmental sensor and a plurality of time-sequenced images of the subject from the digital image sensor;

determine a position of a torso region of the subject using an artificial intelligence module trained to determine a presence and a position of the subject in a subset of images of the time-sequenced plurality of images;

identify, using the position of the torso region determined by the artificial intelligence module to perform a biometric analysis on an uncompressed version of the plurality of time-sequenced images, an activity state of the subject; and

cause a change in the environmental control device based on the sensed input and the activity state of the subject.

2. The system of claim 1, wherein the biometric analysis includes determining whether the subject is moving by performing an actigraphic analysis of the plurality of uncompressed, time-sequenced images of the subject.

3. The system of claim 1, wherein identifying the activity state of the subject includes identifying a sleep state of the subject.

4. The system of claim 1, wherein the subset of images consists of a single image.

5. The system of claim 1, wherein the biometric analysis further includes determining a respiration pattern of the subject.

6. The system of claim 5, wherein causing the change in the environmental control device is further based on the respiration pattern of the subject.

7. The system of claim 1, wherein the artificial intelligence module comprises a neural network.

8. The system of claim 1, further comprising a remote notification device in communication with the digital camera, the remote notification device configured to provide information relating to the activity state of the subject to a user located out of sight of the subject.

9. A system for remotely monitoring a sleeping person, the system comprising:

a digital camera configured to capture images of a person, the digital camera including a digital image sensor, one or more local processors in communication with the digital image sensor, and a memory, wherein the digital image sensor, the one or more local processors, and the memory are enclosed in a same housing, and wherein no sensors or markers are attached to the person;

a tripod support having the digital camera articulatably attached thereto, wherein the tripod support includes a pair of short legs each having a first permanent length, and a long leg having a second permanent length at least fifty percent longer than the first permanent length;

a remote notification device in communication with a user located out of sight of the person; and

a set of instructions stored in the memory of the digital camera and executable locally by the one or more local processors to;

receive a plurality of time-sequenced images of the person from the digital image sensor;

determine a position of a torso region of the person, using an artificial intelligence module trained to determine a presence and a position of the person in at least one image of the time-sequenced plurality of images;

identify, using the position of the torso region determined by the artificial intelligence module to perform a biometric analysis on an uncompressed version of the plurality of time-sequenced images, an activity state of the person; and

communicate information corresponding to the activity state of the person to the user via the remote notification device.

10. The system of claim 9, wherein the biometric analysis includes determining a movement state of the person by performing an actigraphic analysis of the plurality of uncompressed, time-sequenced images.

11. The system of claim 9, wherein the activity state of the person includes a sleep state.

12. The system of claim 9, wherein the set of instructions are further executable locally by the one or more local processors to determine a respiration pattern of the person.

13. A method for remotely monitoring a sleep state of a human subject, the method comprising:

capturing a plurality of time-sequenced images of a human subject using a digital camera having a digital image sensor, one or more local processors in communication with the digital image sensor, and a memory, wherein the digital image sensor, the one or more local processors, and the memory are enclosed in a same housing, wherein no sensors or markers are attached to the human subject, and wherein the digital camera is articulably attached to a tripod support including a pair of short legs each having a first permanent length and a long leg having a second permanent length at least fifty percent longer than the first permanent length;

determining, using the one or more local processors to execute a machine intelligence module in the memory of the digital camera, a position of a torso region of the human subject, wherein the machine intelligence module is trained to determine a presence and a position of the human subject in at least one image of the time-sequenced plurality of images;

identifying, using the one or more local processors to execute a biometric analysis on an area associated with the torso region in an uncompressed version of the time-sequenced plurality of images, an activity state of the human subject; and

causing a change in a local environment of the human subject based at least in part on the activity state of the human subject.

14. The method of claim 13, wherein the activity state of the human subject includes a sleep state of the human subject.

15. The method of claim 13, wherein executing the biometric analysis includes determining whether the human subject is moving by executing an actigraphic analysis of the time-sequenced plurality of images.

16. The method of claim 13, further including determining a respiration pattern of the human subject.

17. The method of claim 16, the respiration pattern comprising a respective duration of each inhalation and each exhalation.

18. The method of claim 16, further comprising:

displaying a changing light pattern on a remote notification device, the changing light pattern corresponding to the respiration pattern of the human subject.

19. The method of claim 13, wherein determining the position of the torso region of the human subject in the at least one image of the time-sequenced plurality of images includes determining coordinates of the torso region and providing a bounding box generally identifying where the human subject is located.

20. The method of claim 19, wherein identifying the activity state of the human subject includes using the one or more local processors to execute the biometric analysis on a respective area of each of the plurality of uncompressed, time-sequenced images, wherein the respective area corresponds to the bounding box.

21. The method of claim 13, wherein determining the position of the torso region of the human subject in the at least one image of the time-sequenced plurality of images includes determining a confidence level with respect to the presence of the human subject in the at least one image.