Real-time video based automated mobile sleep monitoring using state inference

• US 20070156060A1
• Filed: 12/29/2005
• Published: 07/05/2007
• Est. Priority Date: 12/29/2005
• Status: Active Grant

• Current Assignee: Miguel Angel Cervantes
• Original Assignee: Miguel Angel Cervantes
• Inventors: Cervantes, Miguel
• Granted Patent: US 8,532,737 B2
• Time in Patent Office: Days
• US Class Current: 600/534
• CPC Class Codes: A61B 2503/04 Babies, e.g. for SIDS detection
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/4809 Sleep detection, i.e. determining whether a subject is asleep or not
• CPC Class Codes: A61B 5/4815 Sleep quality
• CPC Class Codes: A61B 5/4818 Sleep apnoea
• CPC Class Codes: A61B 5/742 using visual displays A61B5/339 takes precedence
• CPC Class Codes: A61B 5/7435 Displaying user selection data, e.g. icons in a graphical user interface
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 40/63 for local operation

Abstract

Apparatus for automatically monitoring sleep, including a video recorder for recording live images of a subject sleeping, including a transmitter for transmitting the recorded images in real-time to a mobile device, and a computing device communicating with said transmitter, including a receiver for receiving the transmitted images in real-time, a processor for analyzing in real-time the received images and for automatically inferring in real-time information about the state of the subject, and a monitor for displaying in real-time the information inferred by said processor about the state of the subject. A method and a computer-readable storage medium are also described and claimed.

Claims

1. Apparatus for automatically monitoring sleep, comprising:

a video recorder for recording live images of a subject sleeping, comprising a transmitter for transmitting the recorded images in real-time to a mobile device; and

a computing device communicating with said transmitter, comprising;

a receiver for receiving the transmitted images in real-time;

a processor for analyzing in real-time the received images and for automatically inferring in real-time information about the state of the subject; and

a monitor for displaying in real-time the information inferred by said processor about the state of the subject.

2. The apparatus of claim 1 wherein said monitor also displays the received images.

3. The apparatus of claim 1 wherein said video recorder also records live sound of the subject sleeping, wherein said transmitter also transmits the sound to said computing device receiver, and wherein said computing device further comprises a speaker for playing the received sound.

4. The apparatus of claim 1 wherein said processor comprises:

a motion analyzer for performing motion analysis of the subject based on the received images; and

a state detector for inferring the information about the state of the subject based on results generated by said motion analyzer.

5. The apparatus of claim 1 wherein said computing device further comprises a display control for activating said monitor whenever the information inferred about the state of the subject is deemed significant.

6. The apparatus of claim 1, wherein said computing device further comprises an alert control for sounding an alarm whenever the information inferred about the state of the subject is deemed significant.

7. The apparatus of claim 1 wherein states of the subject include any of “

sleeping”

, “

awake”

, “

no motion”

, “

moving”

, “

kneeling”

, “

standing”

, “

lying on back”

, “

lying on stomach”

, “

turning over occasionally”

, “

crying”

, “

thrashing” and

“

vomiting”

.

8. The apparatus of claim 1 wherein states of the subject include any of “

10 second apnea”

, “

low rhythm of breathing”

, “

moving in sleep”

, “

deep refresh sleep”

, “

obstructive apnea of duration between 10 sec. and 20 sec.”

, “

obstructive apnea of duration more than 20 sec.”

, “

central apnea with duration between 10 sec. and 20 sec.”

, “

central apnea with duration longer than 20 sec.”

, and “

abnormal movement during sleep”

.

9. The apparatus of claim 1 wherein said computing device further comprises a log manager for maintaining a log of sleep-related information.

10. The apparatus of claim 9 wherein the log of sleep-related information includes a summary of a night'"'"'s sleep characteristics.

11. The apparatus of claim 9 wherein the log of sleep-related information includes a comparison of a night'"'"'s sleep characteristics to a previous history.

12. The apparatus of claim 1 wherein said computing device further comprises:

a non-volatile memory; and

a storage manager for selectively storing the received images information inferred about the state of the subject in said non-volatile memory, for subsequent post-analysis.

13. The apparatus of claim 1 wherein said video recorder further comprises an infrared detector for enhancing images of objects within a dark surrounding.

14. A method for automated sleep monitoring, comprising:

recording live images of a subject sleeping;

transmitting the recorded images in real-time to a computing device;

receiving the transmitted images in real-time at the computing device;

analyzing the received images at the computing device in real-time;

automatically inferring information about the state of the subject at the computing device in real-time; and

displaying the information inferred about the state of the subject in real-time on a monitor coupled with the computing device.

15. The method of claim 14 wherein said displaying also displays the received images.

16. The method of claim 14 wherein said recording also records live sound of the subject sleeping, wherein said transmitting also transmits in real-time the recorded sound to the computing device, and further comprising playing the received sound in real-time on speakers coupled with the computing device.

17. The method of claim 14 wherein said analyzing comprises performing motion analysis of the subject based on the received images, and wherein said automatically inferring infers the information about the state of the subject based on results of said performing motion analysis.

18. The method of claim 14 further comprising activating said displaying whenever the information inferred about the state of the subject is deemed significant.

19. The method of claim 14 further comprising sounding an alarm whenever the information inferred about the state of the subject is deemed significant.

20. The method of claim 14 wherein states of the subject include any of “

sleeping”

, “

awake”

, “

no motion”

, “

moving”

, “

kneeling”

, “

standing”

, “

lying on back”

, “

lying on stomach”

, “

turning over occasionally”

, “

crying”

, “

thrashing” and

“

vomiting”

.

21. The method of claim 14 wherein states of the subject include any of “

10 second apnea”

, “

low rhythm of breathing”

, “

moving in sleep”

, “

deep refresh sleep”

, “

obstructive apnea of duration between 10 sec. and 20 sec.”

, “

obstructive apnea of duration more than 20 sec.”

, “

central apnea with duration between 10 sec. and 20 sec.”

, “

central apnea with duration longer than 20 sec.”

, and “

abnormal movement during sleep”

.

22. The method of claim 14 further comprising maintaining a log of sleep-related information.

23. The method of claim 22 wherein the log of sleep-related information includes a summary of a night'"'"'s sleep characteristics.

24. The method of claim 22 wherein the log of sleep-related information includes a comparison of a night'"'"'s sleep characteristics to a previous history.

25. The method of claim 14 further comprising selectively storing the recorded images and the information inferred about the state of the subject in a memory, for subsequent post-analysis.

26. Apparatus for automatically monitoring sleep, comprising:

a video recorder for capturing live images of a subject sleeping, comprising;

a processor for analyzing in real-time the captured images and for automatically inferring in real-time information about the state of the subject; and

a transmitter for transmitting the information inferred by said processor about the state of the subject in real-time to a mobile device; and

a display device communicating with said transmitter, comprising;

a receiver for receiving the transmitted information inferred by said processor about the state of the subject; and

a monitor for displaying in real-time the received information inferred by said processor about the state of the subject.

27. The apparatus of claim 26 wherein said transmitter also transmits the captured images to said display device receiver, and wherein said monitor also displays the received images.

28. The apparatus of claim 26 wherein said video recorder also captures live sound of the subject sleeping, wherein said transmitter also transmits the sound to said display device receiver, and wherein said display device further comprises a speaker for playing the received sound.

29. The apparatus of claim 26 wherein said processor comprises:

a motion analyzer for performing motion analysis of the subject based on the captured images; and

a state detector for inferring the information about the state of the subject based on results generated by said motion analyzer.

30. The apparatus of claim 26 wherein said display device further comprises a display control for activating said monitor whenever the information inferred about the state of the subject is deemed significant.

31. The apparatus of claim 26, wherein said display device further comprises an alert control for sounding an alarm whenever the information inferred about the state of the subject is deemed significant.

32. The apparatus of claim 26 wherein states of the subject include any of “

sleeping”

, “

awake”

, “

no motion”

, “

moving”

, “

kneeling”

, “

standing”

, “

lying on back”

, “

lying on stomach”

, “

turning over occasionally”

, “

crying”

, “

thrashing” and

“

vomiting”

.

33. The apparatus of claim 26 wherein states of the subject include any of “

10 second apnea”

, “

low rhythm of breathing”

, “

moving in sleep”

, “

deep refresh sleep”

, “

obstructive apnea of duration between 10 sec. and 20 sec.”

, “

obstructive apnea of duration more than 20 sec.”

, “

central apnea with duration between 10 sec. and 20 sec.”

, “

central apnea with duration longer than 20 sec.”

, and “

abnormal movement during sleep”

.

34. The apparatus of claim 26 wherein said video recorder further comprises a log manager for maintaining a log of sleep-related information.

35. The apparatus of claim 34 wherein the log of sleep-related information includes a summary of a night'"'"'s sleep characteristics.

36. The apparatus of claim 34 wherein the log of sleep-related information includes a comparison of a night'"'"'s sleep characteristics to a previous history.

37. The apparatus of claim 26 wherein said video recorder further comprises:

a non-volatile memory; and

a storage manager for selectively storing the recorded images and the information inferred about the state of the subject in said non-volatile memory, for subsequent post-analysis.

38. The apparatus of claim 26 wherein said video recorder further comprises an infrared detector for enhancing images of objects within a dark surrounding.

39. A method for automated sleep monitoring, comprising:

capturing live images of a subject sleeping;

analyzing the captured images in real-time;

automatically inferring information about the state of the subject in real-time;

transmitting the information inferred about the state of the subject to a display device;

receiving the information inferred about the state of the subject in real-time at the display device; and

displaying the information inferred about the state of the subject in real-time on the display device.

40. The method of claim 39 wherein said transmitting also transmits in real-time the captured images to the display device, wherein said receiving also receives in real-time the transmitted images, and wherein said displaying also displays the received images in real-time.

41. The method of claim 39 wherein said capturing also captures live sound of the subject sleeping, wherein said transmitting also transmits in real-time the captured sound to the display device, wherein said receiving also receives in real-time the transmitted sound, and further comprising playing the received sound in real-time on speakers coupled with the display device.

42. The method of claim 39 wherein said analyzing comprises performing motion analysis of the subject based on the captured images, and wherein said automatically inferring infers the information about the state of the subject based on results of said performing motion analysis.

43. The method of claim 39 further comprising activating said displaying whenever the information inferred about the state of the subject is deemed significant.

44. The method of claim 39, further comprising sounding an alarm whenever the information inferred about the state of the subject is deemed significant.

45. The method of claim 39 wherein states of the subject include any of “

sleeping”

, “

awake”

, “

no motion”

, “

moving”

, “

kneeling”

, “

standing”

, “

lying on back”

, “

lying on stomach”

, “

turning over occasionally”

, “

crying”

, “

thrashing” and

“

vomiting”

.

46. The method of claim 39 wherein states of the subject include any of “

10 second apnea”

, “

low rhythm of breathing”

, “

moving in sleep”

, “

deep refresh sleep”

, “

obstructive apnea of duration between 10 sec. and 20 sec.”

, “

obstructive apnea of duration more than 20 sec.”

, “

central apnea with duration between 10 sec. and 20 sec.”

, “

central apnea with duration longer than 20 sec.”

, and “

abnormal movement during sleep”

.

47. The method of claim 39 further comprising maintaining a log of sleep-related information.

48. The method of claim 47 wherein the log of sleep-related information includes a summary of a night'"'"'s sleep characteristics.

49. The method of claim 47 wherein the log of sleep-related information includes a comparison of a night'"'"'s sleep characteristics to a previous history.

50. The method of claim 39 further comprising selectively storing the recorded images and the information inferred about the state of the subject in a memory, for subsequent post-analysis.

51. A computing device for automatically monitoring sleep, comprising:

a receiver for receiving images of a subject sleeping from a live video recorder;

a processor for analyzing in real-time the received images and for automatically inferring in real-time information about the state of the subject; and

a monitor for displaying in real-time the information inferred by said processor about the state of the subject.

52. The computing device of claim 51 wherein said monitor also displays the received images.

53. The computing device of claim 51 wherein said receiver also receives live sound of the subject sleeping from the video recorder, and wherein said computing device further comprises a speaker for playing in real-time the received sound.

54. The computing device of claim 51 wherein said processor comprises:

a motion analyzer for performing motion analysis of the subject based on the received images; and

a state detector for inferring the information about the state of the subject based on results generated by said motion analyzer.

55. The computing device of claim 51 further comprising a display control for activating said monitor whenever the information inferred about the state of the subject is deemed significant.

56. The computing device of claim 51 further comprising an alert control for sounding an alarm whenever the information inferred about the state of the subject is deemed significant.

57. The computing device of claim 51 wherein states of the subject include any of “

sleeping”

, “

awake”

, “

no motion”

, “

moving”

, “

kneeling”

, “

standing”

, “

lying on back”

, “

lying on stomach”

, “

turning over occasionally”

, “

crying”

, “

thrashing” and

“

vomiting”

.

58. The computing device of claim 51 wherein states of the subject include any of “

10 second apnea”

, “

low rhythm of breathing”

, “

moving in sleep”

, “

deep refresh sleep”

, “

obstructive apnea of duration between 10 sec. and 20 sec.”

, “

obstructive apnea of duration more than 20 sec.”

, “

central apnea with duration between 10 sec. and 20 sec.”

, “

central apnea with duration longer than 20 sec.”

, and “

abnormal movement during sleep”

.

59. The computing device of claim 51 further comprising a log manager for maintaining a log of sleep-related information.

60. The computing device of claim 59 wherein the log of sleep-related information includes a summary of a night'"'"'s sleep characteristics.

61. The computing device of claim 59 wherein the log of sleep-related information includes a comparison of a night'"'"'s sleep characteristics to a previous history.

62. The computing device of claim 51 further comprising:

a non-volatile memory; and

a storage manager for selectively storing the received images and the information inferred about the state of the subject in said non-volatile memory, for subsequent post-analysis.

63. A method for automated sleep monitoring, comprising:

receiving live images of a subject sleeping from a video recorder;

analyzing the received images in real-time;

automatically inferring information about the state of the subject in real-time; and

displaying the information inferred about the state of the subject in real-time on a monitor.

64. The method of claim 63 wherein said displaying also displays the received images.

65. The method of claim 63 wherein said receiving also receives live sound of the subject sleeping from the video recorder, and further comprising playing the received sound in real-time on speakers.

66. The method of claim 63 wherein said analyzing comprises performing motion analysis of the subject based on the received images, and wherein said automatically inferring infers the information about the state of the subject based on results of said performing motion analysis.

67. The method of claim 63 further comprising activating said displaying whenever the information inferred about the state of the subject is deemed significant.

68. The method of claim 63 further comprising sounding an alarm whenever the information inferred about the state of the subject is deemed significant.

69. The method of claim 63 wherein states of the subject include any of “

sleeping”

, “

awake”

, “

no motion”

, “

moving”

, “

kneeling”

, “

standing”

, “

lying on back”

, “

Lying on stomach”

, “

turning over occasionally”

, “

crying”

, “

thrashing” and

“

vomiting”

.

70. The method of claim 63 wherein states of the subject include any of “

10 second apnea”

, “

low rhythm of breathing”

, “

moving in sleep”

, “

deep refresh sleep”

, “

obstructive apnea of duration between 10 sec. and 20 sec.”

, “

obstructive apnea of duration more than 20 sec.”

, “

central apnea with duration between 10 sec. and 20 sec.”

, “

central apnea with duration longer than 20 sec.”

, and “

abnormal movement during sleep”

.

71. The method of claim 63 further comprising maintaining a log of sleep-related information.

72. The method of claim 71 wherein the log of sleep-related information includes a summary of a night'"'"'s sleep characteristics.

73. The method of claim 71 wherein the log of sleep-related information includes a comparison of a night'"'"'s sleep characteristics to a previous history.

74. The method of claim 63 further comprising selectively storing the recorded images and the information inferred about the state of the subject in a memory, for subsequent post-analysis.

75. An enhanced video recorder for automatically monitoring sleep, comprising:

a memory for storing captured digital images of a subject sleeping;

an infrared detector for enhancing images of objects within a dark surrounding;

a processor for analyzing in real-time the captured images and for automatically inferring in real-time information about the state of the subject; and

a transmitter for transmitting the information inferred by said processor about the state of the subject in real-time to a display device.

76. The enhanced video recorder of claim 75 wherein said transmitter also transmits the captured images to the display device.

77. The enhanced video recorder of claim 75 wherein said memory also stores captured sound of the subject sleeping, and wherein said transmitter also transmits the sound to the display device.

78. The enhanced video recorder of claim 75 wherein said processor comprises:

a motion analyzer for performing motion analysis of the subject based on the captured images; and

a state detector for inferring the information about the state of the subject based on results generated by said motion analyzer.

79. The enhanced video recorder of claim 75 wherein states of the subject include any of “

sleeping”

, “

awake”

, “

no motion”

, “

moving”

, “

kneeling”

, “

standing”

, “

lying on back”

, “

lying on stomach”

, “

turning over occasionally”

, “

crying”

, “

thrashing” and

“

vomiting”

.

80. The enhanced video recorder of claim 75 wherein states of the subject include any of “

10 second apnea”

, “

low rhythm of breathing”

, “

moving in sleep”

, “

deep refresh sleep”

, “

obstructive apnea of duration between 10 sec. and 20 sec.”

, “

obstructive apnea of duration more than 20 sec.”

, “

central apnea with duration between 10 sec. and 20 sec.”

, “

central apnea with duration longer than 20 sec.”

, and “

abnormal movement during sleep”

.

81. The enhanced video recorder of claim 75 further comprising a log manager for maintaining a log of sleep-related information.

82. The enhanced video recorder of claim 81 wherein the log of sleep-related information includes a summary of a night'"'"'s sleep characteristics.

83. The enhanced video recorder of claim 81 wherein the log of sleep-related information includes a comparison of a night'"'"'s sleep characteristics to a previous history.

84. The enhanced video recorder of claim 75 further comprising:

a non-volatile memory; and

a storage manager for selectively storing the recorded images and the information inferred about the state of the subject in said non-volatile memory, for subsequent post-analysis.

85. The enhanced video recorder of claim 75 further comprising an infrared detector for enhancing images of objects within a dark surrounding.

86. A method for automated sleep monitoring, comprising:

capturing images of a subject sleeping;

analyzing the capturing images in real-time;

automatically inferring information about the state of the subject in real-time; and

transmitting the information inferred about the state of the subject to a display device in real-time.

87. The method of claim 86 wherein said transmitting also transmits in real-time the captured images to the display device.

88. The method of claim 86 wherein said capturing also captures sound of the subject sleeping, and wherein said transmitting also transmits in real-time the captured sound to the display device.

89. The method of claim 86 wherein said analyzing comprises performing motion analysis of the subject based on the captured images, and wherein said automatically inferring infers the information about the state of the subject based on results of said performing motion analysis.

90. The method of claim 86 wherein states of the subject include any of “

sleeping”

, “

awake”

, “

no motion”

, “

moving”

, “

kneeling”

, “

standing”

, “

lying on back”

, “

lying on stomach”

, “

turning over occasionally”

, “

crying”

, “

thrashing” and

“

vomiting”

.

91. The method of claim 86 wherein states of the subject include any of “

10 second apnea”

, “

low rhythm of breathing”

, “

moving in sleep”

, “

deep refresh sleep”

, “

obstructive apnea of duration between 10 sec. and 20 sec.”

, “

obstructive apnea of duration more than 20 sec.”

, “

central apnea with duration between 10 sec. and 20 sec.”

, “

central apnea with duration longer than 20 sec.”

, and “

abnormal movement during sleep”

.

92. The method of claim 86 further comprising maintaining a log of sleep-related information.

93. The method of claim 92 wherein the log of sleep-related information includes a summary of a night'"'"'s sleep characteristics.

94. The method of claim 92 wherein the log of sleep-related information includes a comparison of a night'"'"'s sleep characteristics to a previous history.

95. The method of claim 86 further comprising selectively storing the recorded images and the information inferred about the state of the subject in a memory, for subsequent post-analysis.

96. A computer-readable storage medium storing program code for causing a computing device to:

receive transmitted live images of a subject sleeping in real-time;

analyze the received images in real-time;

automatically infer information about the state of the subject in real-time; and

display the information inferred about the state of the subject in real-time on a monitor coupled with the computing device.

97. A computer-readable storage medium storing program code for causing a video recorder to:

capture images of a subject sleeping;

analyze the capturing images in real-time;

automatically infer information about the state of the subject in real-time; and

transmit the information inferred about the state of the subject to a display device.