Methods and systems for providing interface components for respiratory therapy

• US 10,220,172 B2
• Filed: 11/07/2016
• Issued: 03/05/2019
• Est. Priority Date: 11/25/2015
• Status: Active Grant

• Current Assignee: Resmed Limited (ResMed Incorporated)
• Original Assignee: Resmed Limited (ResMed Incorporated)
• Inventors: Lucey, Simon Michael, Gupta, Priyanshu, Johnston, Benjamin Peter, Yu, Tzu-Chin
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US15/344,761
• Publication Number:
  US 20170173289A1
• Time in Patent Office: 848 Days
• Field of Search: 345419
• CPC Class Codes: A61B 5/0064 Body surface scanning
• CPC Class Codes: A61B 5/08 Detecting, measuring or recording devices for evaluating the respiratory organs A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/1072 measuring distances on the body, e.g. measuring length, height or thickness A61B5/1076 takes precedence
• CPC Class Codes: A61B 5/1079 using optical or photographic means
• CPC Class Codes: A61M 16/021 operated by electrical means A61M16/202 – A61M16/205 take precedence
• CPC Class Codes: A61M 16/06 Respiratory or anaesthetic masks
• CPC Class Codes: A61M 16/0605 Means for improving the adaptation of the mask to the patient
• CPC Class Codes: A61M 16/0633 with forehead support
• CPC Class Codes: A61M 16/0683 Holding devices therefor
• CPC Class Codes: A61M 16/0816 Joints or connectors
• CPC Class Codes: A61M 2016/0661 with customised shape
• CPC Class Codes: A61M 2205/3561 local, e.g. within room or hospital
• CPC Class Codes: A61M 2205/3592 using telemetric means, e.g. radio or optical transmission
• CPC Class Codes: A61M 2205/505 Touch-screens; Virtual keyboard or keypads; Virtual buttons; Soft keys; Mouse touches
• CPC Class Codes: A61M 2207/00 Methods of manufacture, assembly or production
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/00 Computing arrangements based on biological models
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06V 10/32 : Normalisation of the pattern dimensions
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/165 : using facial parts and geometric relationships
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/169 : Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G16H 10/60 : for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/60 : for the operation of medical equipment or devices
• CPC Class Codes: G16H 50/50 : for simulation or modelling of medical disorders

Abstract

Systems and methods permit generation of a digital scan of a user'"'"'s face such as for obtaining of a patient respiratory mask, or component(s) thereof, based on the digital scan. The method may include: receiving video data comprising a plurality of video frames of the user'"'"'s face taken from a plurality of angles relative to the user'"'"'s face, generating a three-dimensional representation of a surface of the user'"'"'s face based on the plurality of video frames, receiving scale estimation data associated with the received video data, the scale estimation data indicative of a relative size of the user'"'"'s face, and scaling the digital three-dimensional representation of the user'"'"'s face based on the scale estimation data. In some aspects, the scale estimation data may be derived from motion information collected by the same device that collects the scan of the user'"'"'s face.

Claims

1. An apparatus for acquiring data for generation of a three-dimensional facial scan of a user for obtaining a patient respiratory interface component, the apparatus comprising:

an image sensor and lens for capturing two dimensional image data of the user'"'"'s face;

a motion sensor configured to sense movement data of at least one of a movement of the apparatus and a change in orientation of the apparatus;

a processor configured to receive image data from the image sensor and to receive movement data from the motion sensor, to generate a video file based on the image data received from the image sensor, and to generate a motion data file based on the movement data received from the motion sensor, and to associate each of the video file and the motion data file with one another, wherein the video file and the motion data file are generated to include a temporal map that correlates the motion data file with the video file; and

a transmitter, coupled with the processor, to transmit the associated video and motion data files to a system comprising a surface engine at a remote destination for generation of a scaled three-dimensional representation of the user'"'"'s face based on the received image data and the received movement data.

2. A method for acquiring data with a mobile computing device for generation of a three-dimensional facial scan of a user for obtaining a patient respiratory interface component, the method comprising:

controlling an image sensor having a lens to capture two dimensional image data of the user'"'"'s face;

sensing with a motion sensor configured to sense movement data of at least one of a movement of the image sensor and a change in orientation of the image sensor;

controlling, in a processor configured to receive image data from the image sensor and to receive movement data from the motion sensor, generation of a video file based on the image data received from the image sensor, and generation of a motion data file based on the movement data received from the motion sensor, and associating each of the video file and data file with one another wherein the video file and the motion data file are generated with a temporal map that correlates the motion data file with the video file; and

controlling, via a transmitter coupled with the processor, transmission of the associated video and data files to a system comprising a surface engine at a remote destination for generation of a scaled three-dimensional representation of the user'"'"'s face based on the received image data and the received movement data.

3. A non-transitory computer-readable data storage medium having program instructions encoded thereon configured to cause a processor to perform the method of claim 2.

4. A server having the non-transitory computer-readable data storage medium of claim 3, wherein the server is configured to receive requests for downloading the program instructions of the computer-readable data storage medium to a remote mobile computing device over a network.

5. An electronic system for enrolling an end-user for obtaining a patient respiratory interface component, the system comprising:

a customer information subsystem for receiving an enrolment request from an end-user device for obtaining a patient respiratory interface component, the enrolment request comprising a unique identifier of the end-user and for receiving from the end-user device a video file and a motion data file that are generated with a temporal map that correlates the motion data file with the video file;

a three-dimensional surface rendering engine, including at least one processor, the three-dimensional surface rendering engine configured to render and scale a three-dimensional representation of the end-user'"'"'s face based on end-user video frame data and movement data from the video file and the motion data file and using the temporal map; and

a first database for storing said rendered and scaled three-dimensional representation, or an identification of the rendered and scaled three-dimensional representation, in association with said unique identifier.

6. The electronic system of claim 5, wherein the customer information subsystem and customer reference database operate on one or more servers in a network, and wherein the enrolment request and end-user video frame data and movement data are received over the network.

7. The electronic system of claim 5, wherein the temporal map comprises timestamp information, wherein the end-user video frame data and movement data are combined at the three-dimensional surface rendering engine based on the timestamp information.

8. The electronic system of claim 5, wherein the three dimensional surface rendering engine is configured to receive frame selection information designating a pre-selected subset of the end-user video frame data for use in rendering the three-dimensional representation of the end-user'"'"'s face.

9. The electronic system of claim 8, wherein the frame selection information further comprises a number of frames selected.

10. The electronic system of claim 5, further comprising a second database,

wherein the first database stores an identification of the rendered and scaled three-dimensional representation in association with said unique identifier, and

wherein the three-dimensional surface rendering engine generates an object file containing said rendered and scaled three-dimensional representation and stores the object file, or a reference to the object file, in the second database, and

wherein the object file, or reference thereto, is retrieved from the second database based on the identification of said rendered and scaled three-dimensional representation in association with said unique identifier.

11. The electronic system of claim 5 wherein one or more processors are configured to determine one or more dimensions from the rendered and scaled three-dimensional representation for obtaining the patient respiratory interface component.

12. The electronic system of claim 11 wherein the one or more processors are configured to compare the one or more dimensions from the rendered and scaled three-dimensional representation with one or more dimensions from one or more respiratory masks.

13. The system of claim 5 wherein the temporal map correlates inertial measurements in the motion data file over a duration of a plurality of video frames of the video file.

14. The system of claim 13 wherein the temporal map comprises timestamps in both of the video file and in the motion data file.

15. A method of generating a three-dimensional surface scan of a user'"'"'s face for obtaining a patient respiratory interface component, the method using a mobile device, the method comprising:

receiving video data in a video file, the video data comprising a plurality of video frames of the user'"'"'s face taken from a plurality of angles relative to the user'"'"'s face;

generating a digital three-dimensional representation of the user'"'"'s face based on the plurality of video frames;

receiving scale estimation data associated with the received video data, scale estimation data comprising inertial movement data in a motion data file, the inertial movement data capturing movement of the plurality of angles, wherein the video file and the motion data file are generated with a temporal map that correlates the motion data file with the video file, the scale estimation data indicative of a relative size of the user'"'"'s face; and

scaling the digital three-dimensional representation of the user'"'"'s face based on the scale estimation data.

16. The method of claim 15, wherein the plurality of video frames is a subset of the video frames included in the video data, the method further comprising selecting said plurality of video frames from the video data based on one or more predetermined criteria.

17. The method of claim 16, wherein the selected plurality of video frames comprises at least one front view of the user'"'"'s face, one left profile of the user'"'"'s face, and one right profile of the user'"'"'s face.

18. The method of claim 15, wherein the three-dimensional representation of the user'"'"'s face is generated with a Basel face model, and wherein the method further comprises adjusting a three-dimensional representation of a generic face based on data extracted from each frame of the plurality of video frames.

19. The method of claim 18, further comprising, in each frame of the plurality of video frames, identifying a plurality of points associated with landmark features of the user'"'"'s face, wherein adjustment of the three-dimensional representation of a generic face is based on the identified plurality of points.

20. The method of claim 19, wherein identifying a plurality of points associated with landmark features of the user'"'"'s face is based at least in part on a machine learning algorithm.

21. The method of claim 19, wherein identifying a plurality of points associated with landmark features of the user'"'"'s face comprises:

for each of the plurality of video frames, identifying one or more features of the user'"'"'s face;

detecting said landmark features of the user'"'"'s face based on the identified features; and

associating each of the detected landmark features with one or more of the plurality of points.

22. The method of claim 21, wherein identifying one or more features further comprises using cascade classifiers.

23. The method of claim 19, wherein the plurality of points comprises at least eighty points.

24. The method of claim 19, wherein the plurality of points comprises at least one hundred thirty two points.

25. The method of claim 19, wherein at least some of the plurality of points are representative of the user'"'"'s eyes, at least some of the plurality of points are representative of the user'"'"'s nose, and at least some of the plurality of points are representative of the user'"'"'s mouth.

26. The method of claim 19, wherein at least some of the plurality of points are representative of an edge of the user'"'"'s face, wherein the edge of the user'"'"'s face in a given video frame is represented by a boundary between the user'"'"'s face and a background of the video frame.

27. The method of claim 19, wherein the plurality of points are subdivided into discrete regions of the user'"'"'s face, at least one of the discrete regions being identified in each frame of the plurality of video frames, and wherein the three-dimensional representation of the user'"'"'s face is generated based at least in part on said discrete regions.

28. The method of claim 27, wherein each of the user'"'"'s eyes is included in a separate discrete region of the user'"'"'s face.

29. The method of claim 27, wherein user'"'"'s nose is included in a different discrete region from the user'"'"'s eyes.

30. The method of claim 15, further comprising:

in each frame of the plurality of video frames, identifying a plurality of points associated with landmark features of the user'"'"'s face; and

in each frame of the plurality of video frames, detecting at least one edge of the user'"'"'s face, wherein generating the three-dimensional representation of the user'"'"'s face is based on a combination of the plurality of points and the detected edge of the user'"'"'s face.

31. The method of claim 15, wherein the scale estimation data comprises an interpupil distance of the user'"'"'s face.

32. The method of claim 15, wherein the scale estimation data comprises a measurement of an article placed on the user'"'"'s face and having a predetermined size or shape.

33. The method of claim 15, wherein the scale estimation data comprises information from one or more signals transmitted from a transmitter positioned at the user'"'"'s face.

34. The method of claim 15, wherein the scale estimation data comprises a manually input measurement relating to a scale factor of the user'"'"'s face.

35. The method of claim 15, wherein the received video data is collected by two or more mobile devices, and wherein the scale estimation data comprises a measurement relating to a distance between at least two of the mobile devices.

36. The method of claim 15, wherein the received video data is collected by two or more mobile devices mounted to a helmet worn by the user, and wherein the scale estimation data is a predetermined value based on a fixed distance of the two or more devices from the user'"'"'s face.

37. The method of claim 15, wherein the scale estimation data comprises a measurement of a distance between the mobile device and the user'"'"'s face based on an adjustable focus property of the mobile device.

38. The method of claim 15, wherein the scale estimation data comprises a time delay measured by an ultrasonic or electromagnetic measuring device.

39. The method of claim 15, wherein the scale estimation data comprises a measurement of a structured light emitted by one or more flash bulbs of the mobile device.

40. A non-transitory computer-readable data storage medium having program instructions encoded thereon configured to cause a processor to perform a method of generating a three-dimensional surface scan of a user'"'"'s face for obtaining a patient respiratory interface component according to claim 15.

41. A non-transitory computer-readable data storage medium having a file encoded thereon, the file encoding a three-dimensional representation of a user'"'"'s face, the representation being generated using the method of claim 15.

42. A system for generating a three dimensional surface scan of a user'"'"'s face for obtaining a patient respiratory interface component, the system comprising:

one or more servers, the one or more servers including one or more processors, the one or more processors configured to perform the method of claim 15.

43. The method of claim 15 wherein the three-dimensional representation of the user'"'"'s face is generated based on identifying, in the end-user video frame data, points located around, and representative of, the end-user'"'"'s mouth that may ensure a respiratory mask seal against the end-user'"'"'s face.

44. The method of claim 30, further comprising:

receiving IMU data associated with the received video data; and

generating a three-dimensional representation of the user'"'"'s face based on a combination of the plurality of points, the detected edge of the user'"'"'s face, and the received IMU data.

45. The method of claim 44, further comprising correlating the received IMU data with respective video frames of the received video data.

46. The method of claim 44, wherein the scale estimation data comprises the received IMU data.

47. A system for obtaining a patient respiratory interface component comprising:

one or more data stores storing a training dataset of three-dimensional surfaces, each three-dimensional surface representative of a human face; and

one or more processors configured to receive a plurality of two-dimensional images of a face of a human subject, each of the images having been taken at an unspecified distance from and an unspecified angle relative to the human subject, to receive a request to generate a new three-dimensional surface from said images of the face of the human subject, and to generate the new three-dimensional surface based on a support vector machine and said images, wherein the support vector machine is derived from the training dataset of three-dimensional surfaces, and wherein the generated new three-dimensional surface is representative of the face of the human subject.

48. The system of claim 47, wherein the generated new three-dimensional surface comprises a grid superimposed over a representation of the face of the human subject.

49. A method for obtaining a patient respiratory interface component comprising:

accessing, in one or more data stores, a training dataset of three-dimensional surfaces, each three-dimensional surface representative of a human face; and

in one or more processors;

receiving a plurality of two-dimensional images of a face of a human subject, each of the images having been taken at an unspecified distance from and an unspecified angle relative to the human subject, receiving a request to generate a new three-dimensional surface from said images of the face of the human subject, and generating the new three-dimensional surface based on a support vector machine and said images, wherein the support vector machine is derived from the training dataset of three-dimensional surfaces and wherein the generated new three-dimensional surface is representative of the face of the human subject.

50. A non-transitory computer-readable data storage medium having program instructions encoded thereon configured to cause a processor to perform a method comprising the method of claim 49.

51. A method for obtaining a patient respiratory interface component for a user'"'"'s face, comprising:

receiving video data from the user, the video data including a video scan of the user'"'"'s face;

determining one or more anthropometric measurements based on information derived from the video scan;

generating a three-dimensional representation of the user'"'"'s face based on the determined one or more anthropometric measurements;

scaling the three-dimensional representation of the user'"'"'s face wherein the scaling is based on scale estimation data comprising movement data in a motion data file, the movement data capturing video device movement during the video scan, wherein the motion data file is generated with a temporal map that correlates the motion data file with a video file including the video data; and

obtaining the patient respiratory interface component based on the scaled three-dimensional representation of the user'"'"'s face, wherein the determined one or more anthropometric measurements are made using a processor and have an accuracy of at least about ±

3 mm when scaled.

52. A non-transitory computer-readable data storage medium having program instructions encoded thereon configured to cause one or more processors to perform a method comprising the method of claim 51.

53. The method of claim 51, wherein the one or more anthropometric measurements comprises at least one of:

an alar angle, a nasolabial angle and a nose width.

54. The method of claim 51, wherein the one or more anthropometric measurements comprises at least one of a mouth width and an upper lip height.

55. The method of claim 54, wherein the determined anthropometric measurements have an accuracy of at least about ±

0.5 mm when scaled.