Enhanced video interaction for a user interface of a telepresence network

• US 9,571,789 B2
• Filed: 06/23/2015
• Issued: 02/14/2017
• Est. Priority Date: 11/26/2012
• Status: Active Grant

• Current Assignee: Teladoc Health, Inc.
• Original Assignee: Touch Technologies Incorporated
• Inventors: Pinter, Marco, Jordan, Charles S., Sanchez, Daniel, Hanrahan, Kevin, Temby, Kelton, Lambrecht, Christopher
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US14/747,839
• Publication Number:
  US 20150296177A1
• Time in Patent Office: 602 Days
• Field of Search: 345/671, 348/14.03, 348/14.05, 348/61, 348/150, 348/218.1, 348/14.01, 348/14.02, 376/249, 382/103, 382/173, 382/195, 700/56, 700/259, 709/227, 715/855, 706/11
• US Class Current: 1/1
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/102 Programmed access in sequence to addressed parts of tracks of operating record carriers access by moving the head G11B3/08, G11B5/54, G11B7/085, G11B21/022; by moving the record carrier G11B15/005, G11B17/005, by driving of both record carrier and head G11B15/1816
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A telepresence device may relay video, audio, and/or measurement data to a user operating a control device. A user interface may permit the user to quickly view and/or understand temporally and/or spatially disparate information. The telepresence device may pre-gather looped video of spatially disparate areas in an environment. A temporal control mechanism may start video playback at a desired point in a current or historical video segment. Notations may be associated with time spans in a video and recalled by capturing an image similar to a frame in the time span of the video. An area of interest may be selected and video containing the area of interest may be automatically found. Situational data may be recorded and used to recall video segments of interest. The telepresence device may synchronize video playback and movement. A series of videos may be recorded at predetermined time intervals to capture visually trending information.

Claims

1. A non-transitory computer-readable storage medium including computer-readable instruction code for performing a method of generating pre-session video loops, the method comprising:

receiving, at a telepresence device that includes a camera, a display device, a microphone, and a speaker, an indication to navigate to an indicated location;

adjusting a camera on the telepresence device to identify one or more areas of interest at the indicated location;

sequentially targeting each area of interest with the camera;

recording video of each area of interest for a predetermined time period; and

transmitting the recorded video to a remotely located control device, wherein the control device can establish a two-way audio/video communication session with the telepresence device.

2. The non-transitory computer-readable storage medium of claim 1, wherein recording video of each area of interest comprises continuously cycling through the one or more areas of interest until receiving a connection request from the remotely located control device.

3. The non-transitory computer-readable storage medium of claim 1, wherein the one or more areas of interest contain an object selected from the group consisting of a patient'"'"'s face, an EKG monitor, a chart, and a fluid bag.

4. The non-transitory computer-readable storage medium of claim 1, wherein the method further comprises recording a time code indicating when the video of the one or more areas of interest was recorded.

5. A non-transitory computer-readable storage medium including computer-readable instruction code for performing a method of providing segmented temporal video control of video received from a telepresence device that includes a camera, a display device, a microphone, and a speaker, the method comprising:

receiving, at a control device, streaming video from the telepresence device;

determining a plurality of video segments corresponding to a plurality of telepresence device activities;

displaying a first video segment associated with a first telepresence device activity to a user; and

providing a temporal control mechanism to the user permitting the user to select a first playback point within the first video segment, wherein the control device can establish a two-way audio/video communication session with the telepresence device.

6. The non-transitory computer-readable storage medium of claim 5, wherein the method further comprises receiving an indication of a desired video segment from the user.

7. The non-transitory computer-readable storage medium of claim 5, wherein the method further comprises:

receiving an indication of a desired time span from the user, wherein none of the plurality of determined video segments wholly correspond to the desired time span;

determining a desired video segment corresponding to the desired time span; and

displaying the desired video segment to the user.

8. The non-transitory computer-readable storage medium of claim 7, wherein the method further comprises receiving an indication from the user to play the desired video segment in a loop.

9. A non-transitory computer-readable storage medium including computer-readable instruction code for performing a method of annotating video segments received from a telepresence device that includes a camera, a display device, a microphone, and a speaker, the method comprising:

receiving, at a control device, streaming video from the telepresence device;

displaying a video segment from the streaming video to a user;

receiving an indication of a desired time span from the user;

receiving an annotation corresponding to the desired time span from the user;

storing the received annotation in association with the desired time span; and

displaying an interactive indication in a location corresponding to the desired time span, wherein the user may access the received annotation using the interactive indication, wherein the control device can establish a two-way audio/video communication session with the telepresence device.

10. The non-transitory computer-readable storage medium of claim 9, wherein receiving an indication of a desired time span comprises:

receiving a user indicated start time; and

automatically selecting an end time based on a default time span length.

11. The non-transitory computer-readable storage medium of claim 9, wherein receiving an annotation comprises:

receiving a user indicated title and a user indicated note; and

automatically inserting an author name into the annotation, wherein the author name corresponds to the user.

12. The non-transitory computer-readable storage medium of claim 9, wherein the method further comprises:

receiving a drawn object to be overlaid on the video segment during the desired time span; and

displaying the drawn object overlaid on the video segment when the desired time span is displayed.

13. A non-transitory computer-readable storage medium including computer-readable instruction code for performing a method of recalling annotations based on a captured image, the method comprising:

receiving, at a control device, video from a telepresence device that includes a camera, display device, a microphone, and a speaker;

receiving an annotation from a first user corresponding to a frame in the video;

storing first location data corresponding to the frame;

storing the annotation;

receiving an image from a second device at a later time;

identifying second location data corresponding to the image;

retrieving the first location data and the annotation; and

displaying at least a portion of the annotation if the second location data is within a predetermined threshold of the first location data, wherein the control device can establish a two-way audio/video communication session with the telepresence device.

14. The non-transitory computer-readable storage medium of claim 13, wherein the first location data comprises at least one data element selected from the group consisting of a room number, a position of the telepresence device, an orientation of the telepresence device, a camera tilt, a camera pan, a camera zoom, and a set of image descriptors, and wherein the first location data corresponds to the frame in the video.

15. The non-transitory computer-readable storage medium of claim 13, wherein the first location data comprises general position information and a set of image descriptors.

16. The non-transitory computer-readable storage medium of claim 13, wherein receiving an annotation comprises receiving an annotation corresponding to an object in the frame, and wherein displaying comprises displaying at least the portion of the received annotation if a second set of image descriptors are within the predetermined threshold of a first set of image descriptors corresponding to the object.

17. The non-transitory computer-readable storage medium of claim 13, wherein the telepresence device comprises a robotic platform, wherein the first location data includes position and orientation data determined by a telepresence device component, and wherein inverse kinematics is used to compare the first location data to the second location data.

18. A non-transitory computer-readable storage medium including computer-readable instruction code for performing a method of spatially searching video, the method comprising:

receiving, at a control device, the video from a telepresence device that includes a camera, a display device, a microphone, and a speaker;

storing the video;

receiving an indication of a spatial area of interest in a frame of the video from a user;

identifying one or more video segments from the stored video containing the spatial area of interest; and

providing indications of the one or more video segments to the user, wherein the control device can establish a two-way audio/video communication session with the telepresence device.

19. The non-transitory computer-readable storage medium of claim 18, wherein identifying one or more video segments comprises comparing the spatial area of interest to the stored video using an algorithm selected from the group consisting of a pattern-matching algorithm and an inverse kinematics algorithm.

20. The non-transitory computer-readable storage medium of claim 18, wherein identifying one or more video segments comprises determining whether a zoom level exceeds a predetermined threshold.

21. The non-transitory computer-readable storage medium of claim 18, wherein the method further comprises providing a rating for each of the one or more video segments to the user, wherein the rating is determined based on a size of the spatial area of interest in each of the one or more video segments.

22. A non-transitory computer-readable storage medium including computer-readable instruction code for performing a method of contextually searching video from a telepresence device that includes a camera, a display device, a microphone, and a speaker, the method comprising:

receiving, at a control device, the video from the telepresence device;

storing the video;

storing situational data for each of a plurality of video segments;

receiving an indication of a situation of interest from a user;

identifying one or more matching video segments corresponding to the situation of interest; and

providing indications of the one or more matching video segments to the user, wherein the control device can establish a two-way audio/video communication session with the telepresence device.

23. The non-transitory computer-readable storage medium of claim 22, wherein the situational data comprises at least one data element selected from the group consisting of a patient name, a procedure being performed, and a remote practitioner operating the telepresence device.

24. The non-transitory computer-readable storage medium of claim 22, wherein the method further comprises:

receiving a location-situation mapping from a local practitioner; and

determining the situational data from location data.

25. The non-transitory computer-readable storage medium of claim 22, wherein receiving an indication of a situation of interest comprises receiving a filter criterion, and wherein providing indications comprises providing indications identifying each video segment according to a situational data element corresponding to the filter criterion.

26. A non-transitory computer-readable storage medium including computer-readable instruction code for performing a method of synchronized location and video replay, the method comprising:

storing a plurality of video segments recorded by a telepresence device that includes a camera, a display device, a microphone, and a speaker;

storing corresponding location data for each of the plurality of video segments;

navigating the telepresence device to a first location; and

playing, at a control device, a first video segment corresponding to the first location while the telepresence device is at the first location, wherein the control device can establish a two-way audio/video communication session with the telepresence device.

27. The non-transitory computer-readable storage medium of claim 26, wherein the method further comprises receiving a pause command from a user;

pausing navigation and playback in response to the pause command; and

providing control of the telepresence device to the user.

28. The non-transitory computer-readable storage medium of claim 26, wherein the method further comprises pausing playback of the first video segment if the telepresence device is more than a predetermined distance from the first location.

29. The non-transitory computer-readable storage medium of claim 26, wherein the method further comprises:

storing text received from a user while at the first location; and

associating the text in memory with the first video segment.

30. A non-transitory computer-readable storage medium including computer-readable instruction code for performing a method of monitoring visible trends, the method comprising:

visiting a predetermined location with a telepresence device at predetermined time intervals, the telepresence device including a camera, a display device, a microphone, and a speaker;

scanning the predetermined location for one or more predetermined areas of interest;

recording video of each area of interest during each visit for a predetermined time period;

transmitting the recorded video to a storage device; and

playing, at a control device, the recorded video, wherein the control device can establish a two-way audio/video communication session with the telepresence device.

31. The non-transitory computer-readable storage medium of claim 30, wherein scanning the predetermined location comprises identifying a face using a Haar-like feature-matching algorithm.

32. The non-transitory computer-readable storage medium of claim 30, wherein scanning the predetermined location comprises identifying an object of interest using at least one algorithm selected from the group consisting of scale-invariant feature transform (SIFT), speeded up robust features (SURF), and oriented features from accelerated segment test and rotated binary robust independent elementary features (ORB).

33. The non-transitory computer-readable storage medium of claim 30, wherein the method further comprises transmitting a time stamp corresponding to the recorded video to the storage device.

34. A non-transitory computer-readable storage medium including computer-readable instruction code for performing a method of reorienting an object in a video, the method comprising:

establishing a two-way audio/video communication session between a control device and a telepresence device, the telepresence device including a camera, a display device, a microphone, and a speaker;

selectively displaying a video feed from the telepresence device in a video panel on an electronic display of the control device;

receiving a zoom request associated with a selected portion of the video feed;

magnifying the selected portion of the video feed;

identifying informational content within the magnified selected portion of the video feed that is misaligned with respect to the orientation of the video feed; and

digitally aligning the informational content with respect to the orientation of the video feed.

35. The non-transitory computer-readable storage medium of claim 34, wherein magnifying the selected portion of the video feed comprises at least one of:

transmitting instructions to adjust an optical zoom of a camera;

digitally zooming the selected portion of the video feed; and

transmitting navigation instructions to cause the telepresence device to navigate in the direction of the selected portion of the video feed.

36. The non-transitory computer-readable storage medium of claim 34, wherein identifying informational content comprises identifying informational content selected from the group consisting of written or printed information on a document, information displayed on an electronic display, text, a chart, and an image.

37. The non-transitory computer-readable storage medium of claim 34, wherein digitally aligning the informational content with respect to the video feed comprises deskewing the informational content.

38. The non-transitory computer-readable storage medium of claim 34, wherein digitally aligning the informational content with respect to the video feed comprises rotating the informational content.

39. The non-transitory computer-readable storage medium of claim 34, wherein the operations further comprise:

enhancing the informational content.

40. The non-transitory computer-readable storage medium of claim 39, wherein enhancing the informational content comprises adjusting a contrast of the informational content.

41. The non-transitory computer-readable storage medium of claim 39, wherein enhancing the informational content comprises synchronizing a refresh rate of the video feed based on a refresh rate of the informational content.