Media presentation playback annotation

• US 10,255,929 B2
• Filed: 04/28/2016
• Issued: 04/09/2019
• Est. Priority Date: 05/18/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Matthews, III, Joseph H., Padmanabhan, Udiyan, Peters, Jannes Paul
• Primary Examiner(s): Patel, Manglesh M
• Application Number:
  US15/141,503
• Publication Number:
  US 20160247523A1
• Time in Patent Office: 1,076 Days
• Field of Search: 715233, 715202, 715200
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G10L 2025/783 based on threshold decision
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: G10L 25/57 for processing of video signals

Abstract

Annotations can be automatically added to a media presentation during playback of the presentation without a user having to manually interact with the playback device. The playback device determines whether an annotation is to be added to the media presentation based on characteristics of voice input received at the device, such as voice input signal strength or variances in the voice input signal strength. Characteristics of video input received at the device can be used to determine whether a user is speaking to the computing device as well. The device can handle a new annotation overlapping an existing annotation by either removing the existing annotation or by shifting the existing annotation until there is no more overlap. A media presentation can comprise multiple annotation tracks.

Claims

1. A method of annotating a media presentation, the method comprising:

playing at least a portion of the media presentation at a computing device comprising one or more audio input devices and one or more video input devices;

monitoring the one or more video input devices for video input received during the playing, wherein the one or more video input devices capture video input of a user during the playing of the media presentation;

upon determining, based at least in part upon the video input and without manual interaction with the computing device, that the user is speaking in a direction of the computing device;

creating an annotation by receiving, via the one or more audio input devices, voice input comprising the annotation to be added to the media presentation; and

adding the annotation to the media presentation;

wherein determining that the user is speaking in the direction of the computing device comprises;

determining a distance from the user to the computing device; and

determining a direction that the user'"'"'s head is facing.

2. The method of claim 1, wherein the monitoring begins in response to the playing at least a portion of the media presentation beginning.

3. The method of claim 1, the annotation comprising a portion of the video input received by the one or more video input devices.

4. The method of claim 1, wherein the determining comprises determining that the user is positioned in front of the computing device.

5. The method of claim 1, wherein the computing device comprises an input device capable of determining a distance that the user is positioned from the computing device, the determining that the user is speaking in the direction of the computing device comprising determining that the user is positioned within a specified distance range in front of the computing device.

6. The method of claim 1, the computing device comprising a display, the media presentation being played back at the display during the playing, user interface elements related to annotation being absent from the display during the playing.

7. The method of claim 1, the determining that the user is speaking in the direction of the computing device being based only on the video input.

8. The method of claim 1, the receiving voice input comprising the annotation comprises capturing the annotation while a media element of the media presentation is playing, the method further comprising:

determining that the capturing of the annotation extends past a time that the media element stops playing in the media presentation; and

modifying the media presentation such that the media element is scheduled to be played for the duration of the annotation.

9. The method of claim 1, the method further comprising:

after the adding the annotation to the media presentation, playing the media presentation at the computing device or another computing device, the playing comprising playing the annotation.

10. The method of claim 1, wherein determining that the user is speaking in the direction of the computing device further comprises:

determining a three-dimensional profile of the user using the one or more video input devices, wherein the distance from the user and the direction that the user'"'"'s head is facing are obtained from the three-dimensional profile.

11. The method of claim 1, wherein determining, based at least in part upon the video input, that a user is speaking in the direction of the computing device comprises:

receiving a gesture command from the user to create an annotation.

12. The method of claim 11, wherein the gesture command is a hand, face, or body gesture recognized via the video input.

13. The method of claim 11, wherein the gesture command is a command to start recording the voice input for the annotation.

14. The method of claim 1, wherein determining the distance from the user to the computing device comprises:

determining that the user is within a specified distance range from the computing distance.

15. The method of claim 1, further comprising:

upon determining that the user is not speaking in the direction of the computing device;

determining that the voice input does not comprise an annotation to be added to the media presentation.

16. A computing device comprising:

a processing unit;

memory;

one or more audio input devices; and

one or more video input devices;

the computing device configured to perform operations for annotating a media presentation, the operations comprising;

playing at least a portion of the media presentation;

monitoring the one or more video input devices for video input received during the playing, wherein the one or more video input devices capture video input of a user during the playing of the media presentation;

upon determining, based at least in part upon the video input and without manual interaction with the computing device, that the user is speaking in a direction of the computing device;

creating an annotation by receiving, via the one or more audio input devices, voice input comprising the annotation to be added to the media presentation; and

adding the annotation to the media presentation;

wherein determining that the user is speaking in the direction of the computing device comprises;

determining a distance from the user to the computing device; and

determining a direction that the user'"'"'s head is facing.

17. The computing device of claim 16, wherein determining, based at least in part upon the video input, that a user intends to create an annotation comprises:

receiving a gesture command from the user that indicates the user'"'"'s intention to create an annotation.

18. The computing device of claim 17, wherein the gesture command is a hand, face, or body gesture recognized via the video input.

19. One or more computer-readable memory or storage devices storing computer executable code that when executed by a computing device performs operations for annotating a media presentation, the operations comprising:

playing at least a portion of the media presentation at the computing device comprising one or more audio input devices and one or more video input devices;

monitoring the one or more video input devices for video input received during the playing, wherein the one or more video input devices capture video input of a user during the playing of the media presentation;

upon determining, based at least in part upon the video input and without manual interaction with the computing device, that the user is speaking in a direction of the computing device;

creating an annotation by receiving, via the one or more audio input devices, voice input comprising the annotation to be added to the media presentation; and

adding the annotation to the media presentation;

wherein determining that the user is speaking in the direction of the computing device comprises;

determining a distance from the user to the computing device; and

determining a direction that the user'"'"'s head is facing.