AUTOMATED VIDEOGRAPHY SYSTEMS

• US 20100238262A1
• Filed: 03/23/2009
• Published: 09/23/2010
• Est. Priority Date: 03/23/2009
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Intellectual Ventures Fund 83 LLC (Intellectual Ventures LLC)
• Inventors: Kurtz, Andrew F., Blose, Andrew C.
• Granted Patent: US 8,274,544 B2
• Time in Patent Office: Days
• US Class Current: 348/14.10
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A method of automated videography, in which video images of at least one subject in a local environment are acquired using an automated videography system to capture video images using one or more cameras during a videography event of one or more video scenes; applying current video capture settings which define subject framing and camera parameters; analyzing the video images of a current video scene to assess image quality relative to changes in subject activity and current subject framing; determining a need to change subject framing to improve video image quality based on the assessment of image quality; determining whether the changes in subject activity, relative to the subject framing, correspond to an intra-scene transition or an inter-scene transition; determining new subject framing and new video capture settings for a scene-transition, whether intra-scene or inter-scene; and automatically modifying video image capture in accordance with the newly determined video capture settings.

Claims

1. A method of automated videography, in which video images of at least one subject in a local environment are acquired using an automated videography system, comprising:

capturing video images with the automated videography system using one or more cameras during a videography event of one or more video scenes which involve a first subject and the local environment;

applying current video capture settings which define subject framing within a selected shot, and camera parameters;

analyzing the video images of a current video scene to assess image quality relative to changes in subject activity and current subject framing;

determining a need to change subject framing to improve video image quality based on the assessment of image quality;

determining whether the changes in subject activity, relative to the subject framing, correspond to an intra-scene transition or an inter-scene transition;

determining new subject framing and new video capture settings for a scene-transition, whether intra-scene or inter-scene; and

automatically modifying ongoing video image capture in accordance with the newly determined video capture settings.

2. The method according to claim 1 in which changing levels of subject activity are measured by a subject movement factor or a re-framing frequency, which are tracked over time during the videography event.

3. The method according to claim 2 wherein measured changes in subject activity levels trigger a reframing of video capture from a current framed shot to a new framed shot.

4. The method according to claim 2 wherein decreasing levels of subject activity are analyzed to determine that a current shot can be re-framed to a new tighter shot.

5. The method according to claim 2 wherein the reframing frequency is a measurable rate of reframing, by means of zooming, panning, tilting of a camera, image cropping, or camera switching, or combinations thereof.

6. The method according to claim 5 wherein the reframing frequency is measured in units of reframing events per minute or reframing events per shot or reframing events per video scene or other comparable units.

7. The method according to claim 2 wherein the subject movement factor is measured as a normalized product of subject movement relative to an imaged field of view, where the subject movement factor is determined relative to a variety of factors, including a velocity, acceleration, frequency, position, direction, or a range, or combinations thereof relative to the subject framing.

8. The method according to claim 2 wherein the changing levels of subject activity are associated with the arrival or departure of at least one subject relative to at least a portion of the local environment that is subject to image capture.

9. The method according to claim 1 in which the video images are framed as shots, and the shot selection is determined based on the percentage area occupied by a face or head relative to the total image area, or on the basis of the number of subjects within the framed shot, or on the basis of subject activity, or combinations thereof.

10. The method according to claim 1 in which reframing of video capture from a current framed shot to an alternate shot framing is delayed by a defined re-centering or re-sizing time delay.

11. The method according to claim 10 in which the values for the defined re-centering and re-sizing time delay, which are measured in seconds or frames of elapsed video, depend on a number of subjects and a shot framing.

12. The method according to claim 10 wherein the defined re-centering and re-sizing time delays are applied to enhance a viewing experience of a remote viewer.

13. The method according to claim 1 wherein modifying an ongoing video image capture to reframe from a current shot to a new shot occurs over a defined shot change transition time.

14. The method according to claim 1 wherein the movement of a subject outside a defined action safe area for a current framing triggers a reframing of video capture from a current framed shot to a new framed shot after an inter-scene shot reframing time delay has elapsed.

15. The method according to claim 1 in which the video capture of a current framed shot is reframed to re-center or re-size a subject within the frame after a defined re-centering and re-sizing time delay has elapsed.

16. The method according to claim 1 in which the video images are framed as shots, and the selection of available shots includes a long shot, a wide shot, a medium shot, a medium wide shot a close-up shot, or intermediate shots thereof.

17. The method according to claim 1 wherein subjects are nominally framed or re-centered to fall within a frame center area defined by maximum and minimum horizontal centering lines.

18. The method according to claim 1 wherein the faces or heads of subjects are nominally framed or re-centered to fall within frame image areas defined by maximum and minimum vertical centering lines.

19. The method according to claim 1 wherein current shot framing or new shot framing are constrained by subject provided spatial privacy limits.

20. The method according to claim 1 wherein the rate at which the video capture is reframed from a current shot to a new shot is limited by a defined maximum camera zoom rate or a maximum camera pan or tilt speed.

21. The method according to claim 1 wherein automatic analysis and modification of video image capture is defined by a subject selected automatic capture mode, the automatic capture modes including lock and follow, hierarchical, activity based, contextual cue based, or combinations thereof.

22. The method according to claim 1 wherein shot change confidence values are calculated as probabilities in determining a framing for a subsequent shot to follow a current shot framing.

23. The method according to claim 22 wherein shot change confidence values are used to determine or validate new video capture settings.

24. The method according to claim 1 wherein scene change confidence values, including an intra-scene change probability or an inter-scene change probability, are calculated and then used to determine that an intra-scene transition or an inter-scene transition, respectively.

25. The method according to claim 24 wherein scene change confidence values are used to determine or validate new video capture settings.

26. The method according to claim 24 wherein scene change confidence values or shot change confidence values are tracked over time during a videography event.

27. The method according to claim 24 wherein scene change confidence values or shot change confidence value are determined using a Bayesian probabilistic inference method, a Markovian probabilistic inference method, or the two in combination.

28. The method according to claim 1 wherein analysis of subject framing or subject activity to determine subsequent video capture settings or subject framing, is determined from identified contextual cues that include event classifications, user classifications, user identity, object identity, environmental conditions, user gestures, user behaviors, user privacy settings, or user audio cues, or combinations thereof.

29. The method according to claim 1 wherein the video capture settings, whether current or modified, include settings for video capture and shot framing, camera settings, audio settings, privacy settings, electronics settings, or combinations thereof.

30. The method according to claim 1, wherein the automated videography system further comprises a video communication system, in which the acquired video images acquired according to the defined capture settings, are provided over a network to a remote viewer.

31. The method according to claim 1 wherein the videography system comprises at least one wide field of view camera and at least one narrow field of view camera.

32. The method according to claim 1 wherein the videography system comprises an autonomous camera, comprising one or more camera modules, that captures and frames still images of the subject or environment.

33. A method for framing one or more subjects captured on video, the method comprising the steps of:

receiving video of a first subject in an environment from a camera;

determining a current shot framing of the first subject in the video, with a computer and an associated image processor, relative to a shot selection and subject positioning within the framed shot;

determining a magnitude and a direction of movement of the first subject in the received video, relative to the current framing and level of motion thresholds;

analyzing the subject movement of the first subject, relative to the determined magnitude and direction of subject movement, and the current framing, to determine that the subject is properly framed by the current shot framing, or to determine modifications of the current shot framing that are required to capture the moving first subject, or to determine a new shot selection and new shot framing that are required to capture the moving first subject; and

instructing the image processor or camera to re-frame the first subject if the first subject is determined not to be properly framed by the current shot framing, in accordance with the determined modifications of the current shot framing or the determined new shot selection and new shot framing.

34. The method according to claim 33 wherein subject motion is measured relative to a variety of factors, including a velocity, acceleration, frequency, position, direction, or a range, or combinations thereof, relative to the subject framing.

35. The method according to claim 33 wherein the subject movement is measured by a reframing frequency, which is a measurable rate of reframing the subject in video images, by means of zooming, panning, tilting of a camera, image cropping, or switching from images captured by one camera to images captured by another camera, or combinations thereof.

36. The method according to claim 33 wherein subject motion and shot framing are analyzed with respect to additional image framing metrics, including an action safe area, a frame center area, shot change confidence values, or scene change confidence values, or combinations thereof.

37. The method according to claim 33 wherein changing levels of subject movement or subject activity are associated with the arrival or departure of at least one subject relative to at least a portion of the environment that is being video captured.

38. The method according to claim 33 in which the instructions to re-reframe, in accordance with the determined modifications of the current shot framing or the determined new shot selection and new shot framing, further include instructions to delayed re-framing by a defined re-centering or re-sizing time delay.

39. A method for framing a subject in a local environment in video captured by a video capture system, the video capture system comprising a camera and the method comprising the steps of:

receiving a first set of video images from the video capture system, the first set of video images comprising the subject and the subject'"'"'s local environment;

analyzing the first set of video for subject motion and current subject framing;

comparing subject motion to a level-of-motion threshold associated with the current subject framing;

establishing a first image quality based on the first analysis and first comparison;

instructing the video capture system to change its framing of the subject based on the first image quality;

receiving a second set of video images from the video capture system, the second set of video images being received after the first set of video images, and the second set of video images involving the subject and the subject'"'"'s local environment;

analyzing the second set of video images for subject motion and second subject framing;

comparing subject motion to a level-of-motion threshold associated the second subject framing;

establishing a second image quality based on the second analysis and the second comparison; and

instructing the video capture system to change its framing of the subject based on the second image quality.

40. The method according to claim 39 in which the video images are framed as shots, and the shot selection is determined by a variety of factors, including the percentage image area occupied by a face or head relative to the total image area, the number of subjects within the total image area, or the range or subject movement of one or more subjects within the total image area, or combinations thereof.

41. The method according to claim 39 in which subject motion is measured relative to a variety of factors, including a velocity, acceleration, frequency, position, or a range, or combinations thereof, as compared to the subject framing in the video image.

42. The method according to claim 39 wherein the subject movement is measured by a reframing frequency, which is a measurable rate of reframing the subject in video images, by means of zooming, panning, tilting of a camera, image cropping, or switching from images captured by one camera to images captured by another camera, or combinations thereof.

43. The method according to claim 42 wherein decreasing levels of subject motion are analyzed relative to decreasing level of motion thresholds to determine that the current shot framing can be changed to a to a tighter shot.

44. The method according to claim 42 wherein increasing levels of subject activity are analyzed to determine that the current shot can be re-framed to a new wider shot.

45. The method according to claim 39 further comprising the step of instructing transmission of at least the first set of video images, the second set of video images, subsequently captured video images, or combinations thereof to a remote system in accordance with a video conferencing application.

46. The method according to claim 39, wherein either the first or second image quality is assessed relative to a variety of factors, including, focus, image brightness, image contrast, image color properties, or subject framing, or combinations thereof.

47. The method according to claim 39 wherein either of the steps of instructing the video capture system further comprise instructing the video capture system at least to re-center its framing of the subject.

48. An automated videography system comprising:

a data processing system;

a camera system communicatively connected to the data processing system;

a memory system communicatively connected to the data processing system and storing instructions configured to cause the data processing system to implement a method for automated videography of a subject in an environment, wherein the instructions comprise;

instructions for operating the camera system to defined camera settings that depend on subject activity and that determine camera framing of the subject in the video images;

instructions for receiving, from the camera system, image data representing a captured video images of the environment or a portion thereof;

instructions for analyzing the captured video images to assess image quality relative to changes in subject activity and current subject framing, and particularly to determine whether identified changes in subject activity correspond to an intra-scene transition or an inter-scene transition;

instructions for determining a new subject framing and new video capture settings in accordance with either the identified intra-scene transition or the inter-scene transition; and

instructions for implementing any newly defined video capture settings, with respect to the camera system or other functions of the automated videography system.

49. The system according to claim 48 wherein the instructions are provided to the videography system by means of a computer readable memory device, the device including, but not limited to, a CD-ROM, a DVD, a flash drive, or a hard disk device, or combinations thereof.

50. An automated videography system, comprising:

an image display device adapted to display video images;

at least one image capture device which acquires video images of a local environment having a subject therein, according to defined video capture settings;

a computer cooperating with the at least one image capture device and an image processor to generate video signals based upon the acquired video images during a videography event;

said computer further providing a contextual interface that determines the framing of a subject in the acquired video images, where framing is defined by a shot selection and subject centering within the shot;

the contextual interface further including an intra-scene analysis algorithm for identifying potential intra-scene transitions relative to subject framing during the videography event and a scene capture management algorithm for determining intra-scene adjustments in video capture settings or subject framing when an intra-scene transition is detected;

the contextual interface further including a transition test with an inter-scene analysis algorithm, for identifying potential inter-scene transitions in subject activities, and a transition process structure for determining inter-scene adjustments in video capture settings or shot framing when an inter-scene transition is detected; and

wherein the system then automatically modifies ongoing video image capture in accordance with the newly determined video capture settings or shot framing as determined for an intra-scene transition or an inter-scene transition.