Automated videography systems

• US 8,274,544 B2
• Filed: 03/23/2009
• Issued: 09/25/2012
• Est. Priority Date: 03/23/2009
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Kurtz, Andrew F., Blose, Andrew C.
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US12/408,898
• Publication Number:
  US 20100238262A1
• Time in Patent Office: 1,282 Days
• Field of Search: 348 1401- 1416
• US Class Current: 348/14.01
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A method of automated videography, in which video images of at least one subject in a local environment are acquired using an automated videography system to capture video images using one or more cameras during a videography event of one or more video scenes; applying current video capture settings which define subject framing and camera parameters; analyzing the video images of a current video scene to assess image quality relative to changes in subject activity and current subject framing; determining a need to change subject framing to improve video image quality based on the assessment of image quality; determining whether the changes in subject activity, relative to the subject framing, correspond to an intra-scene transition or an inter-scene transition; determining new subject framing and new video capture settings for a scene-transition, whether intra-scene or inter-scene; and automatically modifying video image capture in accordance with the newly determined video capture settings.

Claims

1. A method of automated videography, in which video images of at least one subject in a local environment are acquired using an automated videography system, comprising:

capturing video images with the automated videography system according to current video capture settings while using one or more cameras during a videography event of one or more video scenes which involves the at least one subject within the local environment, where the current video capture settings at least define current subject framing of the at least one subject within a selected shot;

analyzing the captured video images of a current video scene as captured according to the current video capture settings, to assess subject activity and changes thereof, relative to the current subject framing and video image quality;

determining whether the analyzed changes in subject activity, relative to the current subject framing, correspond to either an intra-scene transition or an inter-scene transition, for which an intra-scene change probability based on the analyzed changes in subject activity and current shot framing indicates that reframing to modify the current shot framing can adapt to the changing subject activity, or to an inter-scene transition, for which an inter-scene change probability based on the analyzed changes in subject activity indicates for a change to a selected new shot with new shot framing;

determining next shot options, including associated shot selection probabilities, based upon the assessed level of subject activity, if the determined inter-scene change probability for providing the new subject framing is greater than a predetermined value;

selecting a next shot from among the determined next shot options based in part upon the determined shot selection probabilities, and determining a next shot subject framing and associated video capture settings; and

automatically modifying ongoing video image capture to apply either the modified current shot framing or the next shot framing, as indicated by the inter-scene change, intra-scene change, and the shot selection probabilities.

2. The method according to claim 1 in which changing levels of subject activity are measured by subject activity metrics or a re-framing frequency, which are tracked over time during the videography event.

3. The method according to claim 2 wherein measured changes in subject activity levels trigger a reframing of video capture from a current framed shot to a new framed shot.

4. The method according to claim 2 wherein decreasing levels of subject activity are analyzed to determine that a current shot can be re-framed to a new tighter shot.

5. The method according to claim 2 wherein the reframing frequency is a measurable rate of reframing, by means of zooming, panning, tilting of a camera, image cropping, or camera switching, or combinations thereof.

6. The method according to claim 5 wherein the reframing frequency is measured in units of reframing events per minute or reframing events per shot or reframing events per video scene or other comparable units.

7. The method according to claim 2 wherein the subject activity metrics are determined relative to factors, selected from a group consisting of velocity, acceleration, frequency, position, direction, or a range of motion, a risk of subject loss, or combinations thereof, relative to the subject framing.

8. The method according to claim 2 wherein the changing levels of subject activity are associated with the arrival or departure of the at least one subject relative to at least a portion of the local environment that is subject to image capture.

9. The method according to claim 1 in which the shot selection is further determined based on the percentage area occupied by a face or head relative to the total image area, or on the basis of the number of subjects within the framed shot, or on the basis of subject activity, or combinations thereof.

10. The method according to claim 1 in which reframing of video capture from the current shot to the next shot is delayed by a defined re-centering or re-sizing time delay.

11. The method according to claim 10 in which the values for the defined re-centering and re-sizing time delay, which are measured in seconds or frames of elapsed video, depend on a number of subjects and the shot framing of the current shot or the next shot.

12. The method according to claim 1 wherein the analysis of image quality is relative to factors, selected from a group consisting of focus, image brightness, image contrast, image color properties, or combinations thereof.

13. The method according to claim 1 wherein modifying an ongoing video image capture to reframe from the current shot to the next shot occurs over a defined shot change transition time.

14. The method according to claim 1 wherein the movement of the at least one subject outside a defined action safe area for a current framing triggers a reframing of video capture from a current shot to the next shot after a shot reframing time delay has elapsed.

15. The method according to claim 1 in which the video capture of the current shot is reframed to re-center or re-size the at least one subject within the frame after a defined re-centering and re-sizing time delay has elapsed.

16. The method according to claim 1 in which the selection of available shots includes a long shot, a wide shot, a medium shot, a medium wide shot a close-up shot, or intermediate shots thereof.

17. The method according to claim 1 wherein the at least one subjects is nominally framed or re-centered to fall within a frame center area defined by maximum and minimum horizontal centering lines.

18. The method according to claim 1 wherein the face or head of the at least one subject is nominally framed or re-centered to fall within frame image areas defined by maximum and minimum vertical centering lines.

19. The method according to claim 1 wherein current shot framing or the next shot framing are constrained by subject provided spatial privacy limits.

20. The method according to claim 1 wherein the rate at which the video capture is reframed from the current shot to the next shot is limited by a defined maximum camera zoom rate or a maximum camera pan or tilt speed.

21. The method according to claim 1 wherein automatic analysis and modification of video image capture is defined by a subject selected automatic capture mode, the automatic capture modes including lock and follow, hierarchical, activity based, contextual cue based, or combinations thereof.

22. The method according to claim 1 wherein shot selection confidence values are calculated in determining the next shot selection probability.

23. The method according to claim 22 wherein the shot selection confidence values are used to determine or validate new video capture settings.

24. The method according to claim 1 wherein scene change confidence values, for the determined intra-scene change probability or for the determined inter-scene change probability, are calculated and then used to determine that an intra-scene transition or an inter-scene transition, respectively.

25. The method according to claim 1 wherein the intra-scene change probabilities, the inter-scene change probabilities, and the shot selection probabilities, are shot dependent, whether for the current shot, the next shot, or combinations thereof.

26. The method according to claim 24 wherein the scene change confidence values or shot change confidence values associated with the determined shot selection probabilities are tracked over time during a videography event.

27. The method according to claim 24 wherein the scene change confidence values or shot change confidence value associated with the determined shot selection probabilities are determined using a Bayesian probabilistic inference method, a Markovian probabilistic inference method, or the two in combination.

28. The method according to claim 1 wherein analysis of subject framing or subject activity to determine subsequent video capture settings or subject framing, is determined from identified contextual cues that include event classifications, user classifications, user identity, object identity, environmental conditions, user gestures, user behaviors, user privacy settings, or user audio cues, or combinations thereof.

29. The method according to claim 1 wherein the video capture settings, whether current or modified, include settings for video capture and shot framing, camera settings, audio settings, privacy settings, electronics settings, or combinations thereof.

30. The method according to claim 1, wherein the automated videography system further comprises a video communication system, in which the acquired video images acquired according to the defined capture settings, are provided over a network to a remote viewer.

31. The method according to claim 1 wherein the videography system comprises at least one wide field of view camera and at least one narrow field of view camera.

32. The method according to claim 1 wherein the videography system comprises an autonomous camera, comprising one or more camera modules, that captures and frames still images of the subject or environment.

33. An automated videography system comprising:

a data processing system;

a camera system communicatively connected to the data processing system;

a memory system communicatively connected to the data processing system and storing instructions configured to cause the data processing system to implement a method for automated videography of a subject in an environment, wherein the instructions comprise;

instructions for operating the camera system to defined camera settings that depend on subject activity and that determine a camera shot and an associated camera framing of the subject in the video images;

instructions for receiving, from the camera system, image data representing a captured video images of the environment or a portion thereof;

instructions for analyzing the captured video images and image data to assess image quality relative to changes in subject activity and current subject framing, using at least one subject activity metric;

instructions for determining a need to change the current subject framing, relative to improving video image quality and reducing a risk of subject loss relative to the assessed changes in subject activity, as indicated by the at least one subject activity metric;

instructions for responding to a determined need to change subject framing, by determining whether the assessed changes in subject activity relative to subject framing and the risk of subject loss, correspond to an intra-scene transition for which the analyzed changes in subject activity relative to the at least one subject activity metric are below a predetermined threshold and modifications to the current shot framing are consistent with that shot framing, or whether the assessed changes in subject activity correspond to an inter-scene transition for which the changes in subject activity are above a predetermined threshold to a selected new shot with new shot framing;

instructions for determining a new subject framing and new video capture settings corresponding to the determined shot framing for the determined scene transition, in accordance with either the determined intra-scene transition or the inter-scene transition; and

instructions for implementing the newly defined subject framing and video capture settings, with respect to the camera system or other functions of the automated videography system.

34. The system according to claim 33 wherein the instructions are provided to the videography system by means of a computer readable memory device, the device including, but not limited to, a CD-ROM, a DVD, a flash drive, or a hard disk device, or combinations thereof.

35. A method of automated videography, in which video images of at least one subject in a local environment are acquired using an automated videography system, comprising:

capturing video images with the automated videography system according to current video capture settings while using one or more cameras during a videography event consisting of one or more video scenes which involve the at least one subject and the local environment, where the current video capture settings define current subject framing within a selected shot and current camera parameters;

analyzing the captured video images of a current video scene as captured according to the current video capture settings, including determining subject activity using at least one subject activity metric appropriate to the current subject framing;

determining scene change probabilities, relative to the determined at least one subject activity metric and current shot subject motion thresholds, to determine whether video capture of the at least one subject can continue using the current subject framing or that framing of the at least one subject can be improved with new subject framing;

determining alternate shot options, including associated shot selection probabilities, based upon the determined subject activity metrics and associated shot dependent subject motion thresholds, if the determined scene change probability for new subject framing is greater than a predetermined value;

electing a next shot from among the determined alternate shot options based upon the determined shot selection probabilities; and

automatically modifying ongoing video image capture in accordance with the newly selected next shot, including associated new subject framing and any new video capture settings.

36. The method according to claim 35 wherein subject activity metrics are determined relative to a variety of factors, including a velocity, acceleration, frequency, position, direction, or a range, or combinations thereof, relative to the subject framing.

37. The method according to claim 35 wherein the subject activity metrics include a reframing frequency, which is a measurable rate of reframing the subject in video images, by means of zooming, panning, tilting of a camera, image cropping, or switching from images captured by one camera to images captured by another camera, or combinations thereof.

38. The method according to claim 35 wherein subject activity and shot framing are analyzed with respect to additional shot framing metrics, including an action safe area, a frame center area, shot change confidence values, or scene change confidence values, or combinations thereof.

39. The method according to claim 35 wherein changing levels of subject movement or subject activity are associated with the arrival or departure of the at least one subject relative to at least a portion of the environment that is being video captured.

40. The method according to claim 35 in which the instructions to re-reframe, in accordance with the determined modifications of the current shot framing or the determined new shot selection and new shot framing, further include instructions to delayed re-framing by a defined re-centering or re-sizing time delay.

41. A method of automated videography, in which video images of at least one subject in a local environment are acquired using an automated videography system, comprising:

capturing video images with the automated videography system according to current video capture settings while using one or more cameras during a videography event consisting of one or more video scenes which involve at least a first subject and the local environment, where the current video capture settings define current subject framing within a selected shot and current camera parameters;

analyzing the captured video images of a current video scene as captured according to the current video capture settings, to assess subject activity and changes thereof, relative to the current subject framing and the image quality thereof;

determining scene change probabilities, relative to the assessed subject activity, to determine whether video capture of the subject can continue using the current subject framing, or that the video image quality can be improved with a new subject framing;

determining next shot options, including associated shot selection probabilities, based upon the assessed level of subject activity, if the determined scene change probability for providing the new subject framing is greater than a predetermined value;

selecting a next shot from among the determined next shot options based upon the determined shot selection probabilities; and

automatically modifying ongoing video image capture in accordance with the newly selected next shot, including associated new subject framing and any new video capture settings.

42. The method according to claim 41 in which the shot selection is further determined by a variety of factors, including the percentage image area occupied by a face or head relative to the total image area, the number of subjects within the total image area, or the range or subject movement of one or more subjects within the total image area, or combinations thereof.

43. The method according to claim 41 in which the subject activity is measured relative to a variety of factors, including a velocity, acceleration, frequency, position, or a range, or combinations thereof, as compared to the subject framing in the video image.

44. The method according to claim 41 wherein the subject activity is measured by a reframing frequency, which is a measurable rate of reframing the subject in video images, by means of zooming, panning, tilting of a camera, image cropping, or switching from images captured by one camera to images captured by another camera, or combinations thereof.

45. The method according to claim 41 wherein either the video image quality is assessed or improved relative to a variety of factors, including, focus, image brightness, image contrast, image color properties, or subject framing, a risk of subject loss, or combinations thereof.

46. A method of automated videography, in which video images of at least one subject in a local environment are acquired using an automated videography system, comprising:

capturing video images with the video communication system according to current video capture settings while using one or more cameras, during a video communication event of one or more video scenes, which involves a subject within the local environment, where the current video capture settings at least define current subject framing within a selected shot;

analyzing the captured video images of a current video scene as captured according to the current video capture settings, to assess image quality relative to current subject framing and changes in subject activity, using at least one subject activity metric;

determining a need to change the current subject framing, relative to improving video image quality and reducing a risk of subject loss relative to the assessed changes in subject activity, as indicated by the at least one subject activity metric;

in response to a determined need to change subject framing, determining whether the analyzed changes in subject activity, relative to the subject framing and the risk of subject loss, correspond to an intra-scene transition or an inter-scene transition, where the analyzed changes in subject activity relative to the at least one subject activity metric are below a predetermined threshold, they correspond to the intra-scene transition for which modifications to the current shot framing are consistent with that shot framing, and where the changes in subject activity relative to the at least one subject activity metric are above a predetermined threshold, and they correspond to the inter-scene transition to a selected new shot having new shot framing;

determining new subject framing and new video capture settings corresponding to the determined shot framing for the determined scene-transition, whether intra-scene or inter-scene;

automatically modifying ongoing video image capture in accordance with the newly determined shot framing and video capture settings.

47. An automated videography system comprising:

a data processing system;

a camera system using one or more cameras communicatively connected to the data processing system;

a memory system communicatively connected to the data processing system and storing instructions configured to cause the data processing system to implement a method for automated videography of a subject in an environment, wherein the instructions comprise;

instructions for operating the camera system to capture video images according to current video capture settings during a videography event consisting of one or more video scenes which involve at least a first subject and the local environment, where the current video capture settings define current subject framing within a selected shot and current camera parameters;

instructions for analyzing the captured video images of a current video scene as captured according to the current video capture settings, to assess subject activity and changes thereof, relative to the current subject framing and the image quality thereof;

instructions for determining scene change probabilities, relative to the assessed subject activity, to determine whether video capture of the subject can continue using the current subject framing, or that the video image quality can be improved with a new subject framing;

instructions for determining alternate shot options, including associated shot selection probabilities, based upon the assessed level of subject activity, if the determined scene change probability for providing the new subject framing is greater than a predetermined value; and

instructions for selecting a next shot from among the determined alternate shot options based upon the determined shot selection probabilities, including determining the associated new subject framing and any new video capture settings;

wherein the automated videography system then implements the newly selected next shot to modify ongoing video image capture of the at least one subject by the camera system and data processor.