Automatic cropping of video content

• US 10,244,175 B2
• Filed: 09/10/2015
• Issued: 03/26/2019
• Est. Priority Date: 03/09/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Khoe, May-Li, Cieplinski, Avi E., Hart, David Joshua
• Primary Examiner(s): Segura, Cynthia
• Application Number:
  US14/850,677
• Publication Number:
  US 20160269645A1
• Time in Patent Office: 1,293 Days
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 25/443 by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming

Abstract

Electronic devices are often equipped with a camera for capturing video content and/or a display for displaying video content. However, amateur users often capture video content without regard to composition, framing, or camera movement, resulting in video content that can be jarring or confusing to viewers. There is a need to automate the processing and presentation of video content in an aesthetically pleasing manner. The embodiments described herein provide a method of automatically cropping video content for presentation on a display.

Claims

1. A method comprising:

at an electronic device comprising one or more processors, memory, a camera, and one or more sensors separate from the camera;

receiving video content captured by the camera;

receiving first information, other than video content and separate from the video content, about a subject in an environment that is captured in the video content, the first information sensed by the one or more sensors;

determining a first cropping for the video content based on the first information from the one or more sensors and the video content; and

generating first cropped video content for presentation on a display based on the determined first cropping.

2. The method of claim 1, the method further comprising:

receiving second information about a change in the environment sensed by the one or more sensors;

determining a second cropping for the video content based on the second information; and

generating second cropped video content for presentation on the display based on the determined second cropping.

3. The method of claim 2, the method further comprising:

determining, based on the second information, that the subject in the environment has moved out of a threshold area of a frame of the video content wherein the second cropping is determined so as to keep the subject within the threshold area in the second cropped video content.

4. The method of claim 3, wherein the second cropping is panned with respect to the first cropping.

5. The method of claim 3, wherein the second cropping is zoomed out with respect to the first cropping.

6. The method of claim 2, the method further comprising:

determining, based on the second information, that the subject in the environment has stopped moving, wherein the second cropping is determined in response to determining that the subject in the environment has stopped moving.

7. The method of claim 2, the method further comprising:

determining that the subject in the environment is within a threshold area of a frame of the video content, wherein the second cropping is determined in response to determining that the subject in the environment is within the threshold area, and the second cropping is zoomed in with respect to the first cropping.

8. The method of claim 2, wherein the first cropping is determined at a first time, the method further comprising:

determining that cropping criteria are met;

in response to determining that the cropping criteria are met;

in accordance with a determination that an elapsed time since the first time at which the first cropping was determined does not exceed a predetermined threshold time period, maintaining the first cropping;

in accordance with a determination that the elapsed time since the first time at which the first cropping was determined exceeds the predetermined threshold time period, providing the second cropped video content for presentation on the display.

9. The method of claim 2, the method further comprising:

determining, based on the second information whether the subject in the environment is active or inactive;

in accordance with a determination that the subject is active, emphasizing the subject in the second cropping with respect to the first cropping; and

in accordance with a determination that the subject is inactive, deemphasizing the subject in the second cropping with respect to the first cropping.

10. The method of claim 2, the method further comprising:

determining, based on the second information, a range of movement of the subject in the environment;

wherein the second cropping is zoomed out with respect to the first cropping in accordance with a determination that the range of movement of the subject is larger than a predetermined threshold; and

wherein the second cropping is zoomed in with respect to the first cropping in accordance with a determination that the range of movement of the subject is smaller than the predetermined threshold.

11. The method of claim 2, the method further comprising:

generating a transition from the first cropped video content to the second cropped video content.

12. The method of claim 1, wherein the display has one or more characteristics and determining the first cropping is based on the one or more characteristics of the display.

13. The method of claim 1, wherein determining the first cropping includes scaling the subject in the environment to life-size on the display.

14. The method of claim 1, the method further comprising:

determining respective locations of a plurality of subjects in a frame of the video content, wherein the first cropping is determined based on the respective locations.

15. The method of claim 1, the method further comprising:

selecting a cropping style, wherein the first cropping is determined based on the selected cropping style.

16. The method of claim 15, wherein the cropping style is automatically selected based on cropping style criteria.

17. The method of claim 1, wherein determining the first cropping is further based on metadata of the video content.

18. The method of claim 1, wherein the first cropping includes an entire frame of the video content, the method further comprising:

downsampling the first cropped video content to match a resolution of the display.

19. The method of claim 1, the method further comprising:

connecting a video call between the electronic device and an additional electronic device.

20. The method of claim 19, wherein the camera that captures the video content is connected to the electronic device that generates the first cropped video content and the display is connected to the additional electronic device.

21. The method of claim 19, wherein the camera that captures the video content is connected to the additional electronic device and the display is connected to the electronic device that generates the first cropped video content.

22. The method of claim 1, wherein the first information is independent of the video content.

23. The method of claim 1, wherein each of the one or more sensors is selected from the group consisting of a microphone, a depth camera, a thermal camera and a video camera.

24. The method of claim 1, wherein the first information is selected from the group consisting of a microphone information, a depth camera information, a thermal camera information and a video camera information.

25. An electronic device, comprising:

one or more processors;

memory;

a camera;

one or more sensors separate from the camera; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more processors including instructions for;

receiving video content captured by the camera;

receiving first information, other than video content and separate from the video content, about a subject in an environment that is captured in the video content, the first information sensed by the one or more sensors;

determining a first cropping for the video content based on the first information from the one or more sensors and the video content; and

generating first cropped video content for presentation on a display based on the determined first cropping.

26. The electronic device of claim 25, wherein the instructions are further for:

receiving second information about a change in the environment sensed by the one or more sensors;

determining a second cropping for the video content based on the second information; and

generating second cropped video content for presentation on the display based on the determined second cropping.

27. The electronic device of claim 26, wherein the instructions are further for:

determining, based on the second information, that the subject in the environment has moved out of a threshold area of a frame of the video content wherein the second cropping is determined so as to keep the subject within the threshold area in the second cropped video content.

28. The electronic device of claim 27, wherein the second cropping is panned with respect to the first cropping.

29. The electronic device of claim 27, wherein the second cropping is zoomed out with respect to the first cropping.

30. The electronic device of claim 26, wherein the instructions are further for:

determining, based on the second information, that the subject in the environment has stopped moving, wherein the second cropping is determined in response to determining that the subject in the environment has stopped moving.

31. The electronic device of claim 26, wherein the instructions are further for:

determining that the subject in the environment is within a threshold area of a frame of the video content, wherein the second cropping is determined in response to determining that the subject in the environment is within the threshold area, and the second cropping is zoomed in with respect to the first cropping.

32. The electronic device of claim 26, wherein the first cropping is determined at a first time, wherein the instructions are further for:

determining that cropping criteria are met;

in response to determining that the cropping criteria are met;

in accordance with a determination that an elapsed time since the first time at which the first cropping was determined does not exceed a predetermined threshold time period, maintaining the first cropping;

in accordance with a determination that the elapsed time since the first time at which the first cropping was determined exceeds the predetermined threshold time period, providing the second cropped video content for presentation on the display.

33. The electronic device of claim 26, wherein the instructions are further for:

determining, based on the second information whether the subject in the environment is active or inactive;

in accordance with a determination that the subject is active, emphasizing the subject in the second cropping with respect to the first cropping; and

in accordance with a determination that the subject is inactive, deemphasizing the subject in the second cropping with respect to the first cropping.

34. The electronic device of claim 26, wherein the instructions are further for:

determining, based on the second information, a range of movement of the subject in the environment;

wherein the second cropping is zoomed out with respect to the first cropping in accordance with a determination that the range of movement of the subject is larger than a predetermined threshold; and

wherein the second cropping is zoomed in with respect to the first cropping in accordance with a determination that the range of movement of the subject is smaller than the predetermined threshold.

35. The electronic device of claim 26, wherein the instructions are further for:

generating a transition from the first cropped video content to the second cropped video content.

36. The electronic device of claim 25, wherein the display has one or more characteristics and determining the first cropping is based on the one or more characteristics of the display.

37. The electronic device of claim 25, wherein determining the first cropping includes scaling the subject in the environment to life-size on the display.

38. The electronic device of claim 25, wherein the instructions are further for:

determining respective locations of a plurality of subjects in a frame of the video content, wherein the first cropping is determined based on the respective locations.

39. The electronic device of claim 25, wherein the instructions are further for:

selecting a cropping style, wherein the first cropping is determined based on the selected cropping style.

40. The electronic device of claim 39, wherein the cropping style is automatically selected based on cropping style criteria.

41. The electronic device of claim 25, wherein determining the first cropping is further based on metadata of the video content.

42. The electronic device of claim 25, wherein the first cropping includes an entire frame of the video content, wherein the instructions are further for:

downsampling the first cropped video content to match a resolution of the display.

43. The electronic device of claim 25, wherein the instructions are further for:

connecting a video call between the electronic device and an additional electronic device.

44. The electronic device of claim 43, wherein the camera that captures the video content is connected to the electronic device that generates the first cropped video content and the display is connected to the additional electronic device.

45. The electronic device of claim 43, wherein the camera that captures the video content is connected to the additional electronic device and the display is connected to the electronic device that generates the first cropped video content.

46. The electronic device of claim 25, wherein the first information is independent of the video content.

47. The electronic device of claim 25, wherein each of the one or more sensors is selected from the group consisting of a microphone, a depth camera, a thermal camera and a video camera.

48. The electronic device of claim 25, wherein the first information is selected from the group consisting of a microphone information, a depth camera information, a thermal camera information and a video camera information.

49. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device comprising one or more processors, memory, a camera, and one or more sensors separate from the camera, cause the electronic device to:

receive video content captured by the camera;

receive first information, other than video content and separate from the video content, about a subject in an environment that is captured in the video content, the first information sensed by the one or more sensors;

determine a first cropping for the video content based on the first information from the one or more sensors and the video content; and

generate first cropped video content for presentation on a display based on the determined first cropping.

50. The non-transitory computer readable storage medium of claim 49, wherein the instructions further cause the device to:

receive second information about a change in the environment sensed by the one or more sensors;

determine a second cropping for the video content based on the second information; and

generate second cropped video content for presentation on the display based on the determined second cropping.

51. The non-transitory computer readable storage medium of claim 50, wherein the instructions further cause the device to:

determine, based on the second information, that the subject in the environment has moved out of a threshold area of a frame of the video content wherein the second cropping is determined so as to keep the subject within the threshold area in the second cropped video content.

52. The non-transitory computer readable storage medium of claim 51, wherein the second cropping is panned with respect to the first cropping.

53. The non-transitory computer readable storage medium of claim 51, wherein the second cropping is zoomed out with respect to the first cropping.

54. The non-transitory computer readable storage medium of claim 50, wherein the instructions further cause the device to:

determine, based on the second information, that the subject in the environment has stopped moving, wherein the second cropping is determined in response to determining that the subject in the environment has stopped moving.

55. The non-transitory computer readable storage medium of claim 50, wherein the instructions further cause the device to:

determine that the subject in the environment is within a threshold area of a frame of the video content, wherein the second cropping is determined in response to determining that the subject in the environment is within the threshold area, and the second cropping is zoomed in with respect to the first cropping.

56. The non-transitory computer readable storage medium of claim 50, wherein the first cropping is determined at a first time, wherein the instructions further cause the device to:

determine that cropping criteria are met;

i

n response to determining that the cropping criteria are met;

in accordance with a determination that an elapsed time since the first time at which the first cropping was determined does not exceed a predetermined threshold time period, maintain the first cropping;

in accordance with a determination that the elapsed time since the first time at which the first cropping was determined exceeds the predetermined threshold time period, provide the second cropped video content for presentation on the display.

57. The non-transitory computer readable storage medium of claim 50, wherein the instructions further cause the device to:

determine, based on the second information whether the subject in the environment is active or inactive;

in accordance with a determination that the subject is active, emphasize the subject in the second cropping with respect to the first cropping; and

in accordance with a determination that the subject is inactive, deemphasize the subject in the second cropping with respect to the first cropping.

58. The non-transitory computer readable storage medium of claim 50, wherein the instructions further cause the device to:

determine, based on the second information, a range of movement of the subject in the environment;

wherein the second cropping is zoomed out with respect to the first cropping in accordance with a determination that the range of movement of the subject is larger than a predetermined threshold; and

wherein the second cropping is zoomed in with respect to the first cropping in accordance with a determination that the range of movement of the subject is smaller than the predetermined threshold.

59. The non-transitory computer readable storage medium of claim 50, wherein the instructions further cause the device to:

generate a transition from the first cropped video content to the second cropped video content.

60. The non-transitory computer readable storage medium of claim 49, wherein the display has one or more characteristics and determining the first cropping is based on the one or more characteristics of the display.

61. The non-transitory computer readable storage medium of claim 49, wherein determining the first cropping includes scaling the subject in the environment to life-size on the display.

62. The non-transitory computer readable storage medium of claim 49, wherein the instructions further cause the device to:

determine respective locations of a plurality of subjects in a frame of the video content, wherein the first cropping is determined based on the respective locations.

63. The non-transitory computer readable storage medium of claim 49, wherein the instructions further cause the device to:

select a cropping style, wherein the first cropping is determined based on the selected cropping style.

64. The non-transitory computer readable storage medium of claim 63, wherein the cropping style is automatically selected based on cropping style criteria.

65. The non-transitory computer readable storage medium of claim 49, wherein determining the first cropping is further based on metadata of the video content.

66. The non-transitory computer readable storage medium of claim 49, wherein the first cropping includes an entire frame of the video content, wherein the instructions further cause the device to:

downsample the first cropped video content to match a resolution of the display.

67. The non-transitory computer readable storage medium of claim 49, wherein the instructions further cause the device to:

connect a video call between the electronic device and an additional electronic device.

68. The non-transitory computer readable storage medium of claim 67, wherein the camera that captures the video content is connected to the electronic device that generates the first cropped video content and the display is connected to the additional electronic device.

69. The non-transitory computer readable storage medium of claim 67, wherein the camera that captures the video content is connected to the additional electronic device and the display is connected to the electronic device that generates the first cropped video content.

70. The non-transitory computer readable storage medium of claim 49, wherein the first information is independent of the video content.

71. The non-transitory computer readable storage medium of claim 49, wherein each of the one or more sensors is selected from the group consisting of a microphone, a depth camera, a thermal camera and a video camera.

72. The non-transitory computer readable storage medium of claim 49, wherein the first information is selected from the group consisting of a microphone information, a depth camera information, a thermal camera information and a video camera information.