Handling Video Frames Compromised By Camera Motion

• US 20150163407A1
• Filed: 12/09/2013
• Published: 06/11/2015
• Est. Priority Date: 12/09/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Khoury, Issa
• Granted Patent: US 9,407,823 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G11B 27/02 Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/685 performed by mechanical compensation
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure

Abstract

Example embodiments concern reducing the negative impacts of camera motion during video capture. Processing, transmitting, or displaying motion compromised video data yields inefficient use of bandwidth, battery power, memory, and other resources. Watching video acquired from a moving video camera may induce motion sickness. Unlike conventional systems that attempt to address motion by increasing frame capture rates to produce better individual frames, example embodiments identify frames that are compromised by motion data and do not provide or present them for viewing. Compromised frames may be logically or physically deleted from a video stream or memory. Motion may be detected by physical apparatus (e.g., accelerometer) or through image processing (e.g., motion artifacts, low correlation between frames, low signal-to-noise ratio). A display device may be controlled to operate in a video display mode or a still frame mode depending on motion detected in captured video frames.

Claims

1. A method, comprising:

accessing a frame captured by a video camera;

determining a viewing factor for the frame as a function of motion of the video camera at the time the frame was captured; and

controlling how the frame will be presented for viewing as a function of the viewing factor.

2. The method of claim 1, the method comprising controlling, as a function of the viewing factor, whether the video camera will operate in a video camera mode or will operate in a still frame mode.

3. The method of claim 2, where operating in the still frame mode includes adjusting a frame capture rate for the video camera to below twenty four frames per second and adjusting a shutter speed for the video camera to faster than 1/500 second.

4. The method of claim 2, where operating in the video camera mode includes adjusting a frame capture rate for the video camera to at least twenty four frames per second and adjusting a shutter speed for the video camera to slower than 1/500 second.

5. The method of claim 1, where accessing the frame occurs on a first apparatus that captured the frame, where determining the viewing factor occurs on the first apparatus, where controlling how the frame will be presented occurs on the first apparatus, where accessing the frame occurs on a second apparatus that did not capture the frame, where determining the viewing factor occurs on the second apparatus, or where controlling how the frame will be presented occurs on the second apparatus.

6. The method of claim 1, where the motion of the video camera is determined from accelerometer data associated with the video camera at the time the frame was captured, from gyroscopic data associated with the video camera at the time the frame was captured, from single frame motion analysis data associated with the frame, or from frame-to-frame motion analysis data associated with the frame and one or more other frames acquired from the video camera.

7. The method of claim 1, where controlling how the frame will be presented for viewing includes controlling whether the frame will be transmitted from an apparatus that captured the frame.

8. The method of claim 1, where controlling how the frame will be presented for viewing includes deleting the frame from a memory.

9. The method of claim 1, comprising, upon determining that the viewing factor does not satisfy a viewing standard, replacing the frame with a previously acquired frame.

10. The method of claim 9, comprising compressing the previously acquired frame into a compressed frame before replacing the frame with the previously acquired frame.

11. The method of claim 1, comprising, upon determining that the viewing factor does not satisfy a viewing standard, providing a signal that indicates that the frame does not satisfy the viewing standard.

12. The method of claim 1, comprising:

computing a continuity factor for a series of frames captured by the video camera, where the continuity factor is based, at least in part, on the viewing factor for members of the series of contiguous frames, and

upon determining that the continuity factor does not satisfy a continuity standard, controlling a device that would have displayed the series of frames if the continuity factor did satisfy the continuity standard to enter a still frame display mode.

13. The method of claim 12, comprising:

upon determining that the continuity factor does satisfy the continuity standard, controlling the device to enter a video display mode.

14. The method of claim 12, where the still frame display mode is configured to selectively display a single frame that satisfies the viewing standard as the single frame becomes available, where the single frame is displayed for at least a pre-determined single frame period of time.

15. The method of claim 12, where the still frame display mode is configured to display a previously provided frame that satisfied the viewing standard until the continuity standard is satisfied.

16. A computer-readable storage medium storing computer-executable instructions that when executed by a computer cause the computer to perform a method, the method comprising:

accessing a frame captured by a video camera, where accessing the frame occurs on a first apparatus that captured the frame or on a second apparatus that did not capture the frame;

determining a viewing factor for the frame as a function of motion of the video camera at the time the frame was captured, where the motion of the video camera is determined from accelerometer data associated with the video camera at the time the frame was captured, from gyroscopic data associated with the video camera at the time the frame was captured, from single frame motion analysis data associated with the frame, or from frame-to-frame motion analysis data associated with the frame and one or more other frames acquired from the video camera, where determining the viewing factor occurs on the first apparatus or the second apparatus;

controlling how the frame will be presented for viewing as a function of the viewing factor, where controlling how the frame will be presented for viewing includes controlling whether the frame will be transmitted from an apparatus that captured the frame, or whether the frame will be deleted from a memory, where controlling how the frame will be presented occurs on the first apparatus or the second apparatus,

controlling, as a function of the viewing factor, whether the video camera will operate in a video camera mode or will operate in a still frame mode, where operating in the still frame mode includes adjusting a frame capture rate for the video camera to below twenty four frames per second and adjusting a shutter speed for the video camera to faster than 1/500 second, and where operating in the video camera mode includes adjusting a frame capture rate for the video camera to at least twenty four frames per second and adjusting a shutter speed for the video camera to slower than 1/500 second;

upon determining that the viewing factor does not satisfy a viewing standard, providing a signal that indicates that the frame does not satisfy the viewing standard, compressing a previously acquired frame into a compressed frame, and replacing the frame with the previously acquired frame;

computing a continuity factor for a series of contiguous frames captured by the video camera, where the continuity factor is based, at least in part, on the viewing factor for members of the series of contiguous frames, upon determining that the continuity factor does not satisfy a continuity standard, controlling a device that would have displayed the series of contiguous frames if the continuity factor did satisfy the continuity standard to enter a still frame display mode, where the still frame display mode is configured to selectively display a single frame that satisfies the viewing standard as the single frame becomes available or to display a previously provided frame that satisfied the viewing standard until the continuity factor is satisfied, and upon determining that the continuity factor does satisfy a continuity standard, controlling the device to enter a video display mode.

17. An apparatus, comprising:

a processor;

a memory configured to store a set of frames acquired by a video camera;

a set of logics; and

an interface configured to connect the processor, the memory, and the set of logics;

the set of logics comprising;

a first logic configured to produce motion data that describes how the video camera was moving while the video camera acquired members of the set of frames, where the motion data is a function of gyroscopic data produced by the video camera, accelerometer data produced by the video camera, single frame motion analysis, or frame-to-frame motion analysis; and

a second logic configured to control whether members of the set of frames are displayed as a continuous video display or as selected still frames based, at least in part, on the motion data, where members of a first series of frames acquired while the motion data does not satisfy a motion threshold are presented in a first discrete display mode and where a second series of frames acquired while the motion data does satisfy the motion threshold is presented in a second continuous display mode.

18. The apparatus of claim 17, comprising a third logic configured to control a frame capture parameter for the video camera based, at least in part, on the motion data, where the frame capture parameter is frame capture rate or shutter speed.

19. The apparatus of claim 18, where the motion data includes a per frame motion data or a per clip motion data, where a clip is a set of related frames acquired within a pre-determined time period.

20. The apparatus of claim 19, where the second logic is configured to selectively exclude individual members of the set of frames based on the per frame motion data, to selectively exclude individual members of the set of frames based on the per clip motion data, or to selectively exclude subsets of members of the set of frames based on the per clip motion data.