ROLLING SHUTTER RECTIFICATION IN IMAGES/VIDEOS USING CONVOLUTIONAL NEURAL NETWORKS WITH APPLICATIONS TO SFM/SLAM WITH ROLLING SHUTTER IMAGES/VIDEOS

• US 20200134389A1
• Filed: 10/04/2019
• Published: 04/30/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Caelux Corporation, NEC Corporation
• Original Assignee: NEC Laboratories America Inc (NEC Corporation)
• Inventors: Tran, Quoc-Huy, Zhuang, Bingbing, Ji, Pan, Chandraker, Manmohan
• Granted Patent: US 11,132,586 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS

Abstract

A method for correcting rolling shutter (RS) effects is presented. The method includes generating a plurality of images from a camera, synthesizing RS images from global shutter (GS) counterparts to generate training data to train the structure-and-motion-aware convolutional neural network (CNN), and predicting an RS camera motion and an RS depth map from a single RS image by employing a structure-and-motion-aware CNN to remove RS distortions from the single RS image.

Claims

1. A computer-implemented method executed on a processor for correcting rolling shutter (RS) effects, the method comprising:

generating a plurality of images from a camera;

synthesizing RS images from global shutter (GS) counterparts to generate training data to train the structure-and-motion-aware convolutional neural network (CNN); and

predicting an RS camera motion and an RS depth map from a single RS image by employing a structure-and-motion-aware CNN to remove RS distortions from the single RS image.

2. The method of claim 1, further comprising, during a training phase, employing a set of GS images and corresponding GS depth maps to generate synthesized RS camera motions and synthesized RS depth maps.

3. The method of claim 2, further comprising providing the synthesized RS camera motions, the synthesized RS depth maps, and the synthesized RS images to the structure-and-motion-aware CNN as supervision signals and input data, respectively, to train the structure-and-motion-aware CNN.

4. The method of claim 3, further comprising, during a testing phase, providing the single RS image to the structure-and-motion-aware CNN to predict the RS camera motion and the RS depth map corresponding to the input RS image.

5. The method of claim 4, further comprising transmitting the single RS image, the RS camera motion, and the RS depth map to a rectification module to remove the RS effects.

6. The method of claim 5, further comprising outputting a rectified image from the rectification module.

7. The method of claim 1, wherein the structure-and-motion-aware CNN includes a depth network for learning the RS depth map and a velocity network for learning the RS camera motion.

8. The method of claim 7, wherein the depth network is trained separately from the velocity network.

9. A non-transitory computer-readable storage medium comprising a computer-readable program for correcting rolling shutter (RS) effects, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:

generating a plurality of images from a camera;

synthesizing RS images from global shutter (GS) counterparts to generate training data to train the structure-and-motion-aware convolutional neural network (CNN); and

predicting an RS camera motion and an RS depth map from a single RS image by employing a structure-and-motion-aware CNN to remove RS distortions from the single RS image.

10. The non-transitory computer-readable storage medium of claim 9, wherein, during a training phase, a set of GS images and corresponding GS depth maps are employed to generate synthesized RS camera motions and synthesized RS depth maps.

11. The transitory computer-readable storage medium of claim 10, wherein the synthesized RS camera motions, the synthesized RS depth maps, and the synthesized RS images are provided to the structure-and-motion-aware CNN as supervision signals and input data, respectively, to train the structure-and-motion-aware CNN.

12. The transitory computer-readable storage medium of claim 11, wherein, during a testing phase, the single RS image is provided to the structure-and-motion-aware CNN to predict the RS camera motion and the RS depth map corresponding to the input RS image.

13. The transitory computer-readable storage medium of claim 12, wherein the single RS image, the RS camera motion, and the RS depth map are transmitted to a rectification module to remove the RS effects.

14. The transitory computer-readable storage medium of claim 13, wherein a rectified image is output from the rectification module.

15. The transitory computer-readable storage medium of claim 9, wherein the structure-and-motion-aware CNN includes a depth network for learning the RS depth map and a velocity network for learning the RS camera motion.

16. The transitory computer-readable storage medium of claim 15, wherein the depth network is trained separately from the velocity network.

17. A system for correcting rolling shutter (RS) effects, the system comprising:

a memory; and

one or more processors in communication with the memory configured to;

generate a plurality of images from a camera;

synthesize RS images from global shutter (GS) counterparts to generate training data to train the structure-and-motion-aware convolutional neural network (CNN); and

predict an RS camera motion and an RS depth map from a single RS image by employing a structure-and-motion-aware CNN to remove RS distortions from the single RS image.

18. The system of claim 17, wherein, during a training phase, a set of GS images and corresponding GS depth maps are employed to generate synthesized RS camera motions and synthesized RS depth maps.

19. The system of claim 18, wherein the synthesized RS camera motions, the synthesized RS depth maps, and the synthesized RS images are provided to the structure-and-motion-aware CNN as supervision signals and input data, respectively, to train the structure-and-motion-aware CNN.

20. The system of claim 19, wherein, during a testing phase, the single RS image is provided to the structure-and-motion-aware CNN to predict the RS camera motion and the RS depth map corresponding to the input RS image.