SIMULATED VIDEO WITH EXTRA VIEWPOINTS AND ENHANCED RESOLUTION FOR TRAFFIC CAMERAS

• US 20110298928A1
• Filed: 06/03/2010
• Published: 12/08/2011
• Est. Priority Date: 06/03/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chen, Billy, Ofek, Eyal
• Granted Patent: US 8,823,797 B2
• Time in Patent Office: Days
• US Class Current: 348/149
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Simulated high resolution, multi-view video based on video input from low resolution, single-direction cameras is provided. Video received from traffic cameras, security cameras, monitoring cameras, and comparable ones is fused with patches from a database of pre-captured images and/or temporally shifted video to create higher quality video, as well as multiple viewpoints for the same camera.

Claims

1. A method to be executed at least in part in a computing device for generating an enhanced simulated video from a camera feed, the method comprising:

receiving an original video input from a single-direction camera;

generating a stream of vehicle video patches;

matching vehicles in the vehicle video patches to corresponding stored images of vehicles;

retrieving images of a different angle corresponding to the stored images of the vehicles;

fusing the retrieved images into a simulated video displaying the vehicles moving in another direction; and

rendering the simulated video.

2. The method of claim 1, wherein the retrieved images are back images of the vehicles and the simulated video represents traffic flow in opposite direction of camera direction.

3. The method of claim 1, wherein the stored images of the vehicles are based on at least one from a set of:

a make, a model, and a color of each vehicle.

4. The method of claim 1, the stored images of the vehicles are obtained from at least one from a set of:

images previously captured by the camera, images received from vehicle manufacturer, and images received from vehicle dealers.

5. The method of claim 1, further comprising:

enhancing the simulated video employing at least one from a set of;

a road image, a background image, a path of each vehicle, a speed of each vehicle, a recorded audio, and a generated audio.

6. The method of claim 1, further comprising:

buffering a predefined number of frames;

capturing images of vehicles in frames where the vehicles appear near the camera;

adjusting a size of the captured images;

superimposing the adjusted images over frames where the same vehicles appear away from the camera; and

fusing the superimposed frames into the simulated video.

7. The method of claim 6, wherein the buffering is performed for future frames.

8. The method of claim 6, wherein the vehicles in buffered frames are matched employing normalized cross-correlation.

9. The method of claim 6, wherein a number of the frames to be buffered is determined based on predefined distance ranges of vehicles from the camera.

10. The method of claim 9, wherein the predefined distance ranges include distant, medium, and near.

11. A system for generating an enhanced simulated video from a camera feed, the system comprising:

at least one single-directional camera;

a server coupled to the at least one single-directional camera, the server executing a video processing application, wherein the video processing application is configured to;

receive an original video input from the single-direction camera;

generate a stream of vehicle video patches;

buffer a predefined number of video frames;

capture images of a first set of vehicles in frames where the vehicles appear near the camera;

superimpose the images over frames where the first group of vehicles appear away from the camera;

match a second group of vehicles in the vehicle video patches to corresponding stored images of vehicles;

retrieve images of a different angle corresponding to the stored images of the vehicles;

fuse the superimposed images into a first simulated video with enhanced resolution;

fuse the retrieved images into a simulated second video displaying the second group of vehicles moving in another direction; and

render the simulated videos.

12. The system of claim 11, wherein the first and second videos are the same video stream.

13. The system of claim 11, wherein the video processing application is further configured to enable a user to select a viewing angle by selecting the images of the different angle based on user input providing the user a virtual rotation of the camera.

14. The system of claim 11, wherein the stored images include at least one of:

pre-captured images and synthetically generated images.

15. The system of claim 11, wherein the second video is generated by fusing the retrieved images over an orthographic map.

16. The system of claim 11, wherein the camera is one of a traffic camera, a monitoring camera, and a security camera.

17. The system of claim 11, further comprising a data store for storing at least one from a set of:

images of vehicles, buffered video frames, and fused videos.

18. A computer-readable storage medium with instructions stored thereon for generating an enhanced simulated video from a camera feed, the instructions comprising:

receiving an original video input from a single-direction camera;

generating a stream of vehicle video patches;

buffering a predefined number of frames;

matching the buffered frames employing normalized cross-correlation such that images of vehicles farther away from the camera are enhanced; and

fusing the matched frames into the simulated video.

19. The computer-readable medium of claim 18, wherein the instructions further comprise:

matching vehicles in the vehicle video patches to corresponding stored images of vehicles;

retrieving images of a different angle corresponding to the stored images of the vehicles; and

fusing the retrieved images into the simulated video displaying the vehicles moving in another direction.

20. The computer-readable medium of claim 18, wherein generating the stream of video patches, buffering the frames, and matching the frames are each performed by at least one from a set of:

a hardware module, a software module, and a combination of a hardware and software module.