Digital processing of video images

• US 8,599,266 B2
• Filed: 07/01/2003
• Issued: 12/03/2013
• Est. Priority Date: 07/01/2002
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Trivedi, Mohan M., Huang, Kohsia S.
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): FINDLEY, CHRISTOPHER G
• Application Number:
  US10/519,818
• Publication Number:
  US 20060187305A1
• Time in Patent Office: 3,808 Days
• Field of Search: 348/143, 348/169
• US Class Current: 348/169
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/164 using holistic features
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Digital video imaging systems and techniques for efficiently transforming warped video images into rectilinear video images, real-time tracking of persons and objects, face recognition of persons, monitoring and tracking head pose of a person and associated perspective view of the person.

Claims

1. A system, comprising:

an array of video cameras to capture warped panoramic video images of a scene and to produce a video stream; and

a digital processor to receive and process said video stream, said digital processor comprising;

a first processing module to unwarp said warped panoramic video images to produce rectilinear video images of the scene, wherein said first processing module provides at least one of pan, tilt, and zoom adjustments to allow for customized viewing of the scene in the produced rectilinear video images in a first perspective view,

a second processing module to detect and track a person'"'"'s head in the rectilinear video images and to extract video images in the tracked person'"'"'s direction of view from the rectilinear video images so as to produce rectilinear video images in a second perspective view different from the first perspective view, and

a third processing module to process images of a face from streaming video images for face recognition, wherein said third processing module performs a single-frame subspace feature analysis on multiple segments of the streaming video images to produce a sequence of classification results and a sequence of feature vectors, each classification result and feature vector corresponding to a length L segment of streaming video images, where L is an integer greater than 1, and then processes said sequences separately to produce video-based face recognition outputs;

wherein the array of video cameras comprises an omni-direction video camera to capture the warped panoramic video images in a full 360-degree view, and

where the customized viewing of the scene in the produced rectilinear video images comprises at least one view from the full 360-degree view.

2. The system as in claim 1, wherein said second processing module performs an edge detection in the rectilinear video images to extract features and an ellipse detection to extract possible head images.

3. The system as in claim 1, wherein said second processing module performs Kalman filtering to predict an orientation of and track motion of the person'"'"'s head.

4. The system as in claim 1, further comprising a mechanism to produce a warning signal according to the orientation of the person'"'"'s head.

5. The system as in claim 1, wherein said third processing module is configured to apply a majority decision rule in processing the sequence of classification results.

6. The system as in claim 1, wherein said third processing module is configured to apply a discrete hidden Markov model decision rule in processing the sequence of classification results.

7. The system as in claim 1, wherein said third processing module is configured to apply a continuous density hidden Markov model decision rule in processing the sequence of feature vectors.

8. The system as in claim 1, further comprising a mechanism to measure a facial temperature pattern of a person.

9. The system as in claim 1, further comprising a mechanism for performing a facial affect analysis on a person.

10. The system as in claim 1, further comprising a mechanism for performing a speech affect analysis.

11. The system as in claim 1, further comprising a plurality of video cameras to capture warped panoramic video images of the same scene from different locations, wherein said digital processor processes signals from said array of video cameras and said one video camera to detect and tract movement of an object in the scene.

12. The system as in claim 11, wherein said digital processor performs shadow detection from each video signal from each video camera to segment the object from the scene.

13. The system as in claim 12, wherein said digital processor performs a triangulation according to positions of the video cameras to produce horizontal positions x, y of the object and a height estimation algorithm to produce averaged vertical position z of the object.

14. The system as in claim 13, wherein said digital processor registers a track of movement for the object according to x and y positions.

15. The system as in claim 13, wherein the triangulation is performed by using an extended N-ocular algorithm.

16. A system, comprising:

an array of video cameras to capture warped panoramic video images of a scene and to produce a video stream;

a video transmission mechanism to deliver said video stream to a plurality of video receivers, said video transmission mechanism comprising a video server which filters information in said video stream according to security levels assigned to video receivers to produce different filtered video streams to different video receivers; and

a digital processor in each video receiver to independently process said video stream to unwarp said warped panoramic video images to produce rectilinear video images of the scene, said digital processor at each video receiving having a user graphic interface with at least one of pan, tilt, and zoom adjustments to allow for customized viewing at each video receiver that are independent from each another and represent different perspective views,

wherein the digital processor further processes images of a face from said video stream for face recognition, by performing a single-frame subspace feature analysis on multiple segments of the video stream images to produce a sequence of classification results and a sequence of feature vectors, each classification result and feature vector corresponding to a length L segment of streaming video images, where L is an integer greater than 1, and then processes said sequences separately to produce video-based face recognition outputs;

wherein the array of video cameras comprises an omni-direction video camera to capture the warped panoramic video images in a full 360-degree view, and

where the customized viewing of the scene in the produced rectilinear video images comprises at least one view from the full 360-degree view.

17. The system as in claim 16, wherein said video servo includes a digital process to tracks a change in the scene and adjusts filtering in a filtered video stream according to the change in the scene.

18. The system as in claim 16, wherein one video receiver includes a PDA.

19. The system as in claim 16, wherein one video receiver includes a laptop computer.

20. The system as in claim 16, wherein one video receiver includes a desktop computer.

21. The system as in claim 16, where said video camera is an omni-directional video camera to capture a 360-degree view of the scene.

22. The system as in claim 16, wherein said digital processor includes a module that detects and tracks a person'"'"'s head.

23. The system as in claim 22, wherein said module further detects a face orientation of the person.

24. The system as in claim 23, wherein said module further extracts a video image in the person'"'"'s view from the video according to estimated face orientation.

25. The system as in claim 16, wherein said digital processor includes a tracking module to detect and track a location of an object or a person in real time.

26. The system as in claim 16, wherein said video transmission mechanism includes a wired communication link.

27. The system as in claim 16, wherein said video transmission mechanism includes a wireless communication link.

28. The system as in claim 16, wherein said video transmission mechanism includes a video server that removes selected image information from a video signal to send a modified video signal to a video receiver.

29. A system, comprising:

an array of video cameras to capture warped panoramic video images of a scene and to produce a video stream; and

a digital processor to receive and process said video stream, said digital processor comprising;

a first processing module to unwarp said warped panoramic video images to produce rectilinear video images of the scene, wherein said first processing module provides at least one of pan, tilt, and zoom adjustments to allow for customized viewing of the scene in a first perspective view, and

a second processing module to extract a face from the streaming rectilinear video images, to perform face recognition on the extracted face, and to generate images viewed by the recognized face in a second perspective view independent of the first perspective view based on estimating an orientation of the face in the rectilinear video images, and

a third processing module to perform face recognition by performing a single-frame subspace feature analysis on multiple segments of the streaming rectilinear video images to produce a sequence of classification results and a sequence of feature vectors, each classification result and feature vector corresponding to a length L segment of streaming video images, where L is an integer greater than 1, and then processes said sequences separately to produce face recognition outputs;

wherein the array of video cameras comprises an omni-direction video camera to capture the warped panoramic video images in a full 360-degree view, and

where the customized viewing of the scene in the produced rectilinear video images comprises at least one view from the full 360-degree view.

30. A system, comprising:

an array of video cameras to capture warped panoramic video images of a scene and to produce a video stream;

a video transmission mechanism to deliver said video stream to a first video receiver and a second video receiver

a digital processor in said first video receiver to process said video stream to unwarp said warped panoramic video images to produce rectilinear video images of the scene,

wherein said digital processor in said first video receiver includes a user graphic interface with at least one of pan, tilt, and zoom adjustments to allow for customized viewing of the scene in a first perspective view, and said digital processor is operable to superimpose a customized video of the scene over a digital image; and

another digital processor in said second video receiver to process said video stream to unwarp said warped panoramic video images to produce rectilinear video images of the scene, wherein the other digital processor in said second video receiver includes another user graphic interface with at least one of pan, tilt, and zoom adjustments to allow for customized viewing of the scene in a second perspective view independent of the digital processor in the first video receiver so that the second perspective view is capable of being different from the first perspective view,

wherein the digital processor further processes images of a face from said video stream for face recognition, by performing a single-frame subspace feature analysis on multiple segments of video stream images to produce a sequence of classification results and a sequence of feature vectors, each classification result and feature vector corresponding to a length L segment of streaming video images, where L is an integer greater than 1, and then processes said sequences separately to produce video-based face recognition outputs,

wherein the array of video cameras comprises an omni-direction video camera to capture the warped panoramic video images in a full 360-degree view, and

where the customized viewing of the scene in the produced rectilinear video images comprises at least one view from the full 360-degree view.

31. The system as in claim 30, wherein said digital image is a map of an area including the scene.