Method and apparatus for intelligent and automatic sensor control using multimedia database system

• US 8,614,741 B2
• Filed: 03/31/2003
• Issued: 12/24/2013
• Est. Priority Date: 03/31/2003
• Status: Active Grant

• Current Assignee: CU Coding Limited
• Original Assignee: Alcatel-Lucent SA (Nokia Corporation)
• Inventors: Carlbom, Ingrid Birgitta, Jean, Yves D., Opalach, Agata, Pingali, Gopal S.
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): FINDLEY, CHRISTOPHER G
• Application Number:
  US10/403,443
• Publication Number:
  US 20040194129A1
• Time in Patent Office: 3,921 Days
• Field of Search: 348/143, 348/152, 348/154, 348/169, 348/208.14
• US Class Current: 348/143
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Techniques for intelligent and automatic control of sensors for capturing data associated with real time events. Preferably, the sensors are associated with a multimedia database system. For example, a technique for controlling one or more sensors used to capture data associated with an event comprises the following steps/operations. First, sensor data captured in accordance with the event is processed. Then, the one or more sensors are automatically controlled based on information pertaining to the continual activity of at least one of one or more objects and one or more persons associated with the event in real time obtained using at least a portion of the processed data captured in accordance with the event.

Claims

1. A method of controlling one or more sensors used to capture data associated with an event, the method comprising the steps of:

processing sensor data captured in accordance with the event; and

automatically controlling the one or more sensors based on information pertaining to a continual activity of at least one of one or more objects and one or more persons associated with the event in real time obtained using at least a portion of the processed data captured in accordance with the event;

wherein the step of automatically controlling the one or more sensors further comprises obtaining one or more user preferences, wherein the one or more user preferences comprise at least a view preference and at least one of an object or person preference and an object or person behavior preference,

wherein a reasoning subsystem is used to identify a behavior specified by the object or person behavior preference when the one or more user preferences comprise the object or person behavior preference, whereby the reasoning subsystem is adapted to conclude that a first detected action of at least one of the object and the person followed by at least a second detected action of at least one of the object and the person comprise the specified behavior when the first detected action or the second detected action alone would not comprise the specified behavior but when detected together as part of a continual activity comprise the specified behavior,

wherein the view preference comprises a user-specified priority of one or more view types and one or more sensor types, and

wherein the step of automatically controlling the one or more sensors further comprises;

selecting a view type and sensor type based on the user-defined priority of view types; and

responsive to determining that at least one of the selected view type and the preferred sensor type is obstructed, selecting another view type or preferred sensor type.

2. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises utilizing the one or more user preferences in conjunction with at least a portion of the activity information to generate one or more signals for controlling the one or more sensors.

3. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises identifying a two dimensional display screen coordinate corresponding to the object or person specified by the object or person preference.

4. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises specifying an identifier corresponding to the object or person specified by the object or person preference.

5. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises analyzing a spatial behavior corresponding to the object or person specified by the behavior preference.

6. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises analyzing a spatial behavior relating to the surrounding three dimensional environment for the object or person specified by the behavior preference.

7. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises analyzing a spatial behavior relating to one or more surrounding objects in the environment for the object or person specified by the behavior preference.

8. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises analyzing a temporal behavior corresponding to the object or person specified by the behavior preference.

9. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises specifying a temporal behavior relating to historical data for the object or person specified by the behavior preference.

10. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises specifying a temporal behavior relating to at least one of the speed, acceleration, and direction of the object or person specified by the behavior preference.

11. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises specifying a temporal behavior relating to the time of actions of the object or person specified by the behavior preference.

12. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises specifying a temporal behavior relating to prediction of location of the object or person specified by the behavior preference.

13. The method of claim 1, wherein the step of automatically controlling the one or more sensors further comprises obtaining a motion trajectory corresponding to the object or person specified by the object or person preference.

14. The method of claim 13, wherein the step of automatically controlling the one or more sensors further comprises finding one or more objects or persons in a neighborhood of the object or person specified by the object or person preference.

15. The method of claim 14, wherein the step of automatically controlling the one or more sensors further comprises predicting the next locations of the object or person specified by the object or person preference and of the one or more neighboring objects or persons, using respective motion trajectories.

16. The method of claim 15, wherein the step of automatically controlling the one or more sensors further comprises selecting at least one sensor for capturing data associated with the object or person specified by the object or person preference at its predicted next location, based on the view preference and at least a portion of the processed, captured data.

17. The method of claim 16, wherein the step of automatically controlling the one or more sensors further comprises determining whether any of the neighboring objects or persons block the view of the at least one selected sensor.

18. The method of claim 17, wherein the step of automatically controlling the one or more sensors further comprises directing the at least one selected sensor to the predicted next location of the object or person specified by the object or person preference, when not blocked or only partially blocked by any of the neighboring objects or persons.

19. The method of claim 18, wherein the step of automatically controlling the one or more sensors further comprises determining the actual position of the object or person specified by the object or person preference.

20. The method of claim 1, wherein the one or more sensors are associated with a multimedia database system.

21. Apparatus for controlling one or more sensors used to capture data associated with an event, the apparatus comprising:

a memory; and

at least one processor coupled to the memory and operative to;

(i) obtain processed sensor data captured in accordance with the event; and

(ii) automatically control the one or more sensors based on information pertaining to a continual activity of at least one of one or more objects and one or more persons associated with the event in real time obtained using at least a portion of the processed data captured in accordance with the event;

wherein the operation of automatically controlling the one or more sensors further comprises obtaining one or more user preferences, wherein the one or more user preferences comprise at least a view preference and at least one of an object or person preference and an object or person behavior preference,

wherein a reasoning subsystem is used to identify a behavior specified by the object or person behavior preference when the one or more user preferences comprise the object or person behavior preference, whereby the reasoning subsystem is adapted to conclude that a first detected action of at least one of the object and the person followed by at least a second detected action of at least one of the object and the person comprise the specified behavior when the first detected action or the second detected action alone would not comprise the specified behavior but when detected together as part of a continual activity comprise the specified behavior,

wherein the view preference comprises a user-specified priority of one or more view types and one or more sensor types, and

wherein the operation of automatically controlling the one or more sensors further comprises;

selecting a view type and sensor type based on the user-defined priority of view types; and

responsive to determining that at least one of the selected view type and the preferred sensor type is obstructed, selecting another view type or preferred sensor type.

22. The apparatus of claim 21, wherein the operation of automatically controlling the one or more sensors further comprises utilizing the one or more user preferences in conjunction with at least a portion of the activity information to generate one or more signals for controlling the one or more sensors.

23. The apparatus of claim 21, wherein the operation of automatically controlling the one or more sensors further comprises identifying a two dimensional display screen coordinate corresponding to the object or person specified by the object or person preference.

24. The apparatus of claim 21, wherein the operation of automatically controlling the one or more sensors further comprises specifying an identifier corresponding to the object or person specified by the object or person preference.

25. The apparatus of claim 21, wherein the operation of automatically controlling the one or more sensors further comprises obtaining a motion trajectory corresponding to the object or person specified by the object or person preference.

26. The apparatus of claim 21, wherein the one or more sensors are associated with a multimedia database system.

27. An article of manufacture for controlling one or more sensors used to capture data associated with an event, comprising a machine readable medium containing one or more programs which when executed implement the steps of:

processing sensor data captured in accordance with the event; and

automatically controlling the one or more sensors based on information pertaining to a continual activity of at least one of one or more objects and one or more persons associated with the event in real time obtained using at least a portion of the processed data captured in accordance with the event;

wherein the operation of automatically controlling the one or more sensors further comprises obtaining one or more user preferences, wherein the one or more user preferences comprise at least a view preference and at least one of an object or person preference and an object or person behavior preference,

wherein a reasoning subsystem is used to identify a behavior specified by the object or person behavior preference when the one or more user preferences comprise the object or person behavior preference, whereby the reasoning subsystem is adapted to conclude that a first detected action of at least one of the object and the person followed by at least a second detected action of at least one of the object and the person comprise the specified behavior when the first detected action or the second detected action alone would not comprise the specified behavior but when detected together as part of a continual activity comprise the specified behavior,

wherein the view preference comprises a user-specified priority of one or more view types and one or more sensor types, and

wherein automatically controlling the one or more sensors further comprises;

selecting a view type and sensor type based on the user-defined priority of view types; and

responsive to determining that at least one of the selected view type and the preferred sensor type is obstructed, selecting another view type or preferred sensor type.