Automatic photography

• US 20030025798A1
• Filed: 07/30/2002
• Published: 02/06/2003
• Est. Priority Date: 07/31/2001
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Grosvenor, David Arthur, Hunter, Andrew Arthur, Cheatle, Stephen Philip
• Granted Patent: US 7,030,909 B2
• Time in Patent Office: Days
• US Class Current: 348/207.99
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules

Abstract

Automated camera apparatus comprises a head mounted electronic camera 1 for providing a video signal 3 representative of an image 2 viewed by the wearer, video signal processor 51 arranged to receive said video signal for judging the degree of interest shown by the wearer in a feature in the scene and for providing a related saliency signal 12, the means 51 including image analyser 5 for identifying the presence of features and their locations relative to a reference position (e.g. the centre) in the frame within each of a series of frames in a time window of the video signal, and track analysis means 7 coupled to the output of said image analyser for analysing the movement of at least one said feature over said series of frames to provide the saliency signal. As shown saliency signals 8 are provided for each or selected features identified in the scene and are thresholded in circuit 9 to provide binary signals 10, ORed together at 11 to provide a two-level or binary saliency signal 12 which is positive for salient features and zero otherwise. The video signal is delayed 4, and the end (only) of a positive saliency signal 12 is delayed 13 prior to recording 15 of both signals. The signal 12 may be derived in real time or from a video recording. Account may be taken of head and/or eye movement.

Claims

1. Automated camera apparatus comprising an electronic camera, the camera comprising a photoelectric sensor optically coupled to imaging optics for providing a video signal representative of sequential frames of an image viewed by the camera, and at least the imaging optics being adapted for securing on the head of a wearer with its axis generally parallel to the normal forward line of sight of the wearer, the apparatus further comprising a video signal processor arranged to receive said video signal for judging the degree of interest shown by the wearer in a feature in the scene and for providing a related saliency signal, the video signal processor including a video signal analyser for identifying the presence of features and their locations in the frame within each of a plurality of frames in a time window of the video signal, and a movement analyser coupled to the output of said video signal analyser for analysing the movement of at least one said feature over said plurality of frames to provide said saliency signal.

2. Apparatus according to claim 1 wherein said plurality of frames is at least three frames.

3. Apparatus according to claim 1 wherein said video signal analyser comprises an image analyser for identifying a said feature from a single frame.

4. Apparatus according to claim 3 wherein said plurality of frames is at least three frames.

5. Apparatus according to claim 4 wherein said movement analyser analyses the two dimensional track of said feature in the frame.

6. Apparatus according to claim 1 wherein the field of view of the camera is at least substantially equal to that of the wearer.

7. Apparatus according to claim 1 and including a head movement sensor for providing a measure of head movement of the wearer.

8. Apparatus according to claim 7 wherein said head movement sensor comprises at least one gyro or accelerometer.

9. Apparatus according to claim 7 wherein said head movement sensor is arranged to determine and analyse the optic flow in said series of frames.

10. Apparatus according to claim 7 wherein the video signal analyser identifies the position of said feature(s) relative to a reference position in the frame and the output of said head movement sensor is coupled to the camera or the video signal processor effectively to adjust said reference position.

11. Apparatus according to claim 7 wherein the output of said head movement sensor is coupled to the video signal processor to control the operation of the movement analyser.

12. Apparatus according to claim 1 further comprising an eye direction detector for determining the deflection of the eyes in the head relative to a standard position.

13. Apparatus according to claim 12 wherein the video signal analyser identifies the position of said feature(s) relative to a reference position in the frame and the output of said eye direction detector is coupled to the camera or the video signal processor effectively to adjust said reference position.

14. Apparatus according to claim 1 wherein the field of view of the camera greater than the field of view of the wearer.

15. Apparatus according to claim 14 and comprising a scan controller for controlling the scan of the camera.

16. Apparatus according to claim 15 and including a head movement sensor for providing a measure of head movement of the wearer, wherein the video signal analyser identifies the position of said feature(s) relative to a reference position in the frame, the output of said head movement sensor is coupled to the camera or the video signal processor effectively to adjust said reference position, and the scan controller is provided with a scan control input coupled to and responsive to the head movement sensor to perform said effective adjustment of the reference position.

17. Apparatus according to claim 12 and comprising a scan controller for controlling the scan of the camera, wherein the scan controller is provided with a scan control input coupled to and responsive to said eye direction means to perform said effective adjustment of the reference position.

18. Apparatus according to claim 17 wherein said effective adjustment is performed by adjusting the scanned area of the camera so that the reference position is brought into substantial coincidence with the line of sight of the wearer.

19. Apparatus according to claim 15 wherein the scan controller is provided with an adjusting input for ensuring during setting up that the reference position in the video frames substantially coincides with the wearer'"'"'s normal forward horizontal line of sight.

20. Apparatus according to claim 1 wherein the image analyser is adapted to identify features by optical segmentation and/or image recognition.

21. Apparatus according to claim 1 wherein the video signal processor is arranged to provide high or positive measure of saliency when a feature remains at or close to a single location in the series of frames.

22. Apparatus according to claim 1 wherein the video signal processor is arranged to provide a high or positive measure of saliency when a feature moves relative to the reference location in a manner resembling a series of repeated glances.

23. Apparatus according to claim 1 wherein the video signal processor is arranged to function in real time.

24. Apparatus according to claim 23 and including a gate coupled to selectively gate the video signal for display or recordal in response to the saliency signal.

25. Apparatus according to claim 23 and including a signal store for recording the saliency signal together with the video signal.

26. Apparatus according to claim 1 and including means for recording the video signal together with any other signal necessary for determining the saliency signal, so that the saliency signal can be produced by the video signal processor at a later time.

27. Apparatus according to claim 1 and including a video signal delay for delaying the video signal relative to the saliency signal.

28. Apparatus according to claim 1, wherein the track analysis means includes a threshold arrangement for providing a discretely varying saliency signal.

29. Apparatus according to claim 28 and including a time adjuster for altering the time of the transitions of said discretely varying threshold signal.

30. Automated camera apparatus comprising an electronic camera, the camera comprising a photoelectric sensor optically coupled to imaging optics for providing a video signal representative of sequential frames of an image viewed by the camera, at least the imaging optics being adapted for securing on the head of a wearer with its axis generally parallel to the normal forward line of sight of the wearer, the apparatus further comprising a video signal processor arranged to receive said video signal for judging the degree of interest shown by the wearer in a feature in the scene and for providing a related saliency signal, the video signal processor including a video signal analyser for identifying the presence of at least one feature which remains within a predetermined (e.g. central) sub-region of the frame within each of a plurality of frames in a time window of the video signal, head movement sensor for sensing movement of the head of the wearer above a threshold level, and a movement analyser coupled to the outputs of said video signal analyser and the head movement sensor for providing a said saliency signal as a function of said outputs.

32. Apparatus according to claim 30 wherein the movement analyser is also responsive to the output of the eye direction measuring means when producing the saliency signal.

33. Apparatus according to claim 30 and further including a head movement sensor for sensing movement of the head of the wearer. the movement responsive means being coupled to the output thereof for control of the production of the saliency signal.

35. Apparatus according to claim 32 and further including a head movement sensor for sensing movement of the head of the wearer, the movement responsive means being coupled to the output thereof for control of the production of the saliency signal.

31. Automated camera apparatus comprising an electronic camera, the camera comprising a photoelectric sensor optically coupled to imaging optics for providing a video signal representative of sequential frames of an image viewed by the camera, at least the imaging optics being adapted for securing on the head of a wearer with its axis generally parallel to the normal forward line of sight of the wearer, the apparatus further comprising a video signal processor arranged to receive said video signal for judging the degree of interest shown by the wearer in a feature in the scene and for providing a related saliency signal, the video signal processor including a video signal analyser for identifying the presence of at least one feature which remains within a specified sub-region of the frame within each of a plurality of frames in a time window of the video signal, eye direction measuring means for determining the direction of the eyes of the wearer relative to the said normal forward line of sight, the video signal analyser being coupled to the output of the eye direction measuring means for adjusting the position of the said sub-region in response thereto prior to identifying the presence of said feature(s) therein, and a movement analyser coupled to the outputs of said video signal analyser for providing a said saliency signal as a function of said outputs.

34. Apparatus according to claim 31 and further including a head movement sensor for sensing movement of the head of the wearer, the movement responsive means being coupled to the output thereof for control of the production of the saliency signal.

36. A method of deriving a saliency signal which indicates the saliency of a feature to a wearer of an camera providing a video signal wherein at least the imaging optics of the camera are mounted on the wearer'"'"'s head, the method comprising the steps of feature analysis in which in a plurality of frames the video signal is analysed to identify at least one feature and its position within the frame area, and pattern analysis in which the pattern or manner of movement of the feature(s) within the frames over said plurality is analysed to derive a said saliency signal corresponding to said at least one feature.

37. A method according to claim 36 wherein said plurality of frames is at least three frames.

38. A method according to claim 36 wherein each single frame of said plurality is analysed for identifying a said feature.

39. A method according to claim 38 wherein said plurality of frames is at least three frames.

40. A method according to claim 36 wherein said pattern analysis takes account of the two dimensional track of said feature in the frame.

41. A method according to claim 36 wherein the position within the frame area is measured with respect to a reference position which corresponds substantially with the wearer'"'"'s normal horizontal line of sight.

42. A method according to claim 36 and further including the step of obtaining a head movement signal indicative of movement of the head of the camera wearer.

43. A method according to claim 42 wherein said head movement signal is used to modify said pattern analysis step.

44. A method according to claim 42 wherein said head movement signal is used to modify the scanning action of the camera.

45. A method according to claim 36 and including the further step of obtaining an eye movement signal indicative of eye movement of the wearer relative to the head.

46. A method according to claim 45 wherein said eye movement signal is used to modify said pattern analysis step.

47. A method according to claim 45 wherein said eye movement signal is used to modify the scanning action of the camera.

48. A method according to claim 36 wherein said pattern analysis step includes classifying the movement of the feature according to a set of predetermined reference patterns and obtaining said saliency signal in a manner related to the classification of said pattern.

49. A method according to claim 36 wherein said saliency signal is derived in real time.

50. A method according to claim 36 wherein said video signal is recorded prior to the obtaining of said saliency signal.