Method and apparatus for tracking an object of interest using a camera associated with a hand-held processing device

• US 7,742,073 B1
• Filed: 11/01/2000
• Issued: 06/22/2010
• Est. Priority Date: 11/01/2000
• Status: Expired due to Fees

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Brodsky, Tomas, Cohen-Solal, Eric
• Primary Examiner(s): Jerabek; Kelly L
• Application Number:
  US09/703,419
• Time in Patent Office: 3,520 Days
• Field of Search: 3482201-2221, 348/231.99, 348/333.01, 348/376, 348/350, 348169-172, 348/208.99, 3482081-2087, 348 1401- 1416, 358/906, 358/909, 382/181, 382190-195, 382/203, 382/209
• US Class Current: 348/169
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: H04N 2007/145 Handheld terminals
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

A hand-held processing device, such as a mobile telephone, a personal digital assistant (PDA), or a portable computer, includes a physically or electronically adjustable camera and is operative to track a specified object of interest. Relative movement between the hand-held device and the object of interest is detected using an orientation determination device such as a gyroscope, an image-based tracking operation, or a hybrid combination of orientation determination and image-based tracking. A setting of the camera is then adjusted so as to maintain a desired framing of the object of interest within an image generated by the camera.

Claims

1. A method for automatically framing and tracking an object of interest using a video camera integrated into hand-held processing devices including PDAs, mobile telephones, palmtops, and portable computers to insure stability of the image content as a user manipulates the device, the method comprising the steps of:

providing said video camera with a wide field of view;

continuously detecting relative movement associated with the hand-held device that tracks the object of interest within a displayed image generated by said camera, the relative movement depending on the tracking of the object of interest; and

continuously electronically adjusting the camera, without use of a motor, in response to the detected relative movement, so as to maintain a desired framing and tracking of the object of interest within an image and/or successive images, as long as the image or images remain in the field of view generated by the camera for selectively providing either one of a still picture of the object or video image of the object, respectively, for providing a stable image in the presence of movement of a user'"'"'s hand holding said device;

wherein the framing of the object involves creating a model of the object; and

wherein the model is adjusted based on a history of prior obtained plurality of calibration parameters used to represent one or more matrices used for the framing and tracking of the object.

2. The method of claim 1 wherein the camera is physically adjustable by a user.

3. The method of claim 1 wherein the camera has one or more of solely electronically adjustable pan setting, an adjustable tilt setting, and an adjustable zoom setting, performed without use of a motor.

4. The method of claim 1, wherein said step of continuously electronically adjusting the camera is based at least in part on an output of an orientation determination device integrated into or otherwise associated with the hand-held device, for detecting relative movement between said device and an object of interest caused by movement of a user'"'"'s hand.

5. The method of claim 4 wherein the orientation determination device comprises one or more gyroscopes integrated into the hand-held device.

6. The method of claim 1, wherein said step of continuously electronically adjusting the camera is based at least in part on an output of an image processing operation applied to an image generated by the camera.

7. The method of claim 1, wherein said step of continuously electronically adjusting the camera is based at least in part on a hybrid combination of an orientation determination operation and an image processing operation.

8. The method of claim 1, wherein the detecting of the relative movement between the hand-held device and the object of interest occurs in response to initialization by the user.

9. The method of claim 8, wherein the initialization is a manual initialization.

10. The method of claim 8, wherein the initialization is a voice-activated initialization.

11. The method of claim 1, wherein the camera is electronically configured to include a plurality of calibration parameters represented as an upper triangular matrix.

12. The method of claim 11, wherein the plurality of calibration parameters of the matrix are adjusted at a time of manufacturing and cannot be subsequently modified by the user of the camera.

13. The method of claim 11, wherein the plurality of calibration parameters of the matrix are adjusted based on one or more actions performed by the user of the camera at a time of use of the camera.

14. The method of claim 13, wherein the one or more actions include zoom setting actions.

15. The method of claim 1, wherein the camera is electronically configured to include a plurality of calibration parameters represented by a plurality of matrices.

16. The method of claim 15, wherein a select number of the plurality of matrices are manipulated in conjunction with a plurality of different zoom settings in order to frame the object.

17. The method of claim 1, wherein the relative movement between the hand-held device and the object of interest includes processing a single movement.

18. The method of claim 1, wherein the relative movement between the hand-held device and the object of interest includes processing a plurality of movements.

19. The method of claim 1, wherein the camera is electronically configured to include a plurality of calibration parameters represented as a homography matrix, where the homography matrix is composed of a rotation matrix and a calibration matrix.

20. The method of claim 1, wherein the model is compared against a predetermined model.

21. The method of claim 1, wherein the model continuously learns of a select number of optimal calibration parameters derived from the history of the prior obtained plurality of calibration parameters for using the select number of the optimal calibration parameters in the framing and tracking of the object.

22. An apparatus for automatically framing and tracking an object of interest, the apparatus comprising:

a hand-held processing device including PDA'"'"'s, mobile telephones, palmtops, and portable computers, having at least one video camera integrated therein, the hand-held device further comprising;

a processor operative to continuously monitor the detection of relative movement associated with the hand-held device that tracks the object of interest, due to movement of a user'"'"'s hand holding said device, the relative movement depending on the tracking of the object of interest,

said processor being responsive to the detected relative movement for continuously solely electronically adjusting, without use of a motor, at least one setting of the camera so as to continuously maintain a desired framing of the object of interest within an image generated by the camera as a user manipulates the device, for providing a stable image;

wherein the framing of the object involves creating a model of the object; and

wherein the model is adjusted based on a history of prior obtained plurality of calibration parameters used to represent one or more matrices used for the framing and tracking of the object.

23. The apparatus according to claim 22, wherein the model continuously learns of a select number of optimal calibration parameters derived from the history of the prior obtained plurality of calibration parameters for using the select number of the optimal calibration parameters in the framing and tracking of the object.

24. An article of manufacture comprising a storage medium for storing one or more programs for tracking an object of interest using at least one video camera having integrated into a hand-held processing device, including PDA'"'"'s, mobile telephones, palmtops, and portable computers, wherein the one or more programs when executed by a processor implement the steps of:

detecting relative movement associated with the hand-held device that tracks the object of interest, the relative movement depending on the tracking of the object of interest; and

adjusting solely electronically, without use of a motor, at least one setting of the camera, in response to the detected relative movement, so as to maintain a desired framing of the object of interest within an image generated by the camera, for providing a stable image;

wherein the framing of the object involves creating a model of the object; and

wherein the model is adjusted based on a history of prior obtained plurality of calibration parameters used to represent one or more matrices used for the framing and tracking of the object.

25. The article of manufacture according to claim 24, wherein the model continuously learns of a select number of optimal calibration parameters derived from the history of the prior obtained plurality of calibration parameters for using the select number of the optimal calibration parameters in the framing and tracking of the object.