VISION BASED REAL-TIME OBJECT TRACKING SYSTEM FOR ROBOTIC GIMBAL CONTROL

• US 20160171330A1
• Filed: 12/14/2015
• Published: 06/16/2016
• Est. Priority Date: 12/15/2014
• Status: Active Grant

• Current Assignee: Reflex Robotics, Inc.
• Original Assignee: Reflex Robotics, Inc.
• Inventors: Mentese, Enes, Babukr, Abdo, Kim, Seongdo
• Granted Patent: US 10,095,942 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: H04N 23/685 performed by mechanical compensation

Abstract

Using various embodiments, methods, systems, and apparatuses for controlling a camera pivoting device (e.g., mechanical gimbal) are described. In one embodiment, the system comprises a main computing device, a gimbal stabilizer controller, and a computer vision camera, and/or a user camera. The system is able to track a target object using the computer vision camera even while the target object is moving, the base of the pivoting device is moving (e.g., when a user controlling the camera moves), or a combination of thereof. The camera pivoting device of the embodiments disclosed herein can be mounted on to any number of devices/objects that can provide mobility and/or transportation.

Claims

1. A real-time interactive display system for dynamic object selection and tracking, the system comprising:

a pivoting device, wherein the pivoting device includes a roll motor, pitch motor, and yaw motor, wherein actuating the roll motor, pitch motor, and yaw motor causes the pivoting device to be operable with at least three degrees of freedom;

a computer vision camera, wherein the computer vision camera is mounted on the pivoting device;

a stabilizing controller device coupled to the pivoting device, wherein the stabilizing controller device can actuate the roll, pitch, and yaw motors of the pivoting device; and

a computing device connected to the stabilizing controller device, wherein the computing device comprises a processing system having one or more cores, the processing system configured to;

periodically receive a frame from the computer vision camera;

receive information about a Region Of Interest (ROI) within the frame, wherein the ROI includes a target subject that needs to be tracked in real-time;

compute a location and size of the target subject based on the information; and

transmit instructions to the stabilizing controller device to control the pitch and yaw motors of the pivoting device to adjust the pivoting device such that the subject target is maintained within the frame of the computer vision camera.

2. The system of claim 1, wherein the stabilizing controller device comprises a microcontroller and brushless motor drivers.

3. The system of claim 2, wherein proportional-integral-derivative (PID) loops are executed on the microcontroller of the stabilizing controller, and wherein the PID loops control the roll, pitch, and yaw motors of the pivoting device in a smooth and desirable manner.

4. The system of claim 1, wherein the pivoting device is a mechanical gimbal.

5. The system of claim 1, wherein the pivoting device includes an inertial measurement unit (IMU), wherein the IMU can provide feedback to the stabilizing controller device using which the pivoting device is maintained in a stabilized and leveled manner with respect to the target subject, and wherein the stabilizing controller device directly controls the roll motor based on feedback received from the IMU.

6. The system of claim 1, wherein the information about the ROI is received using at least one of a vision based tracking algorithm or a infrared beacon based tracking algorithm.

7. The system of claim 1, wherein the ROI is received from a user operating a machine interface, the machine interface connected to the computing device.

8. A method to dynamically select and track a subject interactively, the method comprising:

periodically receiving, by a computing device, a frame from a computer vision camera;

receiving information about a Region Of Interest (ROI) within the frame, wherein the ROI includes a target subject that needs to be tracked in real-time;

computing a location and size of the target subject based on the information; and

transmitting instructions to a stabilizing controller device to control at least one motor of a pivoting device to adjust the pivoting device such that the subject target is maintained within the frame of the computer vision camera in real-time;

wherein the pivoting device includes a roll motor, pitch motor, and yaw motor, and wherein actuating the roll motor, pitch motor, and yaw motor causes the pivoting device to be operable with at least three degrees of freedom, and wherein the computer vision camera is mounted on the pivoting device, and wherein the stabilizing controller device is coupled to the pivoting device, and wherein the stabilizing controller device can actuate the roll, pitch, and yaw motors of the pivoting device, and wherein the computing device is connected to the stabilizing controller device.

9. The method of claim 8, wherein the stabilizing controller device comprises a microcontroller and brushless motor drivers.

10. The method of claim 9, wherein proportional-integral-derivative (PID) loops are executed on the microcontroller of the stabilizing controller, and wherein the PID loops control the roll, pitch, and yaw motors of the pivoting device in a smooth and desirable manner.

11. The method of claim 8, wherein the pivoting device is a mechanical gimbal.

12. The method of claim 8, wherein the pivoting device includes an inertial measurement unit (IMU), wherein the IMU can provide feedback to the stabilizing controller device using which the pivoting device is maintained in a stabilized and leveled manner with respect to the target subject, and wherein the stabilizing controller device directly controls the roll motor based on feedback received from the IMU.

13. The method of claim 8, wherein the information about the ROI is received using at least one of a vision based tracking algorithm or a infrared beacon based tracking algorithm.

14. The method of claim 8, wherein the ROI is received from a user operating a machine interface, the machine interface connected to the computing device.

15. A non-transitory computer readable medium comprising instructions which when executed by a processing system executes a method to dynamically select and track a subject interactively, the method comprising:

periodically receiving, by a computing device, a frame from a computer vision camera;

receiving information about a Region Of Interest (ROI) within the frame, wherein the ROI includes a target subject that needs to be tracked in real-time;

computing a location and size of the target subject based on the information; and

transmitting instructions to a stabilizing controller device to control at least one motor of a pivoting device to adjust the pivoting device such that the subject target is maintained within the frame of the computer vision camera in real-time;

wherein the pivoting device includes a roll motor, pitch motor, and yaw motor, and wherein actuating the roll motor, pitch motor, and yaw motor causes the pivoting device to be operable with at least three degrees of freedom, and wherein the computer vision camera is mounted on the pivoting device, and wherein the stabilizing controller device is coupled to the pivoting device, and wherein the stabilizing controller device can actuate the roll, pitch, and yaw motors of the pivoting device, and wherein the computing device is connected to the stabilizing controller device.

16. The non-transitory computer readable medium of claim 15, wherein the stabilizing controller device comprises a microcontroller and brushless motor drivers.

17. The non-transitory computer readable medium of claim 16, wherein proportional-integral-derivative (PID) loops are executed on the microcontroller of the stabilizing controller, and wherein the PID loops control the roll, pitch, and yaw motors of the pivoting device in a smooth and desirable manner.

18. The non-transitory computer readable medium of claim 15, wherein the pivoting device is a mechanical gimbal.

19. The non-transitory computer readable medium of claim 15, wherein the pivoting device includes an inertial measurement unit (IMU), wherein the IMU can provide feedback to the stabilizing controller device using which the pivoting device is maintained in a stabilized and leveled manner with respect to the target subject, and wherein the stabilizing controller device directly controls the roll motor based on feedback received from the IMU.

20. The non-transitory computer readable medium of claim 15, wherein the information about the ROI is received using at least one of a vision based tracking algorithm or a infrared beacon based tracking algorithm.