Methods and devices for measuring object motion using camera images

• US 10,380,745 B2
• Filed: 02/28/2017
• Issued: 08/13/2019
• Est. Priority Date: 09/01/2016
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Buyukozturk, Oral, Freeman, William T., Durand, Frederic, Davis, Myers Abraham, Wadhwa, Neal, Chen, Justin G.
• Primary Examiner(s): Hance, Robert J
• Application Number:
  US15/445,499
• Publication Number:
  US 20180061063A1
• Time in Patent Office: 896 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters

Abstract

A method and corresponding apparatus for measuring object motion using camera images may include measuring a global optical flow field of a scene. The scene may include target and reference objects captured in an image sequence. Motion of a camera used to capture the image sequence may be determined relative to the scene by measuring an apparent, sub-pixel motion of the reference object with respect to an imaging plane of the camera. Motion of the target object corrected for the camera motion may be calculated based on the optical flow field of the scene and on the apparent, sub-pixel motion of the reference object with respect to the imaging plane of the camera. Embodiments may enable measuring vibration of structures and objects from long distance in relatively uncontrolled settings, with or without accelerometers, with high signal-to-noise ratios.

Claims

1. A method of measuring motion of an object using camera images, the method comprising:

measuring a global optical flow field of a scene including a target object and a reference object captured as representations in an image sequence of the scene;

determining motion, relative to the scene, of a camera used to capture the image sequence by measuring an apparent, sub-pixel motion of the reference object with respect to an imaging plane of the camera, wherein determining the motion of the camera includes measuring the apparent, sub-pixel motion of the reference object within a frequency range of motion on a same order of magnitude as a frequency range of motion of the target object; and

calculating a corrected motion of the target object, corrected for the camera motion, based on the optical flow field of the scene and on the apparent, sub-pixel motion of the reference object with respect to the imaging plane of the camera.

2. The method of claim 1, wherein determining the motion of the camera further includes using measurements from an external sensor to calculate the motion, the external sensor including at least one of an accelerometer, gyroscope, magnetometer, inertial measurement unit (IMU), global positioning system (GPS) unit, or velocity meter.

3. The method of claim 2, wherein using measurements from the external sensor includes using the external sensor attached to one of the target object, reference object, and camera.

4. The method of claim 2, further including using a Kalman filter or other sensor fusion technique to obtain a best estimate external sensor measurement used to determine motion of the camera.

5. The method of claim 1, wherein determining motion of the camera includes measuring the apparent, sub-pixel motion of the reference object in one or two linear axes contained within the imaging plane of the camera.

6. The method of claim 1, wherein determining the motion of the camera includes measuring apparent rotation of the reference object within the imaging plane of the camera.

7. The method of claim 1, wherein measuring the global optical flow field of the scene includes using at least a portion of the scene with the reference object being at least one of a foreground object or a background object.

8. The method of claim 1, wherein the target object includes at least one of a seismic structure, a hydraulic fracturing environment structure, a water or oil or gas reservoir, or a volcano, the method further including using the corrected motion of the target object for seismic oil exploration, monitoring a condition of the reservoir, or monitoring geological features of the volcano.

9. The method of claim 1, wherein the target object includes at least one of a bridge, oil rig, crane, or machine.

10. The method of claim 1, wherein measuring the optical flow field of the scene includes using motion magnification.

11. The method of claim 1, wherein measuring the optical flow field of the scene includes extracting pixel-wise Eulerian motion signals of an object in the scene from an undercomplete representation of frames within the image sequence and downselecting pixel-wise Eulerian motion signals to produce a representative set of Eulerian motion signals of the object.

12. The method of claim 11, wherein downselecting the signals includes choosing signals on a basis of local contrast in the frames within the image sequence.

13. The method of claim 1, wherein measuring the optical flow field of the scene includes observing target and reference objects situated at least 30 meters (m) from the camera used to capture the image sequence.

14. The method of claim 1, wherein the measuring, determining, and calculating occur at a network server and operate on the image sequence received via a network path.

15. The method of claim 1, further including uploading the image sequence to a remote server or downloading a representation of the corrected motion of the target object from the remote server.

16. The method of claim 1, wherein the camera is part of a mobile device and wherein the measuring, determining, and calculating occur in the mobile device.

17. The method of claim 16, wherein the mobile device includes an external sensor including at least one of an accelerometer, gyroscope, magnetometer, IMU, global positioning system (GPS) unit, or velocity meter.

18. The method of claim 1, wherein measuring the apparent, sub-pixel motion of the reference object includes measuring sub-pixel motion in a range of 0.03 to 0.3 pixels.

19. The method of claim 1, wherein calculating the corrected motion of the target object includes calculating a corrected motion in a range of 0.03 to 0.3 pixels or in a range of 0.003 to 0.03 pixels.

20. The method of claim 1, wherein measuring the apparent, sub-pixel motion of the reference object includes measuring sub-pixel motion in a range of 0.003 to 0.03 pixels.

21. A device for measuring the motion of an object using camera images, the device comprising:

memory configured to store an image sequence of a scene including a target object and a reference object captured as representations in the image sequence; and

a processor configured to;

measure a global optical flow field of the scene from the image sequence of the scene;

determine motion, relative to the scene, of a camera used to capture the image sequence by measuring an apparent, sub-pixel motion of the reference object with respect to an imaging plane of the camera by measuring the apparent, sub-pixel motion of the reference object within a frequency range of motion on a same order of magnitude as a frequency range of motion of the target object; and

calculate a corrected motion of the target object, corrected for the camera motion, based on the optical flow field of the scene and on the apparent, sub-pixel motion of the reference object with respect to the imaging plane of the camera.

22. The device of claim 21, wherein the processor is further configured to determine the motion of the camera using measurements from an external sensor to calculate the motion, the external sensor including at least one of an accelerometer, gyroscope, magnetometer, inertial measurement unit (IMU), global positioning system (GPS) unit, or velocity meter.

23. The device of claim 22, wherein the external sensor is attached to one of the target object, reference object, or camera.

24. The device of claim 22, wherein the processor is further configured to implement Kalman filtering or another sensor fusion technique to obtain a best estimate external sensor measurement used to determine the motion of the camera.

25. The device of claim 21, wherein the processor is further configured to determine the motion of the camera by measuring the apparent, sub-pixel motion of the reference object in one or two linear axes contained within the imaging plane of the camera.

26. The device of claim 21, wherein the processor is further configured to determine the motion of the camera by measuring apparent motion of the reference object within the imaging plane of the camera.

27. The device of claim 21, wherein the processor is further configured to measure the global optical flow field of the scene by using at least a portion of the scene with the reference object being at least one of a foreground object or a background object.

28. The device of claim 21, wherein the target object includes at least one of a seismic structure, a hydraulic fracturing environment structure, a water or oil or gas reservoir, or a volcano, and wherein the processor is further configured to use the corrected motion of the target object for seismic oil exploration, monitoring a condition of the reservoir, or monitoring geological features of the volcano.

29. The device of claim 21, wherein the target object includes at least one of a bridge, oil rig, crane, or machine.

30. The device of claim 21, wherein the processor is further configured to measure the optical flow field using motion magnification.

31. The device of claim 21, wherein the processor is further configured to measure the global optical flow field of the scene by extracting pixel-wise Eulerian motion signals of an object in the scene from an undercomplete representation of frames within the image sequence and to downselect pixel-wise Eulerian motion signals to produce a representative set of Eulerian motion signals of the object.

32. The device of claim 31, wherein the processor is further configured to downselect the signals by choosing signals on a basis of local contrast in the frames within the image sequence.

33. The device of claim 21, wherein the image sequence of the scene includes the target and reference objects situated at least 30 meters (m) from the camera used to capture the image sequence.

34. The device of claim 21, wherein the processor is part of a network server, and wherein the processor is configured to operate on, or the memory is configured to receive, the image sequence via a network path.

35. The device of claim 21, wherein the memory is further configured to receive the image sequence from a remote server, or wherein the device further includes a communications interface configured to send a representation of the corrected motion of the target object to a remote server.

36. The device of claim 21, wherein the camera, memory, and processor are part of a mobile device.

37. The device of claim 36, wherein the mobile device includes an external sensor including at least one of an accelerometer, gyroscope, magnetometer, IMU, global positioning system (GPS) unit, or velocity meter.

38. The device of claim 21, wherein the sub-pixel motion of the reference object is in a range of 0.03 to 0.3 pixels.

39. The device of claim 21, wherein the corrected motion of the target object is in a range of 0.03 to 0.3 pixels or in a range of 0.003 to 0.03 pixels.

40. The device of claim 21, wherein the sub-pixel motion of the reference object is in a range of 0.003 to 0.03 pixels.

41. A method of measuring motion of an object using camera images, the method comprising:

measuring a global optical flow field of a scene including a target object captured as representations in an image sequence of the scene;

determining motion, relative to the scene, of a camera used to capture the image sequence by obtaining a motion measurement from an external sensor, the motion measurement being sub-pixel with respect to a pixel array of the camera, wherein determining motion of the camera includes obtaining the sub-pixel motion measurement from the external sensor within a frequency range of motion on a same order of magnitude as a frequency range of motion of the target object; and

calculating a corrected motion of the target object, corrected for the camera motion, based on the optical flow field of the scene and the sub-pixel motion measurement from the external sensor.

42. The method of claim 41, wherein determining motion of the camera further includes measuring an apparent, sub-pixel motion of the reference object with respect to an imaging plane of the camera.

43. The method of claim 41, wherein determining motion of the camera includes obtaining the sub-pixel motion measurement from the external sensor in a range of 0.03 to 0.3 pixels.

44. The method of claim 41, wherein determining motion of the camera includes obtaining the sub-pixel motion measurement from the external sensor in a range of 0.003 to 0.03 pixels.