Methods, systems and robots for processing omni-directional image data

• US 9,930,252 B2
• Filed: 12/06/2012
• Issued: 03/27/2018
• Est. Priority Date: 12/06/2012
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Djugash, Joseph Maria Angelo
• Primary Examiner(s): Shibru, Helen
• Application Number:
  US13/706,760
• Publication Number:
  US 20140160229A1
• Time in Patent Office: 1,937 Days
• Field of Search: 348 36, 382384, 386224, 386227
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: Y10S 901/46 Sensing device

Abstract

Methods, systems, and robots for processing omni-directional image data are disclosed. A method includes receiving omni-directional image data representative of a panoramic field of view and segmenting, by one or more processors, the omni-directional image data into a plurality of image slices. Each image slice of the plurality of image slices is representative of at least a portion of the panoramic field of view of the omni-directional image data. The method further includes calculating a slice descriptor for each image slice of the plurality of image slices and generating a current sequence of slice descriptors. The current sequence of slice descriptors includes the calculated slice descriptor for each image slice of the plurality of image slices.

Claims

1. A method for processing omni-directional image data comprising:

receiving omni-directional image data, wherein the omni-directional image data is representative of a panoramic field of view;

segmenting, by one or more processors, the omni-directional image data into a plurality of image slices, wherein each image slice of the plurality of image slices is representative of at least a portion of the panoramic field of view of the omni-directional image data;

calculating a slice descriptor for each image slice of the plurality of image slices;

generating a current sequence of slice descriptors, wherein the current sequence of slice descriptors includes the calculated slice descriptor for each image slice of the plurality of image slices;

accessing a reference sequence of slice descriptors in a data storage device;

determining whether the current sequence of slice descriptors matches the reference sequence of slice descriptors;

determining a difference between the current sequence of slice descriptors and the reference sequence of slice descriptors in response to the current sequence of slice descriptors not matching the reference sequence of slice descriptors; and

determining a current orientation based on the difference and a known orientation associated with the reference sequence,

wherein determining whether the current sequence of slice descriptors matches the reference sequence of slice descriptors comprises determining whether an order of the reference sequence of slice descriptors matches an order of the current sequence of slice descriptors.

2. The method of claim 1, further comprising storing the generated current sequence in a data storage device.

3. The method of claim 1, wherein whether the current sequence of slice descriptors matches the reference sequence of slice descriptors is determined by:

determining a current order of slice descriptors;

determining a reference order of slice descriptors; and

comparing the current order of slice descriptors to the reference order of slice descriptors.

4. The method of claim 3, wherein the current sequence of slice descriptors is a current circular linked list of slice descriptors and the reference sequence of slice descriptors is a reference circular linked list of slice descriptors, wherein:

the current order of slice descriptors is determined by traversing the current circular linked list of slice descriptors starting at a current starting node;

the reference order of slice descriptors is determined by traversing the reference circular linked list of slice descriptors starting at a reference starting node; and

the current sequence of slice descriptors matches the reference sequence of slice descriptors when the current order of slice descriptors is the same as the reference order of slice descriptors.

5. The method of claim 1, further comprising when the current sequence of slice descriptors matches the reference sequence of slice descriptors, estimating an orientation of an object including one or more image capture devices based on the current sequence of slice descriptors and the reference sequence of slice descriptors.

6. The method of claim 1, further comprising when the current sequence of slice descriptors matches the reference sequence of slice descriptors, estimating a position of an object including one or more image capture devices based on the current sequence of slice descriptors and the reference sequence of slice descriptors.

7. The method of claim 1, further comprising unwarping the received omni-directional image data, wherein the unwarped omni-directional image data is segmented into the plurality of image slices.

8. The method of claim 1, wherein:

the plurality of image slices includes a middle image slice, a preceding image slice, and a subsequent image slice, wherein a middle field of view of the middle image slice is adjacent a preceding field of view of the preceding image slice and the middle field of view of the middle image slice is adjacent to a subsequent view of the subsequent image slice; and

the current sequence of slice descriptors includes a preceding node corresponding to the preceding image slice, a middle node corresponding to the middle image slice, and a subsequent node corresponding to the subsequent image slice, wherein the preceding node is linked to the middle node and the middle node is linked to the subsequent node.

9. The method of claim 1, wherein a number of the plurality of image slices is between 8 and 36.

10. The method of claim 1, wherein each image slice of the plurality of image slices is representative of an equal portion of the panoramic field of view of the omni-directional image data and the collective fields of view of the plurality of image slices is the same as the panoramic field of view of the omni-directional image data.

11. The method of claim 1, wherein the slice descriptor includes a decimal vector.

12. The method of claim 1, wherein the slice descriptor includes a binary vector.

13. The method of claim 1, wherein the slice descriptor is calculated using an algorithm selected from the group consisting of:

scale-invariant feature transform, speeded up robust feature, histogram of oriented gradients, generalized search tree, fast retina keypoint, binary robust invariant scalable keypoints, and stereo algorithms.

14. An omni-directional image data processing system comprising:

one or more processors;

a non-transitory memory component communicatively coupled to the one or more processors;

a data storage device; and

machine readable instructions stored in the non-transitory memory component that cause the omni-directional image data processing system to perform at least the following when executed by the one or more processors;

receive omni-directional image data, wherein the omni-directional image data is representative of a panoramic field of view;

segment the omni-directional image data into a plurality of image slices, wherein each image slice of the plurality of image slices is representative of at least a portion of the panoramic field of view of the omni-directional image data;

calculate a slice descriptor for each image slice of the plurality of image slices;

generate a current sequence of slice descriptors, wherein the current sequence of slice descriptors includes the calculated slice descriptor for each image slice of the plurality of image slices;

access a reference sequence of slice descriptors in the data storage device;

determine whether the current sequence of slice descriptors matches the reference sequence of slice descriptors;

determine a difference between the current sequence of slice descriptors and the reference sequence of slice descriptors in response to the current sequence of slice descriptors not matching the reference sequence of slice descriptors; and

determine a current orientation based on the difference and a known orientation associated with the reference sequence,

wherein whether the current sequence of slice descriptors matches the reference sequence of slice descriptors is determined by determining whether an order of the reference sequence of slice descriptors matches an order of the current sequence of slice descriptors.

15. The omni-directional image data processing system of claim 14, further comprising one or more image capture devices, wherein the omni-directional image data is received from the one or more image capture devices.

16. The omni-directional image data processing system of claim 14, wherein the machine readable instructions stored in the non-transitory memory component further cause the omni-directional image data processing system to perform at least the following when executed by the one or more processors:

determine a current order of slice descriptors;

determine a reference order of slice descriptors; and

compare the current order of slice descriptors to the reference order of slice descriptors.

18. The omni-directional image data processing system of claim 16, wherein the current sequence of slice descriptors is a current circular linked list of slice descriptors and the reference sequence of slice descriptors is a reference circular linked list of slice descriptors, wherein:

the current order of slice descriptors is determined by traversing the current circular linked list of slice descriptors starting at a current starting node;

the reference order of slice descriptors is determined by traversing the reference circular linked list of slice descriptors starting at a reference starting node; and

the current sequence of slice descriptors matches the reference sequence of slice descriptors when the current order of slice descriptors is the same as the reference order of slice descriptors.

17. A robot comprising:

one or more processors;

one or more image capture devices communicatively coupled to the one or more processors;

a non-transitory memory component communicatively coupled to the one or more processors;

a data storage device; and

machine readable instructions stored in the non-transitory memory component that cause the robot to perform at least the following when executed by the one or more processors;

receive omni-directional image data from the one or more image capture devices, wherein the omni-directional image data is representative of a panoramic field of view;

segment the omni-directional image data into a plurality of image slices, wherein each image slice of the plurality of image slices is representative of at least a portion of the panoramic field of view of the omni-directional image data;

calculate a slice descriptor for each image slice of the plurality of image slices;

generate a current sequence of slice descriptors, wherein the current sequence of slice descriptors includes the calculated slice descriptor for each image slice of the plurality of image slices;

access a reference sequence of slice descriptors in the data storage device;

determine whether the current sequence of slice descriptors matches the reference sequence of slice descriptors;

determine a difference between the current sequence of slice descriptors and the reference sequence of slice descriptors in response to the current sequence of slice descriptors not matching the reference sequence of slice descriptors; and

determine a current orientation based on the difference and a known orientation associated with the reference sequence,

wherein whether the current sequence of slice descriptors matches the reference sequence of slice descriptors is determined by determining whether an order of the reference sequence of slice descriptors matches an order of the current sequence of slice descriptors.

19. The robot of claim 17, wherein whether the current sequence of slice descriptors matches the reference sequence of slice descriptors is determined by:

determining a current order of slice descriptors;

determining a reference order of slice descriptors; and

comparing the current order of slice descriptors to the reference order of slice descriptors.

20. The robot of claim 19, wherein the current sequence of slice descriptors is a current circular linked list of slice descriptors and the reference sequence of slice descriptors is a reference circular linked list of slice descriptors, wherein:

the current order of slice descriptors is determined by traversing the current circular linked list of slice descriptors starting at a current starting node;

the reference order of slice descriptors is determined by traversing the reference circular linked list of slice descriptors starting at a reference starting node; and

the current sequence of slice descriptors matches the reference sequence of slice descriptors when the current order of slice descriptors is the same as the reference order of slice descriptors.