Hardware architecture for acceleration of computer vision and imaging processing

• US 10,055,807 B2
• Filed: 03/02/2016
• Issued: 08/21/2018
• Est. Priority Date: 03/02/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Seungjin, Lee, Seok-Jun
• Primary Examiner(s): Liu, Zhengxi
• Assistant Examiner(s): Yang, Yi
• Application Number:
  US15/059,175
• Publication Number:
  US 20170256016A1
• Time in Patent Office: 902 Days
• Field of Search: 345502, 345503
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 1/60 Memory management

Abstract

An image and vision processing architecture included a plurality of image processing hardware accelerators each configured to perform a different one of a plurality of image processing operations on image data. A multi-port memory shared by the hardware accelerators stores the image data and is configurably coupled by a sparse crossbar interconnect to one or more of the hardware accelerators depending on a use case employed. The interconnect processes accesses of the image data by the hardware accelerators. Two or more of the hardware accelerators are chained to operate in sequence in a first order for a first use case, and at least one of the hardware accelerators is set to operate for a second use case. Portions of the memory are allocated to the hardware accelerators based on the use case employed, with an allocated portion of the memory configured as a circular buffer.

Claims

1. A system, comprising:

a plurality of image processing hardware accelerators each configured to perform a different one of a plurality of image processing operations on image data;

a multi-port memory shared by the plurality of image processing hardware accelerators and configured to store the image data; and

an interconnection system configurably coupling the memory as a circular buffer to one or more of the image processing hardware accelerators and configured to process accesses of the image data by the image processing hardware accelerators,

wherein for a first use case of the image processing hardware accelerators, two or more of the image processing hardware accelerators are configurably chained to operate in sequence in a first order,

wherein for a second use case of the image processing hardware accelerators, one or more of the image processing hardware accelerators is configurably set to operate alone or chained to operate in sequence in a second order,

wherein at least one of the image processing hardware accelerators is configurably allocated a portion of the memory based on whether the first use case or the second use case is configured, the allocated portion of the memory configured as part of the circular buffer, and

wherein the interconnection system comprises;

a head pointer bus configured to carry a head pointer, which indicates a write progress, for the circular buffer;

a tail pointer bus configured to carry a tail pointer, which indicates a read progress, for the circular buffer;

a plurality of circular buffer write adapters configured to convert incoming image data into circular buffer data;

a plurality of circular buffer read adapters configured to convert circular buffer data into outgoing image data; and

at least one status bus that synchronizes circular buffer data access within the circular buffer, the at least one status bus communicates between the plurality of circular buffer write adapters and the plurality of circular buffer read adapters.

2. The system according to claim 1, wherein different combinations of the image processing hardware accelerators are configurably chained to operate in sequence in different orders for different use cases for the image processing hardware accelerators.

3. The system according to claim 2, wherein each of the combinations of the image processing hardware accelerators is configurably chained by allocation of portions of the shared memory as buffers between consecutive ones of the image processing hardware accelerators in a corresponding sequence for the respective one of the use cases.

4. The system according to claim 1, wherein the shared memory comprises multiple, concurrently accessible static random access memory (SRAM) banks configured to be allocated in portions to each of the one or more of the image processing hardware accelerators employed in a current use case.

5. The system according to claim 4, wherein portions of at least two of the SRAM banks are allocated as one of an input buffer or an output buffer for each of the image processing hardware accelerators employed in the current use case.

6. The system according to claim 4, wherein the interconnection system is configured to connect a read port for one of the image processing hardware accelerators employed in the current use case to one or more of the SRAM banks for memory allocated to store image data on which the one image processing hardware accelerator is to operate and to connect a write port for the one image processing hardware accelerators to one or more of the SRAM banks for memory allocated to store image data produced by operation of the one image processing hardware accelerator.

7. The system according to claim 1, wherein a portion of the shared memory allocated to at least one of the image processing hardware accelerators employed in a current use case is configured as a tile buffer.

8. The system according to claim 1, wherein the interconnection system is a sparse crossbar.

9. The image processing circuit of claim 1, wherein the multi-port memory subsystem comprises multiple, concurrently accessible static random access memory (SRAM) banks configured to be connected to each of the plurality of circular buffer write adapters and each of the plurality of circular buffer read adapters.

10. The image processing circuit of claim 1, wherein the at least one status bus comprises:

a write status bus configured to allow one of the circular buffer write adapters to broadcast a write status message to all of the circular buffer read adapters.

11. The image processing circuit of claim 1, wherein the at least one status bus comprises:

a read status bus configured to allow one of the circular buffer read adapters to broadcast a read status message to each of the circular buffer read adapters.

12. The image processing circuit of claim 1, wherein each of the circular buffer write adapters comprises:

a write status transmitter configured to transmit write status updates with a programmable writer identifier at programmable time intervals.

13. The image processing circuit of claim 1, wherein each of the circular buffer write adapters comprises:

a read status receiver configured to receive read status updates from a read status bus, and, when a reader identifier of a received read status update matches one of multiple programmed reader identifiers, store information for the read status update in a local tail pointer register corresponding to the matched reader identifier.

14. The image processing circuit of claim 1, wherein each of the circular buffer read adapters comprises:

a read status transmitter configured to transmit read status updates with a programmable reader identifier at programmable time intervals.

15. A method, comprising:

for a plurality of image processing hardware accelerators each configured to perform a different one of a plurality of image processing operations on image data, flexibly interconnecting the image processing hardware accelerators by

chaining, for a first use case, two or more of the image processing hardware accelerators to operate in sequence in a first order, and

setting, for a second use case, one or more of the image processing hardware accelerators to operate alone or chained to operate in sequence in a second order;

for a multi-port memory shared by the plurality of image processing hardware accelerators and configured to store the image data, configuring a portion of the shared memory allocated to one of the image processing hardware accelerators as a circular buffer based on whether the first use case or the second use case is employed; and

configuring an interconnection system coupling the memory to one or more of the image processing hardware accelerators to process accesses of the image data by the image processing hardware accelerators, wherein the interconnection system, comprises;

a head pointer bus configured to carry a head pointer, which indicates write progress, for the circular buffer;

a tail pointer bus configured to carry a tail pointer, which indicates read progress, for the circular buffer,

a plurality of circular buffer write adapters configured to convert incoming image data into circular buffer data;

a plurality of circular buffer read adapters configured to convert circular buffer data into outgoing image data; and

at least one status bus that synchronizes circular buffer data access within the circular buffer, the at least one status bus communicates between the plurality of circular buffer write adapters and the plurality of circular buffer read adapters.

16. The method according to claim 15, wherein different combinations of the image processing hardware accelerators are configurably chained to operate in sequence in different orders for different use cases for the image processing hardware accelerators.

17. The method according to claim 16, wherein each of the combinations of the image processing hardware accelerators is configurably chained by allocation of portions of the shared memory as buffers between consecutive ones of the image processing hardware accelerators in a corresponding sequence for the respective one of the use cases.

18. The method according to claim 16, wherein the shared memory comprises multiple, concurrently accessible static random access memory (SRAM) banks configured to be allocated in portions to each of the one or more of the image processing hardware accelerators employed in a current use case.

19. The method according to claim 18, wherein portions of at least two of the SRAM banks are allocated as one of an input buffer or an output buffer for each of the image processing hardware accelerators employed in the current use case.

20. The method according to claim 18, wherein the interconnection system is configured to connect a read port for one of the image processing hardware accelerators employed in the current use case to one or more of the SRAM banks for memory allocated to store image data on which the one image processing hardware accelerator is to operate and to connect a write port for the one image processing hardware accelerators to one or more of the SRAM banks for memory allocated to store image data produced by operation of the one image processing hardware accelerator.

21. The method according to claim 15, wherein a portion of the shared memory allocated to at least one of the image processing hardware accelerators employed in a current use case is configured as a tile buffer.

22. The method according to claim 15, wherein the interconnection system is a sparse crossbar.