Handling processor-intensive operations in a data processing system

• US 6,389,421 B1
• Filed: 08/04/1998
• Issued: 05/14/2002
• Est. Priority Date: 12/11/1997
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hawkins, John Christopher, Nuttall, Mark Patrick, Ibbotson, John Bryan
• Primary Examiner(s): BANANKHAH, MAJID A
• Application Number:
  US09/129,322
• Time in Patent Office: 1,379 Days
• Field of Search: 709/100, 709/101, 709/102, 709/103, 709/106, 709/108, 709/312, 709/217, 709/218, 709/219, 707/10, 707/104.1, 713/164, 713/187
• US Class Current: 1/1
• CPC Class Codes: G06F 2209/484 Precedence
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Provided are a system and a method for data retrieval which identify from requested data retrieval operations those operations which require a particular processing task to be performed, and then separate those operations from operations not requiring the processing task. The separated sets of tasks are queued separately for independent processing. This enables resource scheduling to be performed which avoids the operations which do not require performance of the task from having to wait for the operations which do require the task. This is an advantage where the task is a processor-intensive task such as digital watermarking of images.

A particular resource allocation method includes enqueuing the set of operations requiring the processing task in a circularly linked list and then employing a scheduler to implement a round-robin allocation of resources for each of the system users in turn.

Also provided is a pre-fetch policy whereby sets of data objects are retrieved from a data repository for processing in response to data retrieval requests and post-retrieval processing such as watermarking is initiated before an individual data object in the set has been selected.

Claims

1. A method for retrieving data using a data retrieval system, in response to data retrieval requests, the method including:

responsive to receipt of data retrieval requests by the data retrieval system, identifying requested data retrieval operations requiring a first predetermined processor-intensive task to be performed;

separating the identified processor-intensive data retrieval operations from other data retrieval operations by inputting into a first processing queue only the data retrieval operations requiring said first predetermined processor-intensive task to be performed;

processing data retrieval operations which do not require said first predetermined task to be performed separately from data retrieval operations in the first pocessing queue;

processing the data retrieval operations in the first processing queue including performing said first predetermined processor-intensive task; and

delivering the processed data to the requester.

2. A data retrieval method according to claim 1 including:

inputting into a second processing queue separate from said first queue data retrieval operations which require performance of a second predetermined task but do not require performance of said first predetermined task; and

processing said first and second queues independently.

3. A data retrieval method according to claim 2 wherein the second predetermined task is a data format conversion task.

4. A data retrieval method according to claim 1, wherein the step of identifying requested data retrieval operations requiring performance of said first task includes comparison between data object metadata obtained from the data retrieval requests and a lookup table relating data objects to required processing tasks.

5. A data retrieval method according to claim 1, wherein said first processing queue comprises a plurality of queues and data retrieval operations are allocated between said plurality of queues according to the size of the requested data objects.

6. The method according to claim 1, wherein said predefined unit of processing corresponds to the amount of processing required to complete a predefined number of processing tasks.

7. The method according to claim 1, wherein said predefined unit of processing is a predefined processor operation time.

8. A method for allocating data processing resources between users of a data retrieval system, the method including:

inputting data retrieval requests to a scheduler within the data retrieval system;

identifying requested data retrieval operations requiring performance of a first predetermined processor-intensive task;

retrieving requested data objects from a repository;

separating the identified processor-intensive data retrieval operations from other data retrieval operations by inputting into a first circularly-linked-list the data retrieval operations requiring performance of said predetermined task, the operations being indexed according to their end user requesters;

processing data retrieval operations which do not require performance of said predetermined task separately from data retrieval operations in the first pocessing queue; and

processing data retrieval operations in said first circularly-linked-list, including performing said predetermined task, in a circular sequence such that a predefined unit of processing is performed for each end user requester in turn.

9. A method according to claim 8, including

inputting into a second circularly-linked list separate from the first circularly-linked-list data retrieval operations which do not require performance of said first predetermined task but do require performance of a second post-retrieval task, the operations in the second list being indexed according to their end user requesters; and

processing the second list in a circular sequence independent of processing of the first list such that for each list a predefined unit of processing is performed for each end user requester in turn.

10. A method according to claim 8, wherein said predefined unit of processing corresponds to the amount of processing required to complete a predefined number of processing tasks.

11. A method according to claim 8, wherein said predefined unit of processing is a predefined processor operation time.

12. A method of retrieving requested data objects via a data retrieval system, wherein the system is responsive to received requests identifying a required data object container to

retrieve from a data repository the contents of the data object container;

determine which of the data objects within the container require performance of a processor-intensive post-retrieval processing task prior to delivery to the requester;

initiate performance of said required processor-intensive post-retrieval processing task for the data objects within the container; and

transfer to a system cache the processed data objects resulting from said post-retrieval processing;

and wherein the system is responsive to subsequent requests identifying a required one or more of said container'"'"'s data objects to access the cache to obtain said one or more data objects.

13. A method according to claim 12, wherein the system is responsive to receipt of a request identifying a required one or more of said container'"'"'s data objects, which request is received after said initiation of performance of the post-retrieval processing but before its completion, to promote said data object as the next data object to be processed.

14. A data retrieval system for retrieving stored data objects from a repository, the system including:

means for receiving data retrieval requests;

means for establishing a connection to said repository for retrieval of one or more stored data objects;

a resource manager adapted to identify from said requests data retrieval operations requiring a first predetermined processor-intensive task to be performed and adapted to enqueue, subsequent to retrieval of said one or more data objects from the repository, said identified data retrieval operations requiring performance of said first predetermined processor-intensive task for processing separately from other data retrieval operations;

a process for performing said first predetermined processor-intensive task on the operations in said queue; and

means for delivering to the requester the processed retrieved data objects.

15. A system according to claim 14, including

a repository wherein stored image objects are arranged in containers, each containing a plurality of thumbnail images and full images; and wherein the resource manager is responsive to data retrieval requests identifying a container to

retrieve the contents of the identified container,

deliver the thumbnail images within said container to the requester, and

enqueue said full images within said container for performance of said first predetermined task prior to receipt by the system of a request for delivery of a specific one of said full images, which request for said full image is separate from said container-identifying request.

16. An article of manufacture comprising a computer usable medium having computer readable program code means embodied therein for retrieving data using a data retrieval system, in response to data retrieval requests, the computer readable program code means in said article of manufacture comprising computer readable program code means for causing a computer to effect:

being responsive to receipt of data retrieval requests by the data retrieval system, identifying requested data retrieval operations requiring a first predetermined task to be performed;

inputting into a first processing queue only the data retrieval operations requiring said first predetermined task to be performed;

handling data retrieval operations which do not require said first predetermined task to be performed without performing said first predetermined task;

processing the data retrieval operations in the first processing queue including performing said first processing task; and

delivering the processed data to the requester.

17. An article of manufacture as recited in claim 16, the computer readable program code means in said article of manufacture further comprising computer readable program code means for causing a computer to effect:

inputting into a second processing queue separate from said first queue data retrieval operations which require performance of a second predetermined task but do not require performance of said first predetermined task; and

processing said first and second queues independently.

18. An article of manufacture as recited in claim 16, wherein the step of identifying requested data retrieval operations requiring performance of said first task includes comparison between data object metadata obtained from the data retrieval requests and a lookup table relating data objects to required processing tasks.

19. An article of manufacture as recited in claim 16, wherein said first processing queue is a circularly-linked list to which requested data retrieval operations are added, the operations being indexed according to the ID'"'"'s of their end user requesters, and wherein the data retrieval operations in said circularly-linked-list are processed in a circular sequence such that a predefined unit of processing is performed for each end user requester in turn.

20. A computer program product comprising a computer usable medium having computer readable program code means embodied therein for allocating data processing resources between users of a data retrieval system, the computer readable program code means in said computer program product comprising computer readable program code means for causing a computer to effect:

inputting data retrieval requests to a scheduler within the data retrieval system;

identifying requested data retrieval operations requiring performance of a first predetermined processor-intensive task;

retrieving requested data objects from a repository;

inputting into a first circularly-linked-list the data retrieval operations requiring performance of said predetermined task, the operations being indexed according to their end user requesters;

handling data retrieval operations which do not require performance of said predetermined task without performing said predetermined task; and

processing data retrieval operations in said first circularly-linked-list, including performing said predetermined task, in a circular sequence such that a predefined unit of processing is performed for each end user requester in turn.

21. A computer program product as recited in claim 20, the computer readable program code means in said computer program product further comprising:

inputting into a second circularly-linked list separate from the first circularly-linked-list data retrieval operations which do not require performance of said first predetermined task but do require performance of a second post-retrieval task, the operations in the second list being indexed according to their end user requesters; and

processing the second list in a circular sequence independent of processing of the first list such that for each list a predefined unit of processing is performed for each end user requester in turn.

22. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for retrieving requested data objects via a data retrieval system, wherein the system is responsive to received requests identifying a required data object container to:

retrieve from a data repository the contents of the data object container;

determine which of the data objects within the container require post-retrieval processing prior to delivery to the requester;

initiate performance of said required post-retrieval processing for the data objects within the container; and

transfer to a system cache the processed data objects resulting from said post-retrieval processing;

and wherein the system is responsive to subsequent requests identifying a required one or more of said container'"'"'s data objects to access the cache to obtain said one or more data objects.

23. A program storage device readable by machine as recited in claim 22, wherein the system is further responsive to receipt of a request identifying a required one or more of said container'"'"'s data objects, which request is received after said initiation of performance of the post-retrieval processing but before its completion, to promote said data object as the next data object to be processed.