US20010014928A1

US20010014928A1 - Dynamically disabling speculative prefetch when high priority demand fetch opportunity use is high

Info

Publication number: US20010014928A1
Application number: US09/835,279
Authority: US
Inventors: George Chrysos; Wilson Snyder
Original assignee: Compaq Information Technologies Group LP
Current assignee: Hewlett Packard Development Co LP
Priority date: 1998-11-02
Filing date: 2001-04-13
Publication date: 2001-08-16
Anticipated expiration: 2018-11-02
Also published as: US6233645B1; US6324616B2

Abstract

A method of limiting, in a digital processor, low-priority utilization of a resource in favor of high-priority utilization of the resource comprises determining a value predictive of high-priority utilization of the resource. Low-priority utilization of the resource is inhibited if the determined predictive value is greater than a threshold. On the other hand, if the predictive value is less than or equal to the threshold, then low-priority utilization of the resource is allowed. In a preferred embodiment, the predictive value is derived by counting the number of actual high-priority utilizations of the resource out of the last N opportunities in which the resource could have been utilized for a high-priority need. Preferably, recent utilizations are given more weight than others. In a preferred embodiment, the resource comprises one of main memory, instruction cache memory, or data cache memory. High-priority utilization comprises high-priority access to memory and low-priority utilization comprises memory prefetching. Low priority utilizations are throttled when the threshold is exceeded. Counting and comparing are done with a shift register, counting logic, and a comparator; or with a counter and a comparator; or with two shift registers in which case no comparator is needed.

Description

RELATED APPLICATION

This application is a continuation of application Ser. No. 09/184,623, filed on Nov. 2, 1998. The entire teachings of the above application is incorporated herein by reference. [0001]

BACKGROUND OF THE INVENTION

Modern microprocessors comprise many resources such as registers and memory for which various requesters contend. For example, a typical microprocessor has various layers of memory including local instruction and data caches, lower level backup caches (B-cache), main memory, storage devices such as disk drives, and storage available over a network. Generally, these layers form a hierarchical set of memories where at one extreme, the local caches are very fast but tend to be small, whereas at the other extreme, main memory, disk storage, and network storage tend to have very large capacities, but access times are also magnitudes larger than for local memory.

To reduce the impact of the long latency between processor and main memory or other long latency storage, hardware prefetching techniques are utilized. The CPU predicts which memory blocks it will utilize next and requests those blocks from the memory hierarchy before they are actually needed.

Because it cannot be predicted with certainty which branch a branch instruction in the instruction stream will take, these prefetched instructions are said to be speculative. Unfortunately, these prefetches can consume bandwidth to the lower level caches, main memory, etc., reducing the amount of available bandwidth for memory transactions that are needed as soon as possible, for example, for demand misses and cache-victim processing.

SUMMARY OF THE INVENTION

The present invention addresses this problem by providing a mechanism which dynamically detects when a high percentage of memory-access bandwidth is needed for high-priority operations such as demand misses, and which, based on this detection, shuts down low priority requesters such as hardware prefetchers until the demand for the bandwidth is reduced.

A preferred embodiment of the present invention realizes this mechanism by counting the number of high-priority utilizations from the last N opportunities in which the high-priority utilizations could have been requested. When the count exceeds some threshold, indicating high usage by the high-priority requester, a portion of low-priority requests are throttled or disabled. Priority may be determined, for example, by measuring the degree of speculative use of the resource. Thus, speculative prefetches have a lower priority than demand accesses.

In other words, the present invention works on the principle that if bandwidth utilization due to high-priority accesses has been high in the recent past, then it will continue to be high due to high-priority accesses in the near future. To that end, the present invention prohibits hardware prefetchers, which have a lower priority, from making requests during high bandwidth utilization periods to increase overall performance.

Accordingly, a method for arbitrating among a plurality of requesters of resources in a computer system comprises selecting a first requester based on a history of program execution. Other requesters which compete with the first requester are inhibited.

Preferably, the history is based on past utilization of the resources. Preferably, memory, including but not limited to main memory and cache memory, and associated data paths, are resources from which a demand fetcher and a speculative prefetcher request instructions or data. The demand fetcher is the first requester, while the speculative prefetcher is a competing requester.

In a preferred embodiment of the present invention, low-priority utilization of a resource in a digital processor, such as speculative prefetching, is limited in favor of high-priority utilization of the resource such as demand fetching, by determining a value which is predictive of high-priority utilization of the resource. If the predictive value is greater than some threshold, low-priority utilization of the resource is inhibited, or throttled. If the predictive value is less than or equal to the threshold, low-priority utilization of the resource is allowed. Preferably, the threshold is a predefined constant.

Furthermore, the predictive value is preferably determined by counting the number of opportunities in which the resource could have been utilized for a high-priority need, regardless of whether it actually was so utilized. The number of actual high-priority utilizations of the resource is also counted, and divided by the opportunity count, yielding the predictive value.

In a preferred embodiment, some actual high-priority utilizations are given more weight than others. Preferably, recent high-priority utilizations are weighted more heavily than less recent high-priority utilizations.

In another preferred embodiment, the number m of actual high-priority uses from the last N use opportunities is first determined, where N is predefined. The value m is compared with some threshold. Speculative accesses are inhibited if m is greater than a threshold, but are allowed otherwise.

In a preferred embodiment, m is determined by providing an N-bit shift register. A 1 is shifted into the N-bit shift register for each high-priority use opportunity in which there has been an actual high-priority use. A 0 is shifted into the shift register for all other use opportunities, such that a count of 1s in the shift register provides a value for m. Optionally, hysteresis can be used about the threshold.

In a further preferred embodiment, a second shift register is provided. When a 0 is shifted in to the N-bit shift register and 1 is shifted out, the second register is shifted left, with a 0 being shifted in from the right. When a 1 is shifted in to the N-bit register and 0 is shifted out, the second register is shifted right, with a 1 being shifted in from the left. When a certain predesignated bit in the second shift register, associated with the threshold has a value of 1, m is deemed to exceed the threshold.

Similarly, where a plurality of options is available, the techniques of the present invention may be employed by associating a priority level with each option. A value is determined based on past behavior of at least one of the options. At least one high-priority option is selected if the determined value exceeds a threshold such that lower priority options are effectively inhibited. On the other hand, any option may be selected if the determined value does not exceed the threshold.

Finally, where multiple threads are executing, the techniques of the present invention may be employed to select a thread based on utilization of bandwidth, or based on a prioritization among the threads.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of preferred embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. [0018]
FIG. 1 is a block diagram of a computer system having a main memory and backup cache. [0019]
FIG. 2 is a schematic diagram of a preferred embodiment of the present invention using a shift register and counting logic. [0020]
FIG. 3 is a schematic diagram of the embodiment of FIG. 2 with weights added. [0021]
FIG. 4 is a schematic diagram depicting the use of an upper and lower threshold to characterize a variation of the embodiments of the FIGS. 2 and 3 with hysteresis. [0022]
FIG. 5A is a schematic diagram of a preferred embodiment with two shift registers. [0023]
FIG. 5B is a schematic diagram of a preferred embodiment of the present invention using a counter. [0024]

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a [0025] general computer system 100. A microprocessor (CPU) 101 generally fetches its instructions and data from a main memory 113. However, this main memory and the system bus 111 over which instructions and data travel, tend to slow down the execution of the microprocessor 101 which must wait to receive the instructions or data.
To speed up execution, a [0026] modem microprocessor 101 typically has an on-board instruction cache, or I-cache 103, and a data cache, or D-cache 105. Based on the fact that recently-fetched instructions or data tend to be reused, these caches 103, 105 provide the microprocessor 101 with near-instantaneous access to the instructions or data. While the discussion hereafter focuses only on instruction fetching, it will be recognized by those skilled in the art that the discussion applies equally to data fetching.
Memory hierarchies may comprise several caching levels. The first level instruction and data caches described above need to be accessed very quickly. This restriction limits the size of these first-level caches, creating the need for other caches which are larger but are slower to access. Therefore, many computer systems have backup caches, or B-[0027] caches 109, which may reside on-chip or off-chip, and which serve as intermediary caches between the first-level caches and main memory 113 or other storage. When needed, instructions or data are retrieved from the first- level caches 103, 105, or from lower levels of the memory hierarchy, as available.
Fetching [0028] hardware 108 fetches instructions or data typically as the need arises. Such fetches are denoted hereafter as “demand requests”. In addition, modem CPUs utilize hardware prefetching techniques to reduce the impact of the long latency between the CPU and main memory. The CPU prefetcher 107 predicts which memory blocks the CPU will utilize in the near future and requests those blocks from the memory hierarchy before they are actually needed. The CPU cannot be certain ahead of time which instructions and data it will need in the future. For instruction prefetch, this uncertainty is typically due to branch instructions in the code. This is referred to as “speculative prefetching”. Because the results are not needed immediately, timing is less critical with prefetching requests than with demand requests.
Unfortunately, prefetching consumes significant bandwidth to the memory hierarchy, thereby reducing the amount of available bandwidth for memory transactions that are needed as soon as possible, such as for demand misses, cache-victim processing, etc. This can be a problem even when the priority of the prefetch requests has been lowered to below that of other demand misses, because, for example, the prefetch cycle could have already begun. Thus there is a need for a mechanism which dynamically detects when a high percentage of the bandwidth is needed for high-priority accesses such as demand misses, and which shuts down the hardware prefetcher until the demand for bandwidth decreases. [0029]
FIG. 2 is a schematic diagram of a preferred embodiment of the present invention. [0030] Instruction fetching hardware 108 is responsible for fetching instructions from memory, e.g., the B-cache 109. Within the fetching hardware 108 is the hardware prefetcher 107 which is responsible for speculative prefetching. Of course, there could be multiple hardware prefetchers, for example, for instructions and for data.
The present invention works on the principle that if bandwidth utilization for high-priority requests has been high in the recent past, then it will continue to be high in the near future. When recent bandwidth utilization has been high for high-priority requests, speculative requests from the [0031] hardware prefetcher 107 are prohibited from making requests.
Information about the recent history of memory bandwidth utilization of a resource, such as a B-[0032] cache 109, is captured using a shift register 115 of arbitrary length N. Each bit 116 in the shift register 115 represents one usage opportunity for the memory resource. The fetcher 108 outputs a signal 119 which is indictive of resource usage. For example, if a high-priority demand request was made in the last opportunity, the fetcher 108 shifts a 1 into the register 115 via signal 119. Otherwise, the fetcher 108 shifts a 0 into the register 115. If the register 115 contains N bits, then it holds information about the last N usage opportunities. When the register 115 is full, the oldest information is shifted out of the register 115 as new information is shifted in. To avoid unnecessarily inhibiting the prefetcher 107, low-priority utilizations such as prefetches are preferably not recorded in the shift register 115 as consuming a usage opportunity.
In the present invention, prefetch requests are inhibited depending on some function based upon the contents of the [0033] shift register 115. Preferably, this function is a count of the bits 116 in the shift register 115 that are set to 1. This is represented schematically in FIG. 2 by counting logic 117 whose output 125 represents the number of 1s in the shift register 115. This count 125 is compared by comparator 121 to some threshold 127, and if the count 125 exceeds the threshold 127, the comparator 121 generates a choke signal 123 which inhibits the prefetcher 107. Otherwise prefetching is not inhibited.
FIG. 3 is a schematic diagram of the embodiment similar to that of FIG. 2 in which the [0034] bits 116 of the shift register 115 are individually assigned weights 129. Thus, the counting logic 117 in FIG. 3 may give more weight to some bits 116 than others, for example, counting some bits 116 more than once. Preferably, more weight is given to more recent bits 120 in the register 115 than to the older bits 122. The ultimate output of counter logic 117 and operation of the remaining parts is the same as in FIG. 2 as previously discussed.
FIG. 4 is a schematic diagram depicting the use of two [0035] thresholds 135—an upper threshold Th_Hand a lower threshold ThL—to characterize a preferred embodiment of the present invention with hysteresis around a nominal threshold. The illustrated thresholds Th_Land Th_Hare used instead of threshold 127 Th in FIGS. 2 and 3. Selector 131 selects the upper threshold Th_Hwhen the comparator output 123 indicates that the usage has been low and that prefetching is allowed, and selects the lower threshold Th_Lwhen the comparator output 123 indicates that the usage has been high and prefetching has been turned off.
FIG. 5A is a schematic diagram of a preferred embodiment using two shift registers, SRA and SRB. Assume that logic in the fetching [0036] hardware 108 produces either a 0 or 1 to be shifted into shift register SRA, according to whether the resource was actually used at the last usage opportunity. As before, SRA can be of arbitrary length N.
The shift register SRA's input bit [0037] 201 and output bit 209 are brought to logic circuit 203. The logic circuit 203 has the following truth table:

SRA input 201 SRA output 209 SHL SHR Comment

0 0 0 0 No change - no action

0 1 1 0 a 1 has been dropped

1 0 0 1 a 1 has been added

1 1 0 0 No change - no action
The SHL and SHR output signals from the [0038] logic circuit 203 command the second shift register SRB to shift left or right respectively. When shift register SRB is shifted to the left via the SHL signal, a 0 is shifted in from the right 213. On the other hand, when register SRB is shifted to the right via the SHR signal, a 1 is shifted in from the left 215. Thus, both registers hold the same number of 1s and 0s, but while register SRA's bits are ordered historically, register SRB groups all of the 1s contiguously to the left side 215 and groups all of the 0s contiguously to the right side 213. Of course, it is obvious to those skilled in the art that the terms left and right are used to differentiate one from the other, and may just as easily be reversed.
Referring to the above logic truth table, when the input and [0039] output bits 201, 209 to shift register SRA are the same, i.e., both 0s or both 1s, there is no change to the total number of is in the register, and so no action is taken with respect to shift register SRB. However, when the input 201 is a 0 and the output bit 209 is a 1, shift register SRA is losing a 1. Therefore register SRB is shifted left. Note again that when SRB is shifted left, a 0 is shifted in. Finally, when the input bit 201 is a 1 and the output bit 209 is a 0, shift register SRA is gaining a 1, and register SRB is shifted right, with a 1 shifting in from the left 215.
Now, to see if there are more than n actual uses out of the last N opportunities where n is the value of the threshold, i.e., to see if there are more than n 1s in shift register SRA or equivalently, register SRB, the hardware only needs to look at predesignated bit n of register SRB. Predesignated bit n is the n+1st bit from the left [0040] 215 end of register SRB as illustrated in FIG. 5 where the leftmost bit is 0. No counting is required. If the value 205 of register SRB bit n is 1, the threshold has been exceeded. In turn, prefetcher 107 is inhibited. Otherwise, when the value 205 of bit n is 0, the threshold is not met and prefetcher 107 remains active.
FIG. 5B shows an alternate embodiment of the present invention in which a [0041] counter 225 and comparator 227 perform essentially the same function as the second shift register SRB of FIG. 5A. The same logic circuit 203 is used, however, its output SHL and SHR signals become decrement and increment signals respectively to the counter 225, which keeps a running tally of the number of is in register SRA. As with the embodiment of FIG. 2, the counter's value 226 is compared with a threshold at comparator 227, which generates a choke signal 123 to inhibit the prefetcher 107.
It will be obvious to those skilled in the art that the above use of two shift registers with the logic described above can be used to measure bandwidth for other goals as well as throttling prefetching, e.g. performance evaluation with respect to any type of resource usage. The roles of 0s and 1s could similarly be reversed. [0042]
In another embodiment of the present invention, a thread is selected based on bandwidth, or prioritization among the threads. [0043]
While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments of the invention described specifically herein. Such equivalents are intended to be encompassed in the scope of the claims. [0044]

Claims

What is claimed is:

1. In a computer system, a method for arbitrating among a plurality of requesters of resources, comprising the steps of:

selecting a first requester based on a history of program execution; and

inhibiting competing requesters of the first requester.

2. The method of

claim 1

wherein the history of program execution comprises a history of utilization of the resources.

3. The method of

claim 2

wherein memory is a resource, and a demand fetcher and a speculative prefetcher are requesters for the memory, the demand fetcher being the first requester and the speculative prefetcher being a competing requester.

4. The method of

claim 3

wherein instructions are fetched by the demand fetcher and the speculative prefetcher.

5. The method of

claim 3

wherein data are fetched by the demand fetcher and the speculative prefetcher.

6. The method of

claim 3

wherein memory comprises main memory.

7. The method of

claim 3

wherein memory comprises cache memory.

8. A method of limiting, in a digital processor, low-priority utilization of a resource in favor of high-priority utilization of the resource, comprising the steps of:

determining a value predictive of high-priority utilization of the resource;

inhibiting low-priority utilization of the resource if the predictive value is greater than a threshold; and

allowing low-priority utilization of the resource if the predictive value is less than or equal to the threshold.

9. The method of

claim 8

wherein the threshold is predefined.

10. The method of

claim 8

wherein the threshold is constant.

11. The method of

claim 8

, wherein the step of determining the predictive value further comprises:

counting opportunities the resource could have been utilized for a high-priority need, regardless of whether it actually was so utilized;

counting actual high-priority utilizations of the resource; and

dividing the actual count by the opportunity count.

12. The method of

claim 8

wherein the step of determining includes weighting some high-priority utilizations relative to others.

13. The method of

claim 12

wherein recent high-priority utilizations are weighted more heavily than less recent high-priority utilizations.

14. The method of

claim 8

, wherein the resource comprises one of main memory, instruction cache memory, or data cache memory.

15. The method of

claim 14

wherein high-priority utilization comprises demand access to memory.

16. The method of

claim 15

wherein low-priority utilization comprises memory prefetching.

17. In a computer systems, a method of throttling resource accesses, comprising:

determining a number m of actual high-priority uses from N last use opportunities, N being a predefined number;

comparing m with a threshold;

inhibiting speculative accesses if m is greater than the threshold; and

allowing speculative accesses otherwise.

18. The method of

claim 17

wherein the threshold is constant.

19. The method of

claim 17

wherein the step of determining includes weighting certain high-priority uses relative to other demand uses to calculate m.

20. The method of

claim 19

wherein recent high-priority uses are given more weight than older high-priority uses.

21. The method of

claim 17

wherein the step of determining m further comprises:

providing an N-bit shift register, the N-bit shift register being a first shift register;

shifting a 1 into the N-bit shift register for a high-priority use opportunity in which there has been an actual high-priority use;

shifting a 0 into the N-bit shift register for all other high-priority use opportunities, such that a count of 1s in the N-bit shift register provides a value for m.

22. The method of

claim 21

wherein the step of comparing m further comprises:

providing a second shift register;

upon a 0 being shifted in to the N-bit shift register and a 1 being shifted out of the N-bit shift register, shifting left the second shift register, such that a 0 is shifted in;

upon a 1 being shifted in to the N-bit register and a 0 being shifted out of the N-bit shift register, shifting right the second shift register, such that a 1 is shifted in; and

determining that m exceeds the threshold when a certain bit, in the second shift register, associated with the threshold has a value of 1.

23. The method of

claim 17

, wherein the step of comparing includes exhibiting properties of hysteresis about the threshold.

24. In a computer system, a method of controlling utilization of resources comprising:

for each resource, measuring a degree of speculation of use of the resource; and

allocating resources based on respective degree of speculation.

25. A method of selecting, in a digital processor, at least one option from among a plurality of options, comprising:

associating a level of priority with each option;

determining a value based on past behavior of at least one of the options;

selecting at least one high-priority option if the determined value exceeds a threshold such that lower priority options are effectively inhibited; and

allowing selection of any option if the determined value does not exceed the threshold.

26. The method of

claim 25

wherein each option is a thread of execution.

27. An apparatus for dynamically throttling a hardware prefetcher in a computer system with fetching hardware and memory, comprising:

a shift register, wherein a 1 is shifted in for each memory high-priority request opportunity in which a memory demand request is made, and wherein a 0 is shifted in for each memory request opportunity in which no memory high-priority request is made;

counting logic coupled to the shift register for producing a count of 1s in the shift register; and

a comparator coupled to receiving the count from the counting logic, said comparator comparing the count of 1s with a predetermined threshold, such that if the count exceeds the threshold, the hardware prefetcher is throttled, and if the count does not exceed the threshold, the hardware prefetcher is not throttled.

28. The apparatus of

claim 27

wherein bits in the shift register are weighted and the counting logic counts corresponding weighted values.

29. The apparatus of

claim 27

wherein the threshold exhibits hysteresis characteristics.

30. An apparatus for dynamically throttling a hardware prefetcher in a computer system with fetching hardware and memory, comprising:

a counter having a value, wherein the counter value is incremented by one for each memory high-priority request opportunity in which a memory high-priority request is made, and wherein the counter value is decremented by one for each memory high-priority request opportunity in which no memory request is made; and

a comparator coupled to the counter and comparing the counter value with a predetermined threshold, such that if the value exceeds the threshold, the hardware prefetcher is throttled, and if the value does not exceed the threshold, the hardware prefetcher is not throttled.

31. An apparatus for dynamically throttling a hardware prefetcher in a computer system with fetching hardware and memory, comprising:

a first shift register, wherein a 1 is shifted in for each memory high-priority request opportunity in which a memory high-priority request is made, and wherein a 0 is shifted in for each memory request opportunity in which no memory high-priority request is made;

a second shift register, responsive to the first shift register, such that

when a 1 is shifted into the first shift register and a 0 is shifted out of the first shift register, the second shift register is shifted in a first direction with a 1 shifting in, and

when a 0 is shifted into the first shift register and a 1 is shifted out of the first shift register, the second shift register is shifted in a second direction with a 0 shifting in, wherein

if a value of a predesignated bit in the second shift register is 1, the hardware prefetcher is throttled, and if the value of the bit is 0, the hardware prefetcher is not throttled.

32. An apparatus for arbitrating among a plurality of requesters of resources in a computer system, comprising:

counting means for determining a count of high priority requests made from a last N opportunities;

inhibitor means coupled to the counting means for throttling low priority requests based on the determined count compared to a threshold.

33. The apparatus of

claim 32

wherein the threshold is predefined.

34. An apparatus for determining resource usage, in a computer system comprising:

a first shift register, wherein a 1 is shifted in for each usage of a certain type of resource, and wherein a 0 is shifted in for each opportunity of the certain type of resource in which no such usage is made;

a second shift register responsive to the first shift register, such that

when a 0 is shifted into the first shift register and a 1 is shifted out of the first shift register, the second shift register is shifted in a second direction with a 0 shifting in,

the second shift register, having a predesignated bit, and the value of the predesignated bit being indicative of recent resource usage.