US20100082300A1 - Storage input/output utilization associated with a software application - Google Patents
Storage input/output utilization associated with a software application Download PDFInfo
- Publication number
- US20100082300A1 US20100082300A1 US12/242,710 US24271008A US2010082300A1 US 20100082300 A1 US20100082300 A1 US 20100082300A1 US 24271008 A US24271008 A US 24271008A US 2010082300 A1 US2010082300 A1 US 2010082300A1
- Authority
- US
- United States
- Prior art keywords
- storage
- input
- output
- capacity
- operating system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0653—Monitoring storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0605—Improving or facilitating administration, e.g. storage management by facilitating the interaction with a user or administrator
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Definitions
- SANs storage area networks
- NAS network attached storage
- a file may have access to one of several operating system volumes on an operating system disk and each operating system disk may be associated with a storage device logical unit number (LUN) that spans multiple physical storage devices in a storage group within a storage array.
- LUN storage device logical unit number
- One or more of these physical storage devices may furthermore be accessed by an entirely different set of applications through a different computer, logical file, and logical unit number (LUN). It may thus not be possible to view a consolidated end-to-end view of the storage I/O capacity available when accessing the file. Without such an end-to-end view, it is difficult to plan, administer, or automatically adjust the storage I/O capacity to meet the expected level of service of a software application without over or under allocating storage resources.
- Total storage input/output (I/O) capacity is determined for a set of storage elements available for use by a software application located on an operating system volume of a computer system.
- the total storage I/O capacity is determined by evaluating storage I/O capacity of the storage elements at a physical device level and at a file level. A number of I/O storage operations performed on behalf of the software application are measured.
- Storage I/O utilization of the software application is determined based on the measured number of I/O storage operations as compared to the total storage I/O capacity of the set of storage elements.
- the file may be calculated to have an estimated total capacity of two hundred I/O operations per second.
- the drives may be measured to receive fifty I/O operations per second occurring at a particular point in time all with the specific I/O profile. The file could thus be calculated as experiencing a utilization level of approximately twenty-five percent of its available capacity.
- FIG. 1 is a block diagram to illustrate a first particular embodiment of a system that includes storage components and includes storage Input/Output (I/O) analysis functionality;
- I/O Input/Output
- FIG. 2 is a block diagram to illustrate a particular embodiment of selected portions of the computing system of FIG. 1 ;
- FIG. 3 is a general diagram to illustrate a particular embodiment of application programming interfaces (APIs) used to retrieve storage I/O capacity information from components of the storage system of FIG. 1 ;
- APIs application programming interfaces
- FIG. 4 is a screen view diagram to illustrate a particular embodiment of a visual display that includes storage I/O utilization information
- FIG. 5 is a flow diagram to illustrate a first particular embodiment of a method of determining storage I/O utilization associated with a software application
- FIG. 6 is a flow diagram to illustrate a second particular embodiment of a method of determining storage I/O utilization associated with a software application.
- FIG. 7 is a flow diagram to illustrate a particular embodiment of a method of calculating a total available storage I/O capacity of a file.
- a method in a particular embodiment, includes determining a total storage input/output capacity of a set of storage elements available for use by a software application located on an operating system volume of a computer system.
- the total storage input/output capacity is determined by evaluating storage input/output capacity of the storage elements at a physical device level and at a file level.
- the method includes measuring a number of input/output storage operations performed on behalf of the software application.
- the method also includes determining a storage input/output utilization of the software application based on the measured number of input/output storage operations as compared to the total storage input/output capacity of the set of storage elements.
- a method in another particular embodiment, includes relating an operating system file to a plurality of physical storage devices.
- a query is sent to application programming interfaces associated with the plurality of physical storage devices to determine one or more device properties, configuration data, and one or more usage statistics.
- Responses from the application programming interfaces are received, and a total available storage input/output capacity of the operating system file to access the plurality of physical storage devices is calculated based on the responses.
- the method includes determining storage input/output usage of the file and determining storage input/output utilization of the file based on the storage input/output usage of the file and based on the total available storage input/output capacity of the file.
- a system in another particular embodiment, includes a system model interface to retrieve system model data identifying components of a modeled system and including information describing relationships between the components. Each of the components is associated with performing an input/output storage operation of a software application.
- the system includes an application programming interface (API) module responsive to the system model interface and operable to call application programming interfaces for each of the components and to receive storage input/output capacity data for each of the components.
- API application programming interface
- the system includes capacity evaluation logic to collect the received storage input/output capacity for each of the components to determine a total available storage input/output capacity for the software application.
- the system also includes a file usage measurement interface to retrieve file storage input/output usage data associated with the software application.
- the system includes utilization logic to calculate a storage input/output utilization associated with the software application based on the retrieved file storage input/output usage data and based on the total available storage input/output capacity for the software application.
- the system also includes a display interface to present the storage input/output utilization.
- a consolidated, end-to-end view of a storage input/output (I/O) capacity and a storage I/O utilization for a particular file enables the storage I/O capacity to be managed to meet an expected level of service for the particular file without over-deploying or under-deploying storage I/O resources.
- the end-to-end view is developed by determining the capacity available by evaluating storage I/O capacity data at a physical device level and projecting the measurements through various levels of abstraction until reaching a file level.
- FIG. 1 is a block diagram of a first particular embodiment of a storage system 100 .
- FIG. 1 includes a computing system 102 , a computing system model 104 , and an input/output (I/O) system analyzer 106 .
- I/O input/output
- the computing system 102 includes a storage array 107 , application programming interfaces (APIs) 110 , an operating system 112 , a database platform 114 , and a monitored application 118 .
- the application 118 is a software application, such as a database application.
- the computing system model 104 includes components and relationships 120 , a file level 122 , an operating system volume level 124 , an operating system disk level 126 , and a physical storage level 128 .
- the operating system volume level 124 and the operating system disk level 126 may be referred to as intermediate levels because they are between the physical storage level 128 and the file level 122 .
- the I/O system analyzer 106 includes a system model interface 130 , an API module 132 , evaluation logic 134 , a file usage measurement interface 136 , utilization logic 138 , and a display interface 140 .
- the operating system 112 of the computing system 102 is configured to manage the storage array 107 .
- the storage array 107 may include multiple storage device elements in an I/O path 109 .
- the storage device elements in the I/O path 109 may include a computer host bus adapter (HBA), a storage area network (SAN) switch, an array front-end port, an array processor, an array cache, an array back-end port, and an array physical disk.
- HBA computer host bus adapter
- SAN storage area network
- the array processor is configured to respond to requests from the APIs 110 to provide information associated with the storage array 107 .
- the database platform 114 of the computing system 102 is configured to access the storage array 107 via the operating system 112 .
- the APIs 110 are configured to provide extensions to the operating system 112 so that storage commands can be seamlessly accessed from the database platform 114 .
- the APIs 110 are further configured to provide information associated with various I/O components.
- the APIs 110 may include a storage vendor API, a Storage Management Initiative (SMI) provider API, a common information management object manager (CIMOM) API, a common information management extensible markup language (CIM-XML) API, a web services for management (WS-Man) API, a system services application API, a performance counter API, other API, or any combination thereof.
- the application 118 is configured to access the storage array 107 via the APIs 110 .
- the application 118 is a database application, such as a structured query language (SQL) database file that is physically constrained to a specific number of I/O storage accesses per second.
- SQL structured query language
- the computing system model 104 is configured to model the computing system 102 .
- the components and relationships 120 of the computer system model 104 include the name of components of the computing system 102 available to perform I/O operations and the relationships between the components.
- the components and relationships 120 are modeled at the file level 122 , at the operating system volume level 124 , at the operating system disk level 126 , and at the physical storage level 128 .
- the operating system volume level 124 and the operating system disk level 126 are also referred to as intermediate levels.
- the I/O system analyzer 106 is configured to use the computing system model 104 to model I/O requests of the application 118 of the computing system 102 at the file level 122 , the intermediate levels 124 and 126 , and at the physical storage level 128 .
- the I/O system analyzer 106 is further configured to systematically evaluate all the components necessary to complete the I/O requests and provide an end-to-end view of the consolidated I/O storage utilization.
- the system model interface 130 of the I/O system analyzer 106 is operable to retrieve system model data 142 identifying components 143 of the computing system model 104 . Each of the components 143 is available to satisfy a storage I/O request of the application 118 of the computing system 102 .
- the system model interface 130 is further operable to retrieve relationship information 144 describing relationships between the components 143 .
- the system model data 142 may include a hierarchical tree data structure based on the components 143 and the relationship information 144 .
- the relationship information 144 may include a data structure that can be accessed to identify one or more alternative I/O paths or concurrent I/O paths to reach a target (e.g., one or more physical storage devices).
- the API module 132 of the I/O system analyzer 106 is responsive to the system model interface 130 and operable to call APIs 110 of the computing system 102 for each of the components 143 and to determine storage I/O capacity 146 for each of the components 143 .
- the evaluation logic 134 is operable to combine the received storage I/O capacity 146 for each of the components 143 to determine a total available storage I/O capacity 148 for the application 118 .
- the file usage measurement interface 136 is operable to retrieve file storage I/O usage data 150 associated with the application 118 .
- a workload of the application 118 is recorded. The recorded workload of the application 118 is replayed at a target component of the computing system 102 .
- the target component of the workload may include one or more physical devices of the computer system 102 (e.g., physical devices of the storage array 107 ). Alternatively, the workload may be targeted at the entire computer system 102 .
- the recorded workload is applied to a target component of a modeled system (e.g., the computing system 102 of FIG. 1 ) at an increased rate or with an increased concurrency until a threshold event is detected.
- the threshold event may indicate that a maximum storage I/O capacity has been reached. For example, a measured rate of increase of I/O latency may become non-linear or another selected capacity-indicating threshold event may be satisfied. When the rate of increase of the detected I/O latency becomes non-linear, or when another selected threshold event is satisfied, the maximum storage I/O capacity corresponding to the detected threshold event represents the total storage I/O capacity accessible to the application 118 .
- the received storage I/O capacity 146 is collected for each component in a single I/O path, in alternate I/O paths or in concurrent I/O paths
- the relationship information 144 may be accessed to identify one or more alternative I/O paths or concurrent I/O paths to reach the target component of the modeled system (e.g., the computing system 102 of FIG. 1 ).
- each of the storage device elements in the I/O path 109 may be identified based on the relationship information 144 .
- the utilization logic 138 of the I/O system analyzer 106 is operable to calculate a storage I/O utilization 152 associated with the application 118 based on the retrieved file storage I/O usage data 150 and based on the total available storage I/O capacity 148 for the application 118 .
- the display interface 140 is operable to present the storage I/O utilization 152 at the display 154 .
- the application 118 of the computing system 102 is mapped to the set of storage elements 109 , which may include a plurality of physical storage devices.
- the components and relationships 120 of the computing system model 104 are a model of the components of the computing system 102 and their relationships at the file level 122 , the operating system volume level 124 , the operating system disk level 126 , and at the physical storage level 128 . Each modeled component is available to satisfy a storage I/O request of the application 118 .
- the computing system model 104 includes the set of storage elements 109 .
- the set of storage elements 109 may include a set of physical storage devices.
- the computing system model 104 includes models of storage I/O requests of the application 118 that identify components available to satisfy the storage I/O requests at the file level 122 , at the operating system volume level 124 , at the operating system disk level 126 , and at the physical storage level 128 .
- the system model interface 130 of the I/O system analyzer 106 receives system model data 142 from the computing system model 104 .
- the API module 132 sends a query to the APIs 110 of the computing system 102 associated with the set of storage elements 109 .
- the evaluation logic 134 receives responses from the APIs 110 that include storage I/O capacity 146 for each of the components 143 .
- the evaluation logic 134 determines the total available storage I/O capacity 148 of the application 118 within the set of storage elements 109 based on the responses from the APIs 110 .
- the total available storage I/O capacity 148 is determined based on a given workload and based on a storage I/O capacity of those components of the computing system 102 identified as available for use by the application 118 .
- the storage I/O capacity available to support storage I/O requests at each of the components 143 is determined and then the total available storage I/O capacity 148 is determined by consolidating the storage I/O capacity at each of the components 143 .
- the total storage I/O capacity 148 is determined by evaluating storage I/O capacity of the storage device elements in an I/O path 109 at the physical device level 128 , at the volume level 124 , and at the file level 122 .
- the number of I/O storage operations performed on behalf of the application 118 is determined, and the storage I/O utilization 152 of the application 118 is determined based on the measured number of I/O storage operations as compared to the total available storage I/O capacity 148 of the set of storage elements 109 .
- the storage I/O utilization 152 may be determined as a ratio of the measured number of I/O storage operations to the total available storage I/O capacity 148 .
- the display interface 140 presents the storage I/O utilization 152 of the application 118 .
- the file usage measurement interface 136 of the I/O system analyzer 106 determines storage I/O usage of the application 118 by querying the operating system 112 or the database platform 114 via the file manager 116 and receiving the file storage I/O usage data 150 .
- the utilization logic 138 determines the storage I/O utilization 152 of the application 118 based on the retrieved file storage I/O usage data 150 of the application 118 and based on the total available storage I/O capacity 148 for the application 118 .
- the I/O system analyzer 106 may determine the file storage I/O usage data 150 of the application 118 periodically and update the storage I/O utilization 152 of the application 118 displayed at the display 154 periodically.
- the I/O system analyzer 106 may be used to detect a usage of the application 118 that approaches a storage I/O resource limit before the usage exceeds the storage I/O resource limit. Various actions may be performed to avoid exceeding the storage I/O resource limit, such as allocating additional storage resources to the application 118 or terminating execution of the application 118 .
- FIG. 2 is a block diagram of a particular embodiment of selected portions of the computing system 102 of FIG. 1 .
- FIG. 2 includes a first computer 202 , a second computer 204 , and a storage array 206 .
- the storage array 206 is also known as a set of storage elements (e.g., the set of storage elements 109 in FIG. 1 ).
- the first computer 202 and the second computer 204 include one or more operating system volumes (e.g., Windows volumes) and one or more operating system disks (e.g., Windows disks).
- the first computer 202 includes a first Windows disk 212 partitioned into a first Windows volume 208 and a second Windows volume 210 .
- the second computer 204 includes a third Windows volume 214 spanning a second Windows disk 216 and a third Windows disk 218 .
- the storage array 206 is the storage array 107 in FIG. 1 .
- the storage array 206 includes a set of storage device elements 220 in a particular representative I/O path.
- the set of storage device elements 220 in the I/O path may include a computer host bus adapter (HBA), a storage area network (SAN) switch, an array front-end port, an array processor, an array cache, an array back-end port, and an array physical disk.
- HBA computer host bus adapter
- SAN storage area network
- the set of storage device elements 220 (e.g., physical storage elements) includes a first logical unit number (LUN) 222 , a second LUN 224 , and a third LUN 226 .
- the physical storage elements 220 are also referred to as a storage pool.
- a LUN is a logical component that maps one or more physical storage devices to logical storage space for use by the Windows disks 212 , 216 , and 218 .
- the first LUN 222 maps a first physical storage device 228 , a second physical storage device 230 , and a third physical storage device 232 to logical storage space for the first Windows disk 212 of the first computer 202 .
- the second LUN 224 maps the third physical storage device 232 , a fourth physical storage device 234 , and a fifth physical storage device 236 to logical storage space for the second Windows disk 216 of the second computer 204 .
- the third LUN 226 maps a sixth physical storage device 238 , a seventh physical storage device 240 , and an eighth physical storage device 242 to the third Windows disk 218 of the second computer 204 .
- the set of storage elements 220 includes the separate physical storage devices 228 , 230 , 232 , 234 , 236 , 238 , 240 , and 242 . At least some of the physical storage devices 228 , 230 , 232 , 234 , 236 , 238 , 240 , and 242 are associated with at least one of the LUNs 222 , 224 , and 226 . At least one of the LUNs 222 , 224 , and 226 is associated with at least one of the Windows disks 212 , 216 , and 218 and each of the Windows disks 212 , 216 , and 218 is associated with at least one of the Windows volumes 208 , 210 , and 214 .
- each of the Windows disks has multiple volumes.
- the first Windows disk 212 has the first Windows volume 208 and the second Windows volume 210 .
- each of the Windows volumes has multiple Windows disks.
- the third Windows volume 214 has the second Windows disk 216 and the third Windows disk 218 .
- the set of storage elements 220 includes a storage array 206 comprising multiple physical storage devices 228 , 230 , 232 , 234 , 236 , 238 , 240 , and 242 and multiple LUNs 222 , 224 , and 226 .
- the third physical storage device 232 is shared by the first LUN 222 and the second LUN 224 .
- the I/O system analyzer 106 determines characteristics of each storage element at a physical storage device level, including the physical disks or portions of shared physical disks of the set of storage elements 220 of the storage array 206 .
- the I/O system analyzer 106 collects metrics of each storage element at a volume level, such as at the level of Windows volumes 208 , 210 , and 214 .
- the I/O system analyzer 106 determines the total available storage I/O capacity 148 based on the metrics and based on the characteristics.
- a method of determining I/O capacity uses synthetic I/O load generation.
- the method includes measuring an I/O workload profile of a file using storage instrumentation APIs.
- the measurement may run for a time period that is long enough to capture periodic behavior in the I/O workload.
- properties of an I/O workload profile may include an I/O pattern (e.g., composition of random and sequential operations), an I/O type (e.g., a composition of read and write operations), an I/O size (e.g., an average number of bytes per I/O), or an I/O rate (e.g., an average number of input/output operations per unit time).
- the storage instrumentation APIs may include Microsoft SQL Server Dynamic Management Views (DMVs), or Windows performance counters, among other alternatives.
- a temporary file is created on the same logical volume as the file under analysis.
- the temporary file is used as the target for generating synthetic I/O workload.
- a synthetic I/O workload generator is configured to reflect the I/O workload profile.
- the synthetic I/O workload generator is run against the temporary file using the I/O workload profile.
- the I/O rate is incrementally increased, and an I/O latency is recorded with each increase in the I/O rate. When a rate of increase of I/O latency becomes non-linear, the I/O rate is no longer increased.
- the maximum I/O rate (e.g., the I/O rate recorded when the rate of increase of the detected I/O latency becomes non-linear) represents the total available storage I/O capacity of the file under analysis.
- other thresholds and measurements of detectable effects on capacity may also be used.
- a method of determining I/O capacity uses individual storage device analysis. Statistics are measured over a period of time to generate a representation of utilization of devices in general. Storage instrumentation APIs are used to measure I/O utilization of each storage device element in the I/O path of the file under analysis.
- storage device elements in the I/O path may include a computer host bus adapter (HBA), a storage area network (SAN) switch, an array front-end port, an array processor, an array cache, an array back-end port, and an array physical disk.
- HBA computer host bus adapter
- SAN storage area network
- each storage device element in the I/O path is obtained by using a combination of the storage instrumentation APIs.
- the measurement is run for a long enough period of time to capture periodic behavior of I/O statistics.
- the I/O rate of the database file under analysis is measured over the same time interval as the measurement of the I/O utilization of each of the storage elements in the I/O path.
- a time-average value of the I/O statistics is determined.
- the storage device element in the I/O path with the minimum storage I/O capacity is identified, representing a potential bottleneck in the I/O path.
- the identified storage device element that has the minimum storage I/O capacity represents the total available storage I/O capacity of the file under analysis.
- An operating system level I/O request may traverse one or more alternative I/O paths to identify a potential bottleneck in each of the one or more alternative I/O paths.
- a capacity measurement may be obtained for each alternative path. For example, if there are two alternative paths to a target, a first capacity measurement is obtained for a first alternative path, and a second capacity measurement is obtained for a second alternative path.
- the first capacity measurement for the first alternative path corresponds to a first storage device in the first alternative path that has a first minimum storage I/O capacity.
- the second capacity measurement for the second alternative path corresponds to a second storage device in the second alternative path that has a second minimum storage I/O capacity.
- the total available storage I/O capacity may be selected as the minimum storage I/O capacity of the original I/O path or the minimum storage I/O capacity of any of the alternative I/O paths (e.g., the first minimum storage I/O capacity of the first alternative path or the second minimum storage I/O capacity of the second alternative path).
- the selection may be made by the system based on particular system or application design considerations. A conservative approach may select the smallest minimum storage I/O capacity of all paths to protect the most resource constrained path. This selection protects against system operation performance issues where the system uses the most resource constrained path.
- the system selects the path with the most resources available (e.g., a “smart system”) and in this case, the total available storage I/O capacity may be the capacity of the path with the largest available capacity.
- the system can determine a total available storage I/O capacity where there are multiple available paths.
- concurrent I/O path scenario when the operating system level I/O request traverses concurrent I/O paths to reach the target, multiple measurements may be obtained for each concurrent path. For example, if there are two concurrent paths to the target, a first capacity measurement is obtained for a first concurrent path, and a second capacity measurement is obtained for a second concurrent path. In a particular embodiment, the first capacity measurement and the second capacity measurement are then summed to determine the total storage capacity available.
- the I/O system analyzer 106 can accurately determine the storage I/O capacity and storage I/O usage for a file through the various layers of abstraction.
- FIG. 3 is a general diagram that illustrates a particular embodiment of application programming interfaces (APIs) 110 used to retrieve storage I/O capacity information from components of the storage system of FIG. 1 .
- the APIs 110 include a performance counters API 302 , a system services API 304 , a web services for management (WS-Man) API 306 , a common information management extensible markup language (CIM-XML) API 308 , a common information management object manager (CIMOM) API 310 , a Storage Management Initiative (SMI) provider API 312 , a first vendor API 314 , a second vendor API 316 , and a third vendor API 318 .
- Each of the APIs 302 - 318 passes storage performance and configuration information from one or more storage devices associated with each respective API.
- FIG. 4 is a screen view diagram to illustrate a particular embodiment of a visual display that includes storage I/O utilization information.
- FIG. 4 includes a capacity viewpoints tab 402 , a storage explorer column 404 , a total space utilization column 406 , an allocated space utilization column 408 , and a total I/O utilization column 410 .
- the capacity viewpoints tab 402 is selectable to visually display different views of I/O utilization.
- the capacity viewpoints 402 may display I/O utilization at a physical storage level, at an operating system disk level (e.g., Windows disk level), at a volume level (e.g., Windows volume level, at a file level, or at a database level.
- the storage explorer column 404 may be used to navigate the various storage levels to enable the display of storage I/O utilization for each respective level.
- the total space utilization column 406 displays a percentage of the total storage space utilized at each level for each storage device, such as a disk, an operating system storage volume, or a logical unit number.
- the allocated space utilization column 408 displays a percentage of the allocated space utilized by a particular file when it is limited from using the total space available for a volume by some other software based allocation restriction.
- the total I/O utilization 410 column displays the storage I/O utilization such as determined for the system 100 , for a volume, or for a database.
- a system analyzer such as the I/O system analyzer 106 , as shown in FIG. 1 , enables a system administrator to detect usage of a software application that approaches a storage I/O resource limit before the file usage exceeds the storage I/O resource limit. Alternatively, a deviation from a predetermined storage I/O utilization range may be automatically determined in order to alert the system administrator to take remedial action.
- a system administrator may stop execution of a particular software application or allocate additional storage elements to the file to prevent the storage I/O resource limit from being exceeded.
- An example file that may be monitored is the illustrated file “RAD.mdf” 412 in FIG. 4 . If the storage I/O utilization of the file 412 exceeded a threshold, such as 90%, then a database administrator could take action proactively to avoid a storage I/O utilization event that could affect performance.
- FIG. 5 is a flow diagram that illustrates a first particular embodiment of a method of determining storage I/O utilization associated with a software application.
- a total storage I/O capacity of a set of storage elements available for use by a software application of a computer system is determined.
- the software application is located on an operating system volume of the computer system.
- the total storage I/O capacity is determined by evaluating storage I/O capacity of the storage elements at a physical device level and at a file level as previously described.
- a system model may be used to identify the storage I/O capacity for each storage element.
- the set of storage elements includes a storage array comprising a plurality of physical storage devices and a plurality of logical unit numbers, where at least one of the physical storage devices is shared by at least two of the logical unit numbers.
- the set of storage elements includes a plurality of physical storage devices, and the method further includes identifying one of the storage elements as a potential bottleneck in an I/O path.
- an operating system level I/O request traverses one or more alternative paths to reach a target, where the target includes one or more physical storage devices.
- evaluating storage I/O capacity includes determining characteristics of each storage element at a physical storage device level.
- Each storage element includes physical storage devices or portions of shared physical storage devices. Metrics are collected for each storage element at a file level. A total storage I/O capacity is determined based on the characteristics and based on the metrics.
- a number of I/O storage operations performed on behalf of the software application is measured.
- a storage I/O utilization of the software application is determined based on the measured number of I/O storage operations as compared to the total storage I/O capacity of the set of storage elements.
- the method further comprises determining a resource governed utilization of the storage I/O capacity based on historical usage of the storage I/O capacity as compared to a governed storage I/O capacity available and a non-governed storage I/O capacity available.
- the resource governed utilization may include a storage I/O capacity utilization after taking into account software based restrictions on storage I/O capacity available to the software application.
- the software application may include an operating system file residing on an operating system volume, where the operating system volume is constrained to a predetermined number of input/output storage accesses per second.
- the resource governed utilization of the storage I/O capacity is determined based on the software based restrictions.
- the method includes determining the total storage I/O utilization as a ratio of the measured number of I/O storage operations and the total storage I/O capacity, as shown at 508 . Proceeding to 510 , the storage I/O utilization of the software application is displayed. Based on the storage I/O utilization, a usage of the software application that approaches a storage I/O resource limit may be detected before the usage exceeds the storage I/O resource limit, as shown at 512 . The method ends at 514 .
- FIG. 6 is a flow diagram to illustrate a second particular embodiment of a method of determining storage I/O utilization associated with an application, such as a software application.
- the method includes relating an operating system file to a plurality of physical storage devices, at 602 .
- a query is sent to application programming interfaces (APIs) associated with the plurality of physical storage devices to determine one or more device properties, configuration data, and one or more usage statistics, at 604 .
- Responses are received from the APIs, at 606 .
- a total available storage I/O capacity of the operating system file to access the plurality of physical storage devices is calculated based on the responses, at 608 .
- Storage I/O usage of the operating system file is determined, at 610 .
- storage I/O utilization of the operating system file is determined based on the storage I/O usage of the operating system file and based on the total available storage I/O capacity. In a particular embodiment, determining the storage I/O utilization includes determining a percentage of the total available storage I/O capacity used by the operating system file. The method ends at 614 .
- the method further comprises modeling components of a computing system that include the plurality of physical storage devices, modeling relationships between the components, and determining the total available storage I/O capacity based on a given workload and based on a storage I/O capacity of those components of the computing system identified as available for use by the operating system file.
- FIG. 7 is a flow diagram to illustrate a particular embodiment of a method of calculating a total available storage I/O capacity of an operating system file.
- the flow diagram of FIG. 7 illustrates the calculation shown at 608 of FIG. 6 .
- the method includes computing an I/O capacity of each of the multiple storage device elements in an I/O path of the operating system file, at 702 .
- the method includes identifying the storage device element that has a minimum storage I/O capacity, at 704 .
- the total available storage I/O capacity of the operating system file is determined, at 706 .
- the storage device element identified as having the minimum storage I/O capacity represents the total available storage I/O capacity of the operating system file, and thereby determining the total available storage I/O capacity of the file, as shown at 706 .
- the method further includes measuring an I/O utilization and an I/O rate of each storage device element in the I/O path of the operating system file.
- the I/O capacity of each of the storage device elements in the I/O path is computed based on the measured I/O utilization and the measured I/O rate, at 710 .
- the method ends.
- the method further comprises periodically determining the storage I/O usage of the operating system file and updating the storage I/O utilization of the operating system file periodically.
- a software module may reside in computer readable media, such as random access memory (RAM), flash memory, read only memory (ROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
- An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium.
- the storage medium may be integral to the processor or the processor and the storage medium may reside as discrete components in a computing device or computer system.
Abstract
Description
- Software applications use hardware resources such as storage devices which have limited capacities. At the same time, applications often have specific service level expectations which are reliant in part on the resource capacity of the hardware. While some types of hardware resource usage and capacity are easy to measure, determining the storage I/O usage and capacity available to a specific file at a specific point in time is difficult, especially when storage system resources are concurrently shared among several computers and software applications.
- For example, the use of storage area networks (SANs) and network attached storage (NAS) that may be shared across multiple computers results in many layers of abstraction between a physical storage device and a logical file. A file may have access to one of several operating system volumes on an operating system disk and each operating system disk may be associated with a storage device logical unit number (LUN) that spans multiple physical storage devices in a storage group within a storage array. One or more of these physical storage devices may furthermore be accessed by an entirely different set of applications through a different computer, logical file, and logical unit number (LUN). It may thus not be possible to view a consolidated end-to-end view of the storage I/O capacity available when accessing the file. Without such an end-to-end view, it is difficult to plan, administer, or automatically adjust the storage I/O capacity to meet the expected level of service of a software application without over or under allocating storage resources.
- Total storage input/output (I/O) capacity is determined for a set of storage elements available for use by a software application located on an operating system volume of a computer system. The total storage I/O capacity is determined by evaluating storage I/O capacity of the storage elements at a physical device level and at a file level. A number of I/O storage operations performed on behalf of the software application are measured. Storage I/O utilization of the software application is determined based on the measured number of I/O storage operations as compared to the total storage I/O capacity of the set of storage elements. As an example, if a file is mapped to a LUN which in turn is striped across two physical storage devices which each can satisfy one hundred I/O operations per second for a specific software application I/O profile, the file may be calculated to have an estimated total capacity of two hundred I/O operations per second. The drives may be measured to receive fifty I/O operations per second occurring at a particular point in time all with the specific I/O profile. The file could thus be calculated as experiencing a utilization level of approximately twenty-five percent of its available capacity.
- This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
-
FIG. 1 is a block diagram to illustrate a first particular embodiment of a system that includes storage components and includes storage Input/Output (I/O) analysis functionality; -
FIG. 2 is a block diagram to illustrate a particular embodiment of selected portions of the computing system ofFIG. 1 ; -
FIG. 3 is a general diagram to illustrate a particular embodiment of application programming interfaces (APIs) used to retrieve storage I/O capacity information from components of the storage system ofFIG. 1 ; -
FIG. 4 is a screen view diagram to illustrate a particular embodiment of a visual display that includes storage I/O utilization information; -
FIG. 5 is a flow diagram to illustrate a first particular embodiment of a method of determining storage I/O utilization associated with a software application; -
FIG. 6 is a flow diagram to illustrate a second particular embodiment of a method of determining storage I/O utilization associated with a software application; and -
FIG. 7 is a flow diagram to illustrate a particular embodiment of a method of calculating a total available storage I/O capacity of a file. - In a particular embodiment, a method is disclosed that includes determining a total storage input/output capacity of a set of storage elements available for use by a software application located on an operating system volume of a computer system. The total storage input/output capacity is determined by evaluating storage input/output capacity of the storage elements at a physical device level and at a file level. The method includes measuring a number of input/output storage operations performed on behalf of the software application. The method also includes determining a storage input/output utilization of the software application based on the measured number of input/output storage operations as compared to the total storage input/output capacity of the set of storage elements.
- In another particular embodiment, a method is disclosed that includes relating an operating system file to a plurality of physical storage devices. A query is sent to application programming interfaces associated with the plurality of physical storage devices to determine one or more device properties, configuration data, and one or more usage statistics. Responses from the application programming interfaces are received, and a total available storage input/output capacity of the operating system file to access the plurality of physical storage devices is calculated based on the responses. The method includes determining storage input/output usage of the file and determining storage input/output utilization of the file based on the storage input/output usage of the file and based on the total available storage input/output capacity of the file.
- In another particular embodiment, a system is disclosed that includes a system model interface to retrieve system model data identifying components of a modeled system and including information describing relationships between the components. Each of the components is associated with performing an input/output storage operation of a software application. The system includes an application programming interface (API) module responsive to the system model interface and operable to call application programming interfaces for each of the components and to receive storage input/output capacity data for each of the components. The system includes capacity evaluation logic to collect the received storage input/output capacity for each of the components to determine a total available storage input/output capacity for the software application. The system also includes a file usage measurement interface to retrieve file storage input/output usage data associated with the software application. The system includes utilization logic to calculate a storage input/output utilization associated with the software application based on the retrieved file storage input/output usage data and based on the total available storage input/output capacity for the software application. The system also includes a display interface to present the storage input/output utilization.
- A consolidated, end-to-end view of a storage input/output (I/O) capacity and a storage I/O utilization for a particular file enables the storage I/O capacity to be managed to meet an expected level of service for the particular file without over-deploying or under-deploying storage I/O resources. The end-to-end view is developed by determining the capacity available by evaluating storage I/O capacity data at a physical device level and projecting the measurements through various levels of abstraction until reaching a file level.
-
FIG. 1 is a block diagram of a first particular embodiment of astorage system 100.FIG. 1 includes acomputing system 102, acomputing system model 104, and an input/output (I/O)system analyzer 106. - The
computing system 102 includes astorage array 107, application programming interfaces (APIs) 110, anoperating system 112, adatabase platform 114, and a monitoredapplication 118. In a particular embodiment, theapplication 118 is a software application, such as a database application. Thecomputing system model 104 includes components andrelationships 120, afile level 122, an operatingsystem volume level 124, an operatingsystem disk level 126, and aphysical storage level 128. The operatingsystem volume level 124 and the operatingsystem disk level 126 may be referred to as intermediate levels because they are between thephysical storage level 128 and thefile level 122. The I/O system analyzer 106 includes asystem model interface 130, anAPI module 132,evaluation logic 134, a fileusage measurement interface 136,utilization logic 138, and adisplay interface 140. - The
operating system 112 of thecomputing system 102 is configured to manage thestorage array 107. Thestorage array 107 may include multiple storage device elements in an I/O path 109. The storage device elements in the I/O path 109 may include a computer host bus adapter (HBA), a storage area network (SAN) switch, an array front-end port, an array processor, an array cache, an array back-end port, and an array physical disk. There may be more than one particular type of storage element in the I/O path 109. For example, there may be more than one computer HBA, array front-end port, array processor, array back-end port or array physical disk. The array processor is configured to respond to requests from theAPIs 110 to provide information associated with thestorage array 107. - The
database platform 114 of thecomputing system 102 is configured to access thestorage array 107 via theoperating system 112. TheAPIs 110 are configured to provide extensions to theoperating system 112 so that storage commands can be seamlessly accessed from thedatabase platform 114. TheAPIs 110 are further configured to provide information associated with various I/O components. TheAPIs 110 may include a storage vendor API, a Storage Management Initiative (SMI) provider API, a common information management object manager (CIMOM) API, a common information management extensible markup language (CIM-XML) API, a web services for management (WS-Man) API, a system services application API, a performance counter API, other API, or any combination thereof. Theapplication 118 is configured to access thestorage array 107 via theAPIs 110. In a particular embodiment, theapplication 118 is a database application, such as a structured query language (SQL) database file that is physically constrained to a specific number of I/O storage accesses per second. - The
computing system model 104 is configured to model thecomputing system 102. The components andrelationships 120 of thecomputer system model 104 include the name of components of thecomputing system 102 available to perform I/O operations and the relationships between the components. The components andrelationships 120 are modeled at thefile level 122, at the operatingsystem volume level 124, at the operatingsystem disk level 126, and at thephysical storage level 128. The operatingsystem volume level 124 and the operatingsystem disk level 126 are also referred to as intermediate levels. - The I/
O system analyzer 106 is configured to use thecomputing system model 104 to model I/O requests of theapplication 118 of thecomputing system 102 at thefile level 122, theintermediate levels physical storage level 128. The I/O system analyzer 106 is further configured to systematically evaluate all the components necessary to complete the I/O requests and provide an end-to-end view of the consolidated I/O storage utilization. - The
system model interface 130 of the I/O system analyzer 106 is operable to retrievesystem model data 142 identifyingcomponents 143 of thecomputing system model 104. Each of thecomponents 143 is available to satisfy a storage I/O request of theapplication 118 of thecomputing system 102. Thesystem model interface 130 is further operable to retrieverelationship information 144 describing relationships between thecomponents 143. Thesystem model data 142 may include a hierarchical tree data structure based on thecomponents 143 and therelationship information 144. Therelationship information 144 may include a data structure that can be accessed to identify one or more alternative I/O paths or concurrent I/O paths to reach a target (e.g., one or more physical storage devices). - The
API module 132 of the I/O system analyzer 106 is responsive to thesystem model interface 130 and operable to callAPIs 110 of thecomputing system 102 for each of thecomponents 143 and to determine storage I/O capacity 146 for each of thecomponents 143. Theevaluation logic 134 is operable to combine the received storage I/O capacity 146 for each of thecomponents 143 to determine a total available storage I/O capacity 148 for theapplication 118. The fileusage measurement interface 136 is operable to retrieve file storage I/O usage data 150 associated with theapplication 118. In a particular illustrative embodiment, a workload of theapplication 118 is recorded. The recorded workload of theapplication 118 is replayed at a target component of thecomputing system 102. The target component of the workload may include one or more physical devices of the computer system 102 (e.g., physical devices of the storage array 107). Alternatively, the workload may be targeted at theentire computer system 102. The recorded workload is applied to a target component of a modeled system (e.g., thecomputing system 102 ofFIG. 1 ) at an increased rate or with an increased concurrency until a threshold event is detected. The threshold event may indicate that a maximum storage I/O capacity has been reached. For example, a measured rate of increase of I/O latency may become non-linear or another selected capacity-indicating threshold event may be satisfied. When the rate of increase of the detected I/O latency becomes non-linear, or when another selected threshold event is satisfied, the maximum storage I/O capacity corresponding to the detected threshold event represents the total storage I/O capacity accessible to theapplication 118. - In a particular embodiment, the received storage I/
O capacity 146 is collected for each component in a single I/O path, in alternate I/O paths or in concurrent I/O paths Therelationship information 144 may be accessed to identify one or more alternative I/O paths or concurrent I/O paths to reach the target component of the modeled system (e.g., thecomputing system 102 ofFIG. 1 ). For example, each of the storage device elements in the I/O path 109 may be identified based on therelationship information 144. - The
utilization logic 138 of the I/O system analyzer 106 is operable to calculate a storage I/O utilization 152 associated with theapplication 118 based on the retrieved file storage I/O usage data 150 and based on the total available storage I/O capacity 148 for theapplication 118. Thedisplay interface 140 is operable to present the storage I/O utilization 152 at thedisplay 154. - In operation, the
application 118 of thecomputing system 102 is mapped to the set ofstorage elements 109, which may include a plurality of physical storage devices. The components andrelationships 120 of thecomputing system model 104 are a model of the components of thecomputing system 102 and their relationships at thefile level 122, the operatingsystem volume level 124, the operatingsystem disk level 126, and at thephysical storage level 128. Each modeled component is available to satisfy a storage I/O request of theapplication 118. Thecomputing system model 104 includes the set ofstorage elements 109. The set ofstorage elements 109 may include a set of physical storage devices. Thecomputing system model 104 includes models of storage I/O requests of theapplication 118 that identify components available to satisfy the storage I/O requests at thefile level 122, at the operatingsystem volume level 124, at the operatingsystem disk level 126, and at thephysical storage level 128. - The
system model interface 130 of the I/O system analyzer 106 receivessystem model data 142 from thecomputing system model 104. TheAPI module 132 sends a query to theAPIs 110 of thecomputing system 102 associated with the set ofstorage elements 109. Theevaluation logic 134 receives responses from theAPIs 110 that include storage I/O capacity 146 for each of thecomponents 143. Theevaluation logic 134 determines the total available storage I/O capacity 148 of theapplication 118 within the set ofstorage elements 109 based on the responses from theAPIs 110. - In a particular embodiment, the total available storage I/
O capacity 148 is determined based on a given workload and based on a storage I/O capacity of those components of thecomputing system 102 identified as available for use by theapplication 118. In another particular embodiment, the storage I/O capacity available to support storage I/O requests at each of thecomponents 143 is determined and then the total available storage I/O capacity 148 is determined by consolidating the storage I/O capacity at each of thecomponents 143. - Thus, the total storage I/
O capacity 148 is determined by evaluating storage I/O capacity of the storage device elements in an I/O path 109 at thephysical device level 128, at thevolume level 124, and at thefile level 122. The number of I/O storage operations performed on behalf of theapplication 118 is determined, and the storage I/O utilization 152 of theapplication 118 is determined based on the measured number of I/O storage operations as compared to the total available storage I/O capacity 148 of the set ofstorage elements 109. The storage I/O utilization 152 may be determined as a ratio of the measured number of I/O storage operations to the total available storage I/O capacity 148. Thedisplay interface 140 presents the storage I/O utilization 152 of theapplication 118. - The file
usage measurement interface 136 of the I/O system analyzer 106 determines storage I/O usage of theapplication 118 by querying theoperating system 112 or thedatabase platform 114 via the file manager 116 and receiving the file storage I/O usage data 150. Theutilization logic 138 determines the storage I/O utilization 152 of theapplication 118 based on the retrieved file storage I/O usage data 150 of theapplication 118 and based on the total available storage I/O capacity 148 for theapplication 118. The I/O system analyzer 106 may determine the file storage I/O usage data 150 of theapplication 118 periodically and update the storage I/O utilization 152 of theapplication 118 displayed at thedisplay 154 periodically. - By periodically displaying the storage I/
O utilization 152, the I/O system analyzer 106 may be used to detect a usage of theapplication 118 that approaches a storage I/O resource limit before the usage exceeds the storage I/O resource limit. Various actions may be performed to avoid exceeding the storage I/O resource limit, such as allocating additional storage resources to theapplication 118 or terminating execution of theapplication 118. -
FIG. 2 is a block diagram of a particular embodiment of selected portions of thecomputing system 102 ofFIG. 1 .FIG. 2 includes afirst computer 202, asecond computer 204, and astorage array 206. Thestorage array 206 is also known as a set of storage elements (e.g., the set ofstorage elements 109 inFIG. 1 ). Thefirst computer 202 and thesecond computer 204 include one or more operating system volumes (e.g., Windows volumes) and one or more operating system disks (e.g., Windows disks). Thefirst computer 202 includes afirst Windows disk 212 partitioned into afirst Windows volume 208 and asecond Windows volume 210. Thesecond computer 204 includes athird Windows volume 214 spanning asecond Windows disk 216 and athird Windows disk 218. In a particular embodiment, thestorage array 206 is thestorage array 107 inFIG. 1 . - The
storage array 206 includes a set ofstorage device elements 220 in a particular representative I/O path. The set ofstorage device elements 220 in the I/O path may include a computer host bus adapter (HBA), a storage area network (SAN) switch, an array front-end port, an array processor, an array cache, an array back-end port, and an array physical disk. There may be more than one particular type of storage device element in the I/O path. For example, there may be more than one computer HBA, array front-end port, array processor, array back-end port or array physical disks. In the embodiment shown inFIG. 2 , the set of storage device elements 220 (e.g., physical storage elements) includes a first logical unit number (LUN) 222, asecond LUN 224, and athird LUN 226. Thephysical storage elements 220 are also referred to as a storage pool. A LUN is a logical component that maps one or more physical storage devices to logical storage space for use by theWindows disks first LUN 222 maps a firstphysical storage device 228, a secondphysical storage device 230, and a thirdphysical storage device 232 to logical storage space for thefirst Windows disk 212 of thefirst computer 202. Thesecond LUN 224 maps the thirdphysical storage device 232, a fourthphysical storage device 234, and a fifthphysical storage device 236 to logical storage space for thesecond Windows disk 216 of thesecond computer 204. Thethird LUN 226 maps a sixthphysical storage device 238, a seventhphysical storage device 240, and an eighthphysical storage device 242 to thethird Windows disk 218 of thesecond computer 204. - The set of
storage elements 220 includes the separatephysical storage devices physical storage devices LUNs LUNs Windows disks Windows disks Windows volumes FIG. 2 , thefirst Windows disk 212 has thefirst Windows volume 208 and thesecond Windows volume 210. In a particular embodiment, each of the Windows volumes has multiple Windows disks. For example, inFIG. 2 , thethird Windows volume 214 has thesecond Windows disk 216 and thethird Windows disk 218. - In the particular embodiment shown in
FIG. 2 , the set ofstorage elements 220 includes astorage array 206 comprising multiplephysical storage devices multiple LUNs physical storage device 232 is shared by thefirst LUN 222 and thesecond LUN 224. - When calculating the storage I/
O utilization 152 ofFIG. 1 , the I/O system analyzer 106 determines characteristics of each storage element at a physical storage device level, including the physical disks or portions of shared physical disks of the set ofstorage elements 220 of thestorage array 206. The I/O system analyzer 106 collects metrics of each storage element at a volume level, such as at the level ofWindows volumes O system analyzer 106 determines the total available storage I/O capacity 148 based on the metrics and based on the characteristics. - In a particular embodiment, a method of determining I/O capacity uses synthetic I/O load generation. The method includes measuring an I/O workload profile of a file using storage instrumentation APIs. The measurement may run for a time period that is long enough to capture periodic behavior in the I/O workload. For example, properties of an I/O workload profile may include an I/O pattern (e.g., composition of random and sequential operations), an I/O type (e.g., a composition of read and write operations), an I/O size (e.g., an average number of bytes per I/O), or an I/O rate (e.g., an average number of input/output operations per unit time). The storage instrumentation APIs may include Microsoft SQL Server Dynamic Management Views (DMVs), or Windows performance counters, among other alternatives.
- A temporary file is created on the same logical volume as the file under analysis. The temporary file is used as the target for generating synthetic I/O workload. A synthetic I/O workload generator is configured to reflect the I/O workload profile. The synthetic I/O workload generator is run against the temporary file using the I/O workload profile. The I/O rate is incrementally increased, and an I/O latency is recorded with each increase in the I/O rate. When a rate of increase of I/O latency becomes non-linear, the I/O rate is no longer increased. The maximum I/O rate (e.g., the I/O rate recorded when the rate of increase of the detected I/O latency becomes non-linear) represents the total available storage I/O capacity of the file under analysis. In addition to I/O latency, other thresholds and measurements of detectable effects on capacity may also be used.
- In another particular embodiment, a method of determining I/O capacity uses individual storage device analysis. Statistics are measured over a period of time to generate a representation of utilization of devices in general. Storage instrumentation APIs are used to measure I/O utilization of each storage device element in the I/O path of the file under analysis. For example, storage device elements in the I/O path may include a computer host bus adapter (HBA), a storage area network (SAN) switch, an array front-end port, an array processor, an array cache, an array back-end port, and an array physical disk. There may be more than one particular type of storage element in the I/O path. For example, there may be more than one computer HBA, array front-end port, array processor, array back-end port or array physical disks. The utilization of each storage device element in the I/O path is obtained by using a combination of the storage instrumentation APIs. The measurement is run for a long enough period of time to capture periodic behavior of I/O statistics. The I/O rate of the database file under analysis is measured over the same time interval as the measurement of the I/O utilization of each of the storage elements in the I/O path. Optionally, a time-average value of the I/O statistics is determined. The storage device element in the I/O path with the minimum storage I/O capacity is identified, representing a potential bottleneck in the I/O path. The identified storage device element that has the minimum storage I/O capacity represents the total available storage I/O capacity of the file under analysis.
- An operating system level I/O request may traverse one or more alternative I/O paths to identify a potential bottleneck in each of the one or more alternative I/O paths. When an operating system level I/O request traverses each alternative I/O path to reach a target, a capacity measurement may be obtained for each alternative path. For example, if there are two alternative paths to a target, a first capacity measurement is obtained for a first alternative path, and a second capacity measurement is obtained for a second alternative path. The first capacity measurement for the first alternative path corresponds to a first storage device in the first alternative path that has a first minimum storage I/O capacity. Similarly, the second capacity measurement for the second alternative path corresponds to a second storage device in the second alternative path that has a second minimum storage I/O capacity. Since multiple I/O paths are available, the total available storage I/O capacity may be selected as the minimum storage I/O capacity of the original I/O path or the minimum storage I/O capacity of any of the alternative I/O paths (e.g., the first minimum storage I/O capacity of the first alternative path or the second minimum storage I/O capacity of the second alternative path). The selection may be made by the system based on particular system or application design considerations. A conservative approach may select the smallest minimum storage I/O capacity of all paths to protect the most resource constrained path. This selection protects against system operation performance issues where the system uses the most resource constrained path. Alternatively, an assumption may be made that the system selects the path with the most resources available (e.g., a “smart system”) and in this case, the total available storage I/O capacity may be the capacity of the path with the largest available capacity. Thus, the system can determine a total available storage I/O capacity where there are multiple available paths.
- In the concurrent I/O path scenario, when the operating system level I/O request traverses concurrent I/O paths to reach the target, multiple measurements may be obtained for each concurrent path. For example, if there are two concurrent paths to the target, a first capacity measurement is obtained for a first concurrent path, and a second capacity measurement is obtained for a second concurrent path. In a particular embodiment, the first capacity measurement and the second capacity measurement are then summed to determine the total storage capacity available.
- By measuring storage I/O capacity and storage I/O usage starting at the physical storage device level up to the volume level, the I/
O system analyzer 106 can accurately determine the storage I/O capacity and storage I/O usage for a file through the various layers of abstraction. -
FIG. 3 is a general diagram that illustrates a particular embodiment of application programming interfaces (APIs) 110 used to retrieve storage I/O capacity information from components of the storage system ofFIG. 1 . TheAPIs 110 include a performance countersAPI 302, a system servicesAPI 304, a web services for management (WS-Man)API 306, a common information management extensible markup language (CIM-XML)API 308, a common information management object manager (CIMOM)API 310, a Storage Management Initiative (SMI)provider API 312, afirst vendor API 314, asecond vendor API 316, and athird vendor API 318. Each of the APIs 302-318 passes storage performance and configuration information from one or more storage devices associated with each respective API. -
FIG. 4 is a screen view diagram to illustrate a particular embodiment of a visual display that includes storage I/O utilization information.FIG. 4 includes acapacity viewpoints tab 402, astorage explorer column 404, a totalspace utilization column 406, an allocatedspace utilization column 408, and a total I/O utilization column 410. Thecapacity viewpoints tab 402 is selectable to visually display different views of I/O utilization. For example, thecapacity viewpoints 402 may display I/O utilization at a physical storage level, at an operating system disk level (e.g., Windows disk level), at a volume level (e.g., Windows volume level, at a file level, or at a database level. Thestorage explorer column 404 may be used to navigate the various storage levels to enable the display of storage I/O utilization for each respective level. - The total
space utilization column 406 displays a percentage of the total storage space utilized at each level for each storage device, such as a disk, an operating system storage volume, or a logical unit number. The allocatedspace utilization column 408 displays a percentage of the allocated space utilized by a particular file when it is limited from using the total space available for a volume by some other software based allocation restriction. The total I/O utilization 410 column displays the storage I/O utilization such as determined for thesystem 100, for a volume, or for a database. - By periodically determining and displaying the
screen 400 including thecapacity viewpoints tab 402, thestorage explorer column 404, the totalspace utilization column 406, the allocatedspace utilization column 408, and the total I/O utilization column 410, a system analyzer, such as the I/O system analyzer 106, as shown inFIG. 1 , enables a system administrator to detect usage of a software application that approaches a storage I/O resource limit before the file usage exceeds the storage I/O resource limit. Alternatively, a deviation from a predetermined storage I/O utilization range may be automatically determined in order to alert the system administrator to take remedial action. For example, a system administrator may stop execution of a particular software application or allocate additional storage elements to the file to prevent the storage I/O resource limit from being exceeded. An example file that may be monitored is the illustrated file “RAD.mdf” 412 inFIG. 4 . If the storage I/O utilization of thefile 412 exceeded a threshold, such as 90%, then a database administrator could take action proactively to avoid a storage I/O utilization event that could affect performance. -
FIG. 5 is a flow diagram that illustrates a first particular embodiment of a method of determining storage I/O utilization associated with a software application. - At 502, a total storage I/O capacity of a set of storage elements available for use by a software application of a computer system is determined. The software application is located on an operating system volume of the computer system. The total storage I/O capacity is determined by evaluating storage I/O capacity of the storage elements at a physical device level and at a file level as previously described. A system model may be used to identify the storage I/O capacity for each storage element. In a particular embodiment, the set of storage elements includes a storage array comprising a plurality of physical storage devices and a plurality of logical unit numbers, where at least one of the physical storage devices is shared by at least two of the logical unit numbers. In a particular embodiment, the set of storage elements includes a plurality of physical storage devices, and the method further includes identifying one of the storage elements as a potential bottleneck in an I/O path. In one embodiment, an operating system level I/O request traverses one or more alternative paths to reach a target, where the target includes one or more physical storage devices.
- In a particular embodiment, evaluating storage I/O capacity includes determining characteristics of each storage element at a physical storage device level. Each storage element includes physical storage devices or portions of shared physical storage devices. Metrics are collected for each storage element at a file level. A total storage I/O capacity is determined based on the characteristics and based on the metrics.
- Continuing to 504, a number of I/O storage operations performed on behalf of the software application is measured. Moving to 506, a storage I/O utilization of the software application is determined based on the measured number of I/O storage operations as compared to the total storage I/O capacity of the set of storage elements. In a particular embodiment, the method further comprises determining a resource governed utilization of the storage I/O capacity based on historical usage of the storage I/O capacity as compared to a governed storage I/O capacity available and a non-governed storage I/O capacity available. The resource governed utilization may include a storage I/O capacity utilization after taking into account software based restrictions on storage I/O capacity available to the software application. For example, the software application may include an operating system file residing on an operating system volume, where the operating system volume is constrained to a predetermined number of input/output storage accesses per second. The resource governed utilization of the storage I/O capacity is determined based on the software based restrictions.
- In another particular embodiment, the method includes determining the total storage I/O utilization as a ratio of the measured number of I/O storage operations and the total storage I/O capacity, as shown at 508. Proceeding to 510, the storage I/O utilization of the software application is displayed. Based on the storage I/O utilization, a usage of the software application that approaches a storage I/O resource limit may be detected before the usage exceeds the storage I/O resource limit, as shown at 512. The method ends at 514.
-
FIG. 6 is a flow diagram to illustrate a second particular embodiment of a method of determining storage I/O utilization associated with an application, such as a software application. The method includes relating an operating system file to a plurality of physical storage devices, at 602. A query is sent to application programming interfaces (APIs) associated with the plurality of physical storage devices to determine one or more device properties, configuration data, and one or more usage statistics, at 604. Responses are received from the APIs, at 606. A total available storage I/O capacity of the operating system file to access the plurality of physical storage devices is calculated based on the responses, at 608. Storage I/O usage of the operating system file is determined, at 610. Moving to 612, storage I/O utilization of the operating system file is determined based on the storage I/O usage of the operating system file and based on the total available storage I/O capacity. In a particular embodiment, determining the storage I/O utilization includes determining a percentage of the total available storage I/O capacity used by the operating system file. The method ends at 614. - In a particular embodiment, the method further comprises modeling components of a computing system that include the plurality of physical storage devices, modeling relationships between the components, and determining the total available storage I/O capacity based on a given workload and based on a storage I/O capacity of those components of the computing system identified as available for use by the operating system file.
-
FIG. 7 is a flow diagram to illustrate a particular embodiment of a method of calculating a total available storage I/O capacity of an operating system file. In a particular embodiment, the flow diagram ofFIG. 7 illustrates the calculation shown at 608 ofFIG. 6 . - The method includes computing an I/O capacity of each of the multiple storage device elements in an I/O path of the operating system file, at 702. The method includes identifying the storage device element that has a minimum storage I/O capacity, at 704. The total available storage I/O capacity of the operating system file is determined, at 706. The storage device element identified as having the minimum storage I/O capacity represents the total available storage I/O capacity of the operating system file, and thereby determining the total available storage I/O capacity of the file, as shown at 706. In a particular embodiment, at 708, the method further includes measuring an I/O utilization and an I/O rate of each storage device element in the I/O path of the operating system file. In a particular embodiment, the I/O capacity of each of the storage device elements in the I/O path is computed based on the measured I/O utilization and the measured I/O rate, at 710. At 712, the method ends.
- In a particular embodiment, the method further comprises periodically determining the storage I/O usage of the operating system file and updating the storage I/O utilization of the operating system file periodically.
- The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to serve as a complete description of all of the elements and features of apparatus and systems that utilize the structures or methods described herein. Many other embodiments may be apparent to those of skill in the art upon reviewing the disclosure. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Accordingly, the disclosure and the figures are to be regarded as illustrative rather than restrictive.
- Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, configurations, modules, circuits, or steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
- The steps of a method described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in computer readable media, such as random access memory (RAM), flash memory, read only memory (ROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor or the processor and the storage medium may reside as discrete components in a computing device or computer system.
- Although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments.
- The Abstract of the Disclosure is provided with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. This disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all of the features of any of the disclosed embodiments.
- The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the disclosed embodiments. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/242,710 US8527238B2 (en) | 2008-09-30 | 2008-09-30 | Storage input/output utilization associated with a software application |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/242,710 US8527238B2 (en) | 2008-09-30 | 2008-09-30 | Storage input/output utilization associated with a software application |
Publications (2)
Publication Number | Publication Date |
---|---|
US20100082300A1 true US20100082300A1 (en) | 2010-04-01 |
US8527238B2 US8527238B2 (en) | 2013-09-03 |
Family
ID=42058355
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/242,710 Active 2031-03-27 US8527238B2 (en) | 2008-09-30 | 2008-09-30 | Storage input/output utilization associated with a software application |
Country Status (1)
Country | Link |
---|---|
US (1) | US8527238B2 (en) |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100211681A1 (en) * | 2009-02-19 | 2010-08-19 | Oracle International Corporation | Intelligent flood control management |
US20110093588A1 (en) * | 2009-09-02 | 2011-04-21 | Karayi Sumir | Monitoring the performance of a Computer |
US20140330961A1 (en) * | 2011-03-09 | 2014-11-06 | International Business Machines Corporation | Comprehensive bottleneck detection in a multi-tier enterprise storage system |
US8954979B1 (en) * | 2011-08-15 | 2015-02-10 | Netapp, Inc. | Method and system for managing resources in a network |
US9063669B2 (en) | 2013-04-09 | 2015-06-23 | International Business Machines Corporation | Self-detecting storage bottleneck while handling sequential I/O operations |
US9361216B2 (en) | 2012-08-31 | 2016-06-07 | International Business Machines Corporation | Thin provisioning storage resources associated with an application program |
US20170322725A1 (en) * | 2016-05-03 | 2017-11-09 | International Business Machines Corporation | Estimating file level input/output operations per second (iops) |
US10055128B2 (en) | 2010-01-20 | 2018-08-21 | Oracle International Corporation | Hybrid binary XML storage model for efficient XML processing |
US10089208B1 (en) * | 2013-03-15 | 2018-10-02 | Teradata Us, Inc. | Performance continuity management |
US10101920B2 (en) * | 2016-06-30 | 2018-10-16 | Microsoft Technology Licensing, Llc | Disk I/O attribution |
US11093836B2 (en) * | 2016-06-15 | 2021-08-17 | International Business Machines Corporation | Detecting and predicting bottlenecks in complex systems |
US11314613B2 (en) * | 2014-04-15 | 2022-04-26 | Splunk Inc. | Graphical user interface for visual correlation of virtual machine information and storage volume information |
Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6044335A (en) * | 1997-12-23 | 2000-03-28 | At&T Corp. | Productivity metrics for application software systems |
US20020152305A1 (en) * | 2000-03-03 | 2002-10-17 | Jackson Gregory J. | Systems and methods for resource utilization analysis in information management environments |
US20030212792A1 (en) * | 2002-05-10 | 2003-11-13 | Silicon Graphics, Inc. | Real-time storage area network |
US20040025162A1 (en) * | 2002-07-31 | 2004-02-05 | Fisk David C. | Data storage management system and method |
US20040088404A1 (en) * | 2002-11-01 | 2004-05-06 | Vikas Aggarwal | Administering users in a fault and performance monitoring system using distributed data gathering and storage |
US20050022185A1 (en) * | 2003-07-10 | 2005-01-27 | Romero Francisco J. | Systems and methods for monitoring resource utilization and application performance |
US20060215552A1 (en) * | 2005-03-25 | 2006-09-28 | Morihiro Iwata | Method and an apparatus for managing access path |
US20060218285A1 (en) * | 2005-03-25 | 2006-09-28 | Vanish Talwar | Remote desktop performance model for assigning resources |
US7243145B1 (en) * | 2002-09-30 | 2007-07-10 | Electronic Data Systems Corporation | Generation of computer resource utilization data per computer application |
US7325234B2 (en) * | 2001-05-25 | 2008-01-29 | Siemens Medical Solutions Health Services Corporation | System and method for monitoring computer application and resource utilization |
US20080052719A1 (en) * | 2006-08-23 | 2008-02-28 | Peter John Briscoe | Resource management system |
US20080077868A1 (en) * | 2006-09-22 | 2008-03-27 | Bartucca Francis M | System and Method for Visually Representing Resource Usage in a Multi-Node Data Processing System |
US20100076805A1 (en) * | 2008-09-24 | 2010-03-25 | Netapp, Inc. | Adaptive Scheduling Of Storage Operations Based On Utilization Of Multiple Client And Server Resources In A Distributed Network Storage System |
-
2008
- 2008-09-30 US US12/242,710 patent/US8527238B2/en active Active
Patent Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6044335A (en) * | 1997-12-23 | 2000-03-28 | At&T Corp. | Productivity metrics for application software systems |
US20020152305A1 (en) * | 2000-03-03 | 2002-10-17 | Jackson Gregory J. | Systems and methods for resource utilization analysis in information management environments |
US7325234B2 (en) * | 2001-05-25 | 2008-01-29 | Siemens Medical Solutions Health Services Corporation | System and method for monitoring computer application and resource utilization |
US20030212792A1 (en) * | 2002-05-10 | 2003-11-13 | Silicon Graphics, Inc. | Real-time storage area network |
US20040025162A1 (en) * | 2002-07-31 | 2004-02-05 | Fisk David C. | Data storage management system and method |
US7243145B1 (en) * | 2002-09-30 | 2007-07-10 | Electronic Data Systems Corporation | Generation of computer resource utilization data per computer application |
US20040088404A1 (en) * | 2002-11-01 | 2004-05-06 | Vikas Aggarwal | Administering users in a fault and performance monitoring system using distributed data gathering and storage |
US20050022185A1 (en) * | 2003-07-10 | 2005-01-27 | Romero Francisco J. | Systems and methods for monitoring resource utilization and application performance |
US20060215552A1 (en) * | 2005-03-25 | 2006-09-28 | Morihiro Iwata | Method and an apparatus for managing access path |
US20060218285A1 (en) * | 2005-03-25 | 2006-09-28 | Vanish Talwar | Remote desktop performance model for assigning resources |
US20080052719A1 (en) * | 2006-08-23 | 2008-02-28 | Peter John Briscoe | Resource management system |
US20080077868A1 (en) * | 2006-09-22 | 2008-03-27 | Bartucca Francis M | System and Method for Visually Representing Resource Usage in a Multi-Node Data Processing System |
US20100076805A1 (en) * | 2008-09-24 | 2010-03-25 | Netapp, Inc. | Adaptive Scheduling Of Storage Operations Based On Utilization Of Multiple Client And Server Resources In A Distributed Network Storage System |
Cited By (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100211681A1 (en) * | 2009-02-19 | 2010-08-19 | Oracle International Corporation | Intelligent flood control management |
US9128895B2 (en) * | 2009-02-19 | 2015-09-08 | Oracle International Corporation | Intelligent flood control management |
US20110093588A1 (en) * | 2009-09-02 | 2011-04-21 | Karayi Sumir | Monitoring the performance of a Computer |
US9292406B2 (en) * | 2009-09-02 | 2016-03-22 | 1E Limited | Monitoring the performance of a computer |
US10191656B2 (en) | 2010-01-20 | 2019-01-29 | Oracle International Corporation | Hybrid binary XML storage model for efficient XML processing |
US10055128B2 (en) | 2010-01-20 | 2018-08-21 | Oracle International Corporation | Hybrid binary XML storage model for efficient XML processing |
US9866481B2 (en) * | 2011-03-09 | 2018-01-09 | International Business Machines Corporation | Comprehensive bottleneck detection in a multi-tier enterprise storage system |
US20140330961A1 (en) * | 2011-03-09 | 2014-11-06 | International Business Machines Corporation | Comprehensive bottleneck detection in a multi-tier enterprise storage system |
US8954979B1 (en) * | 2011-08-15 | 2015-02-10 | Netapp, Inc. | Method and system for managing resources in a network |
US9361216B2 (en) | 2012-08-31 | 2016-06-07 | International Business Machines Corporation | Thin provisioning storage resources associated with an application program |
US9678675B2 (en) | 2012-08-31 | 2017-06-13 | International Business Machines Corporation | Thin provisioning |
US10089208B1 (en) * | 2013-03-15 | 2018-10-02 | Teradata Us, Inc. | Performance continuity management |
US9645740B2 (en) * | 2013-04-09 | 2017-05-09 | International Business Machines Corporation | Self-detecting storage bottleneck while handling sequential I/O operations |
US20150248243A1 (en) * | 2013-04-09 | 2015-09-03 | International Business Machines Corporation | Self-detecting storage bottleneck while handling sequential i/o operations |
US9063669B2 (en) | 2013-04-09 | 2015-06-23 | International Business Machines Corporation | Self-detecting storage bottleneck while handling sequential I/O operations |
US11314613B2 (en) * | 2014-04-15 | 2022-04-26 | Splunk Inc. | Graphical user interface for visual correlation of virtual machine information and storage volume information |
US11645183B1 (en) | 2014-04-15 | 2023-05-09 | Splunk Inc. | User interface for correlation of virtual machine information and storage information |
US20170322725A1 (en) * | 2016-05-03 | 2017-11-09 | International Business Machines Corporation | Estimating file level input/output operations per second (iops) |
US10032115B2 (en) * | 2016-05-03 | 2018-07-24 | International Business Machines Corporation | Estimating file level input/output operations per second (IOPS) |
US11093836B2 (en) * | 2016-06-15 | 2021-08-17 | International Business Machines Corporation | Detecting and predicting bottlenecks in complex systems |
US10101920B2 (en) * | 2016-06-30 | 2018-10-16 | Microsoft Technology Licensing, Llc | Disk I/O attribution |
Also Published As
Publication number | Publication date |
---|---|
US8527238B2 (en) | 2013-09-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8527238B2 (en) | Storage input/output utilization associated with a software application | |
US10078569B1 (en) | Self adaptive workload classification and forecasting in multi-tiered storage system using arima time series modeling | |
US10175909B2 (en) | Extent migration in multi-tier storage systems | |
US9870330B2 (en) | Methods and systems for filtering collected QOS data for predicting an expected range for future QOS data | |
US10324633B2 (en) | Managing SSD write quotas in data storage systems | |
US9983795B1 (en) | Techniques for determining a storage configuration | |
JP6165886B2 (en) | Management system and method for dynamic storage service level monitoring | |
US9898224B1 (en) | Automatic adjustment of capacity usage by data storage optimizer for data migration | |
US10339455B1 (en) | Techniques for determining workload skew | |
US10013170B1 (en) | Intelligent data compression | |
US9477407B1 (en) | Intelligent migration of a virtual storage unit to another data storage system | |
US9141305B2 (en) | Storage resource usage analysis for customized application options | |
US9026765B1 (en) | Performing write operations in a multi-tiered storage environment | |
US8239584B1 (en) | Techniques for automated storage management | |
US9411834B2 (en) | Method and system for monitoring and analyzing quality of service in a storage system | |
US20150263986A1 (en) | Relationship-Based Resource-Contention Analysis System and Method | |
US8467281B1 (en) | Techniques for identifying devices having slow response times | |
US20150199136A1 (en) | Method and system for monitoring and analyzing quality of service in a storage system | |
US9658778B2 (en) | Method and system for monitoring and analyzing quality of service in a metro-cluster | |
US10225158B1 (en) | Policy based system management | |
US9773026B1 (en) | Calculation of system utilization | |
US10432483B1 (en) | General-purpose metrics publishing with variable resolution | |
EP2404231A1 (en) | Method, system and computer program product for managing the placement of storage data in a multi tier virtualized storage infrastructure | |
US20180267728A1 (en) | Provisioning volumes | |
US10067704B2 (en) | Method for optimizing storage configuration for future demand and system thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MICROSOFT CORPORATION,WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HOLLINGSWORTH, MATT;OSLAKE, JOHN M.;REEL/FRAME:022260/0025 Effective date: 20081212 Owner name: MICROSOFT CORPORATION, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HOLLINGSWORTH, MATT;OSLAKE, JOHN M.;REEL/FRAME:022260/0025 Effective date: 20081212 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034564/0001 Effective date: 20141014 |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |