CN103635887A - Data caching method and storage system - Google Patents
Data caching method and storage system Download PDFInfo
- Publication number
- CN103635887A CN103635887A CN201380001620.0A CN201380001620A CN103635887A CN 103635887 A CN103635887 A CN 103635887A CN 201380001620 A CN201380001620 A CN 201380001620A CN 103635887 A CN103635887 A CN 103635887A
- Authority
- CN
- China
- Prior art keywords
- controller
- address information
- target data
- read
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0862—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches with prefetch
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
- G06F12/0868—Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
Abstract
The embodiment of the invention provides a data caching method and a storage system. The storage system includes a plurality of controllers, and each of the controllers comprises buffer memory. The method comprises steps that a first controller receives a reading data demand carrying address information; a second controller is determined according to the address information carried by the reading data demand; the address information is transmitted to the second controller; and the second controller obtains address information of target data to be read according to the address information, and the target data can be read to buffer memory according to the address information of the target data. Target data of to be read can be accurately predicted.
Description
Technical field
The present invention relates to memory technology, relate in particular to a kind of data cached method and storage system.
Background technology
Cache memory (be called for short buffer memory, claim again cache), is for example, memory buffer between CPU and primary memory (, hard disk) in storage system, and volume ratio hard disk is little, but speed is faster than hard disk.Generally, CPU, when processing read data request, if find data available (being called cache hits) in buffer memory, can return to the result of this read data request immediately, while not hitting, just read data request is sent in hard disk and is read and write data.Due to the speed of the cache access speed much larger than disk read-write, so cache hit rate is higher, and the performance of storage system is higher.Therefore, existing way is that the data of " soon may be accessed to " are read in cache in advance, and follow-up read data request can be hit immediately, and this way is just referred to as to look ahead.
For the storage system that comprises a plurality of controllers, if a plurality of controllers are active/standby (A/P) patterns, only have a controller in running order, so concerning a storage system, its data that are kept at cache the inside all can centralized stores in the cache of this controller, therefore can identify to carry out looking ahead of cache data by read data request being carried out to sequential flow.Yet, if a plurality of controllers are main/main (A/A) patterns, each controller is in running order, read data request may be distributed on each controller, therefore thereby when each controller carries out sequential flow identification and looks ahead read data request, the information of institute's foundation is comprehensive not, and the data of therefore looking ahead are not accurate enough.
Summary of the invention
The embodiment of the present invention provides a kind of data cached method and storage system, to realize in the situation that storage system comprises a plurality of controller, predicts exactly target data to be read.
Embodiment of the present invention first aspect provides a kind of data cached method, and described method is applied in storage system, and described storage system comprises a plurality of controllers, and wherein, each controller comprises buffer memory; Described method comprises:
The read data request that the first controller Receiving Host sends, described read data request is carried address information; The address information of carrying according to described read data request is determined second controller; To described second controller, send described address information;
Described second controller obtains the address information of target data to be read according to described address information, described target data is read in buffer memory according to the address information of described target data.
In the first embodiment of embodiment of the present invention first aspect, the address information that described read data request is carried comprises the start address that described read data request is carried;
Described address information of carrying according to described read data request determines that second controller comprises:
The start address of carrying according to described read data request, according to the hashing algorithm of setting, determines second controller.
In conjunction with the first embodiment of embodiment of the present invention first aspect, in the second embodiment of embodiment of the present invention first aspect, the hashing algorithm of described setting comprises consistance hash algorithm.
In the third embodiment of embodiment of the present invention first aspect, the address information that described read data request is carried comprises the start address that described read data request is carried;
Described address information of carrying according to described read data request determines that second controller comprises:
According to described start address, inquire about default allocation list, obtain second controller corresponding to described start address.
In the 4th kind of embodiment of embodiment of the present invention first aspect, describedly according to the address information of described target data, described target data is read to buffer memory and comprises:
Described second controller reads described target data in the buffer memory of described second controller according to the address information of described target data.
In the 5th kind of embodiment of embodiment of the present invention first aspect, describedly according to the address information of described target data, described target data is read to buffer memory and comprises:
Described second controller determines according to the address information of described target data the 3rd controller that described target data is corresponding; To described the 3rd controller, send prefetched command, described prefetched command comprises the address information of described target data;
Described the 3rd controller reads described target data in the buffer memory of described the 3rd controller according to the address information of described target data.
Embodiment of the present invention second aspect provides a kind of storage system, comprising:
The read data request that the first controller sends for Receiving Host, described read data request is carried address information; The address information of carrying according to described read data request is determined second controller; To described second controller, send described address information;
Described second controller is for obtain the address information of target data to be read according to described address information, described target data is read in buffer memory according to the address information of described target data.
In the first embodiment of embodiment of the present invention second aspect, the address information that described read data request is carried comprises the start address that described read data request is carried;
Described the first controller is the start address for carrying according to described read data request specifically, according to the hashing algorithm of setting, determines second controller.
In conjunction with the first embodiment of embodiment of the present invention second aspect, in the second embodiment of embodiment of the present invention second aspect, the hashing algorithm of described setting comprises consistance hash algorithm.
In the third embodiment of embodiment of the present invention second aspect, the address information that described read data request is carried comprises the start address that described read data request is carried;
Described the first controller, specifically for inquiring about default allocation list according to described start address, obtains second controller corresponding to described start address.
In the 4th kind of embodiment of embodiment of the present invention second aspect, described second controller is also for reading described target data according to the address information of described target data the buffer memory of described second controller.
In the 5th kind of embodiment of embodiment of the present invention second aspect, described system also comprises the 3rd controller;
Described second controller is the 3rd controller for determining that according to the address information of described target data described target data is corresponding also; To described the 3rd controller, send prefetched command, described prefetched command comprises the address information of described target data;
Described the 3rd controller is for reading described target data according to the address information of described target data the buffer memory of described the 3rd controller.
In embodiments of the present invention, at the first controller, receive after the read data request of main frame transmission, the address information of carrying according to described read data request is determined second controller, and described address information is sent to described second controller, by second controller, according to address information acquisition target data to be read, to carry out, read described target data to the operation of buffer memory.Owing to carrying out the controller of the operation that obtains target data to be read, be that the address information of being carried by read data request is determined, therefore can be by a controller collective analysis one section of read data request occurring above logical address, for this section of logical address, the information of the read data request obtaining is comprehensive, therefore can predict exactly target data to be read, and read in buffer memory.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, to the accompanying drawing of required use in embodiment or description of the Prior Art be briefly described below, apparently, accompanying drawing in the following describes is some embodiments of the present invention, for those of ordinary skills, do not paying under the prerequisite of creative work, can also obtain according to these accompanying drawings other accompanying drawing.
The application network configuration diagram of a kind of data cached method that Fig. 1 provides for the embodiment of the present invention;
The process flow diagram of a kind of data cached method that Fig. 2 provides for the embodiment of the present invention;
The process flow diagram of the method that another kind that Fig. 3 provides for the embodiment of the present invention is data cached;
The data structure diagram of the data pre-fetching unit that Fig. 4 provides for the embodiment of the present invention;
The data structure diagram of the data block table in the data pre-fetching unit that Fig. 5 provides for the embodiment of the present invention;
The idiographic flow schematic diagram of the method that another kind that Fig. 6 provides for the embodiment of the present invention is data cached;
The read data request that Fig. 7 provides for the embodiment of the present invention and the corresponding schematic diagram of data block;
The structural representation of the storage system that Fig. 8 provides for the embodiment of the present invention.
Embodiment
For making object, technical scheme and the advantage of the embodiment of the present invention clearer, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is the present invention's part embodiment, rather than whole embodiment.Embodiment based in the present invention, those of ordinary skills, not making the every other embodiment obtaining under creative work prerequisite, belong to the scope of protection of the invention.
The system architecture of the embodiment of the present invention
The data cached method that the embodiment of the present invention provides can realize in storage system.The system architecture schematic diagram of the data cached method that Fig. 1 provides for the embodiment of the present invention, as shown in Figure 1, this storage system comprises a plurality of controllers (take in figure comprise 4 controllers be example) and memory device.In the present embodiment, memory device be take hard disk as example explanation.
Fig. 1 is only exemplary illustration, does not limit concrete networking mode, as: the tree-like networking of cascade, ring-type networking can.As long as can intercom mutually between controller and memory device.
Controller can comprise any computing equipment that current techniques is known, as server, desk-top computer etc.In a kind of application scenarios of the embodiment of the present invention, each controller can be processed the read data request from main frame, and the data of storing in also can accessing storage device, such as the data in memory device are read out, are stored in cache.Or, in another kind of application scenarios, each controller can be processed the read data request from main frame, but one section of storage space in each controller corresponding stored equipment (for example part storage space of part disk or a disk), that is to say that every section of storage space in memory device has the specific controller of its correspondence, can not be by other controller managements or access.It should be noted that, the memory device here refers to disk, hard disk or other storage mediums, does not comprise controller.
Inner at controller, include buffer memory (cache), cache is the memory buffer between CPU and hard disk, volume ratio hard disk is little, but speed is faster than hard disk.In Cache, store partial data, when CPU is when processing read data request, if find data available in cache, cache hits.
Between each controller, can intercom mutually, can access the data of storing in the cache of other controllers.For example, controller 0 receives the read data request from main frame (not shown) visit data A, in the cache of controller 0, do not store data A, but in the cache of controller 1, store data A, therefore controller 0 can send data read command to controller 1, make controller 1 reading out data A from cache, and send to controller 0, thus controller 0 just can be directly to main frame return data A.It should be noted that, because the data communication between cache is to adopt high speed data transfer passage, therefore between controller, cache data shared access speed is very fast, compare with the miss data that need to read in memory device of cache, the time that data-generating process spends from the cache of other controllers is very short.
Operating system and other software programs are installed in controller.For example, include at least one administrative unit of looking ahead in each controller, in embodiments of the present invention, the quantity of the administrative unit of looking ahead comprising in each controller about equally.Each administrative unit of looking ahead is for carrying out the logical memory space data read operation of a sector address scope.Here the look ahead logical memory space of administrative unit management of each, can be a logic storage unit (Logic Unit Number, LUN), can be also one section of region of LUN, can also be file etc., in this no limit.The distribution situation of the administrative unit of looking ahead in controller can be determined according to the address realm of logical memory space.For example, for example, for a logical address (Logic Block Address, LBA), can, according to consistance hash algorithm or other hashing algorithms, obtain out the controller at the administrative unit place of looking ahead corresponding to described logical address.
Memory device can comprise the memory device that current techniques is known, as SSD or direct access storage device (Direct Access Storage Device, DASD) etc.The storage space of memory device can be divided into several logical blocks (chunk), and each chunk has unique ID.In the present embodiment, the management of the data in memory device be take to chunk as unit, the chunk of for example can take reads data in cache as unit.
Known according to description above, in a kind of application scenarios of the embodiment of the present invention, each controller can carry out read-write operation to memory device, for example, each controller can read the data in memory device the cache the inside of oneself, can realize cache when carrying out follow-up read data request to hit; In another kind of application scenarios, a part of storage space of each controller corresponding stored equipment, can only carry out read-write operation to the data of storing in this part storage space, for example the data of storing in this part storage space are read in the cache of oneself, and the data of storing in other storage spaces of this memory device are by other controller managements.
Data cached method
Introduce the data cached method that the embodiment of the present invention provides below, as shown in Figure 2, the process flow diagram of the data cached method providing for the embodiment of the present invention, described method is applied in storage system, described storage system comprises a plurality of controllers, and wherein, each controller comprises buffer memory; Described method comprises:
Step S201: the read data request that the first controller Receiving Host sends, described read data request is carried address information; The address information of carrying according to described read data request is determined second controller; To described second controller, send described address information.
Optionally, described address information comprises start address and the length of data to be read, can be according to the start address of data to be read, and the hashing algorithm according to setting, obtains second controller corresponding to described read data request.
Step S202: described second controller obtains the address information of target data to be read according to described address information, to read described target data in buffer memory according to the address information of described target data.
Optionally, second controller obtains according to described address information after the address information of target data to be read, can described target data be read in the buffer memory of second controller according to the address information of described target data; Or according to the address information of described target data, to the 3rd controller, send prefetched command, the 3rd controller reads described target data in the buffer memory of the 3rd controller according to the address information of described target data.
Optionally, the embodiment of the present invention can also be applied in distributed system, and described distributed system comprises a plurality of nodes, and each node is a server, and the function that each server is carried out and each controller class in storage system seemingly, do not repeat them here.
It should be noted that, the read data request that the main frame that described the first controller receives sends can be one, can be also a plurality of.When read data request is while being a plurality of, the address information that can carry according to described a plurality of read data request judges that whether described a plurality of read data request is continuous, if continuously, described a plurality of read data request can be merged, obtains one section of continuous address information.Described address information of carrying according to described read data request determines that second controller specifically refers to, according to continuous address information after merging, determines second controller.If discontinuous between described a plurality of read data request, the address information of carrying according to each read data request is respectively determined second controller.
In embodiments of the present invention, at the first controller, receive after the read data request of main frame transmission, the address information of carrying according to described read data request is determined second controller, and described address information is sent to described second controller, by second controller, according to address information acquisition target data to be read, to carry out, read described target data to the operation of buffer memory.Owing to carrying out the controller of the operation that obtains target data to be read, be that the address information of being carried by read data request is determined, therefore can be by a controller collective analysis one section of read data request occurring above logical address, for this section of logical address, the information of the read data request obtaining is comprehensive, therefore can predict exactly target data to be read, and read in buffer memory.
As shown in Figure 3, the process flow diagram of another the data cached method providing for the embodiment of the present invention, for convenience of description, it is example that the embodiment of the present invention be take three controllers, but is in fact not limited to three controllers.Referring to Fig. 3, what specifically carry out following step can be the processor in controller, and described method comprises:
Step S301: the first read data request that the first controller Receiving Host sends, described the first read data request is carried the address information of the first data to be read, and wherein said address information comprises start address (LBA) and the length of the first data to be read.
It should be noted that, the logical address in the embodiment of the present invention, claims again start address, claims again LBA.
Step S302: the first controller is determined controller corresponding to described the first read data request.
In embodiments of the present invention, in each controller, include at least one administrative unit of looking ahead, in embodiments of the present invention, the quantity of the administrative unit of looking ahead comprising in each controller about equally, each administrative unit of looking ahead is for managing the storage space of a sector address scope, for example one of LUN section of region.Concrete, can according to consistance hash algorithm or other hashing algorithms, obtain controller corresponding to the first read data request according to the LBA of the first data to be read, and then obtain the administrative unit of looking ahead in this controller.In controller, comprise one while looking ahead administrative unit, controller corresponding to the first read data request determined, the administrative unit of looking ahead that the first read data request is corresponding so has also just been determined; In controller, comprise a plurality of looking ahead during administrative unit, each administrative unit of looking ahead is managed the storage space of a sector address scope, therefore also can be according to the unique administrative unit of looking ahead of determining in a controller of LBA.For instance, controller corresponding to described the first data to be read is second controllers.
Hashing algorithm, claims again hash algorithm, refers to a kind of data structure of determining unique reference address according to certain key (key) value, its objective is and accelerates to search.Wherein, key refers to start address.Optionally, common hashing algorithm can be realized with a hash table, according to key value, searches and can obtain its corresponding reference address in hash table.It should be noted that, general hashing algorithm is linear account form.In embodiments of the present invention, can adopt hashing algorithm to carry out unique controller of determining by the input of start address.
Optionally, consistance hashing algorithm can adopt annular data structure to realize key value to the location of access location.In the data structure of described annular, can comprise a plurality of dummy nodes, for example dummy node 0, dummy node 1, dummy node 2, dummy node 3 ... dummy node 10,, the corresponding one section of reference address of each dummy node.Wherein every two adjacent dummy nodes connect successively, and dummy node 10 is connected with dummy node 0 head and the tail, jointly forms an annular.It is example that the storage system of take has three controllers, three controllers are distinguished called after controller A, controller B and controller C, dummy node in the corresponding several ring data structures of each controller difference, for example corresponding dummy node 0 of controller A, dummy node 1, dummy node 2, dummy node 3; The corresponding dummy node 4 of controller B, dummy node 5, dummy node 6, dummy node 7; The corresponding dummy node 8 of controller C, dummy node 9, dummy node 10.In this way, also can make the unique corresponding controller of start address.
Yet, for consistance hashing algorithm, when storage system increases new controller, for example increase controller D, without data structure or the modification algorithm of again arranging, only need to adjust the corresponding dummy node of each controller, just can make the corresponding segment base of newly-increased controller D.
Optionally, in each controller of storage system, can preserve a default allocation list, described allocation list comprises the corresponding relation between start address and each controller, the controller that receives read data request can be inquired about according to the start address of carrying in read data request in allocation list, thereby obtains controller corresponding to described start address.
Optionally, above-mentioned allocation list can only be kept in a controller, when other controllers receive read data request, can send inquiry request to the controller of preserving allocation list, described inquiry request comprises the start address that described read data request is carried, the controller of described preservation allocation list can be inquired about according to described start address in allocation list, thereby obtain controller corresponding to described start address, and Query Result is sent to the controller that receives read data request.
Optionally, while breaking down in order to prevent from preserving the controller of allocation list, described allocation list is lost, and can in another controller of described storage system, preserve the copy of described allocation list.
Optionally, can also obtain by the mode of delivery controller corresponding to described start address, concrete, the number by described start address divided by controller, can obtain corresponding controller according to the mould calculating.
Step S303: the second read data request that second controller Receiving Host sends, described the second read data request is carried the address information of the second data to be read, and wherein said address information comprises start address (LBA) and the length of the second data to be read.
It should be noted that between step S301 and step S303, there is no dividing of sequencing.
Step S304: second controller is determined controller corresponding to described the second read data request.
Concrete, can obtain its corresponding controller according to the LBA of the second data to be read, S302 is similar with step, does not repeat them here.
When controller corresponding to the first read data request controller corresponding with the second read data request is not identical, two controllers can carry out respectively data cached operation of looking ahead, and are independent of each other.Here discuss the identical situation of controller that controller corresponding to the first read data request is corresponding with the second read data request.
Step S305: the first controller sends the drop point information of the first read data request to second controller, described drop point information comprises the address information of the first data to be read, in addition, drop point information can also comprise the ID of the main frame that sends the first read data request, and the ID of the first controller etc., described drop point information can be used as the analysis foundation of data pre-fetching.
Because controller corresponding to the second read data request is second controller, therefore the processor of second controller can be stored in the drop point information of the second read data request the administrative unit of looking ahead that address in buffer memory is pushed to second controller, or the administrative unit of looking ahead that otherwise makes second controller obtains the drop point information of the second read data request, in this no limit.
Step S306: second controller, according to the drop point information of the drop point information of the first read data request and the second read data request, is predicted the target data of next read data request.
Next read data request refers to that storage system is about to the read data request (also not receiving at present) receiving, and for convenience of description, is called third reading request of data by next read data request.It should be noted that, next read data request is not limited to the and then read data request of the first read data request, the second read data request, so long as the read data request receiving after the first read data request, the second read data request can be called next read data request.
Second controller can utilize the administrative unit of looking ahead to predict the target data of third reading request of data.The administrative unit of looking ahead be in controller, comprise for carrying out the functional unit of data cached read operation, as shown in Figure 4, the administrative unit of looking ahead comprises: data block table, interface and prefetch policy module.Wherein data block table comprises a plurality of data blocks, the logical block (chunk) on the corresponding disk of each data block, and size is identical.Described data block is for recording drop point information and other information of read data request.Described data block table can sort from small to large according to the LBA comprising in drop point information, or sequence from big to small.In addition, the data block table of the embodiment of the present invention also can be not limited to the form of table, can also be RBTree, or binary tree etc. other can realize the data management structure of sequential search.It should be noted that, what in each data block, record is drop point information and other information of read data request, but does not comprise data to be read itself.In addition, interface is for the drop point information of the read data request that receives other controllers and send, or sends prefetched command to other controllers; Prefetch policy module is for carrying out read operation according to the prefetch policy of setting.
As shown in Figure 5, the information recording in each data block can comprise: the ID of Chunk ID, drop point information and the last chunk reading, and wherein drop point information specifically can comprise host id, controller ID, LBA and length;
Described Chunk ID specifically refers to the ID of the chunk on the corresponding disk of described data block, can obtain the start address of chunk by the size that Chunk ID is multiplied by each chunk.Because looking ahead of data be take chunk as unit, so also need to know the start address of the chunk at target data place when prefetch target data.It should be noted that, previously described LBA refers to the start address of read data request, is different from the start address of described chunk.In some cases, the start address of read data request is identical with the start address of chunk, but most applications, the start address of read data request is not identical with the start address of chunk.
The ID of the chunk that described the last time reads, for being identified at the scope of looking ahead of the read operation of the last time occurring in described data block, due to when carrying out read operation, the chunk of take carries out as unit, so the ID of the chunk that can read according to described the last time judges that this looks ahead and whether have overlapping with the scope of looking ahead last time, if had, when carrying out read operation, can not comprise the chunk of lap.
In addition can also record, the information such as sequence number, timestamp, bitmap of read data request.
As shown in Figure 6, step S306 specifically can comprise the following steps:
S3061: according to the drop point information of the drop point information of the first read data request and the second read data request, determine that the first read data request and the second read data request have continuous ordinal relation.
Concrete determination methods is, according to the start address of the first read data request and length, obtain the end address of the first read data request, if the start address of the end address of the first read data request and the second read data request is continuous, illustrate that the first read data request and the second read data request have continuous ordinal relation; Or, according to the start address of the second read data request and length, obtain the end address of the second read data request, if the start address of the end address of the second read data request and the first read data request is continuous, illustrate that the first read data request and the second read data request have continuous ordinal relation.
It should be noted that, in embodiments of the present invention, the first read data request and the second read data request can be definitely continuously, allowing has space, address to a certain degree between them.
S3062: determine the first read data request and data block corresponding to the second read data request.
Concrete, can, according to the drop point information of the drop point information of the first read data request and the second read data request, determine the first read data request and data block corresponding to the second read data request.Because the first read data request and the second read data request have continuous ordinal relation, so the data block of their correspondence is also continuous.The first read data request and data block corresponding to the second read data request can be as shown in Figure 7.
S3063: determine the data block continuous data block corresponding with described the first read data request and the second read data request, obtain the maximum continuous segment of data block.
Concrete, can be according to the first read data request and data block corresponding to the second read data request, travel through forward described data block table, if there be the data block data block corresponding with the first read data request and the second read data request continuous, according to the drop point information recording in its continuous data block judge the read data request that occurred last time in described data block whether with the first current read data request, the second read data request is continuous, if continuously, continue to obtain continuous data block in data block table, until obtain the maximum continuous segment of the data block corresponding with described the first read data request and the second read data request.
S3064: the length and the start address that obtain target data.
Concrete, the size of the maximum continuous segment of the data block that can obtain according to step S3063, through the calculating of prefetch policy, obtains the length of target data.
Optionally, if the ID of the chunk being read by described the last time know, the wherein part in target data, in the last time of prefetched mistake during read operation, can be removed this part data so.
In addition, predict the target data of next read data request, except determining the length of target data, also need to determine the start address of target data.In the present embodiment, the start address of target data is the end address of the first read data request or the end address of the second read data request.
After second controller dopes the target data of next read data request, can from disk, read described target data according to the start address of described target data and length, be stored in the cache of second controller, can hit by cache when carrying out third reading request of data next time.This situation be mainly applicable to that each controller all can be managed or accessing storage device in the scene of each disk.
Yet, in some application scenarios, each controller can only manage or accessing storage device in part storage space (the part storage space of part disk or a disk), that is to say that every section of storage space in memory device has the specific controller of its correspondence, can not be by other controller managements or access.In this case, the storage space at hypothetical target data place is by the 3rd controller management, and the present embodiment can also comprise so:
Step S307: second controller sends prefetched command to the 3rd controller, and described prefetched command comprises start address and the length of target data, and the 3rd controller is read described target data in its buffer memory.
Optionally, the storage space that second controller can be determined described target data place according to the start address of target data, by the 3rd controller management, therefore can send data pre-fetching order to the 3rd controller.Concrete, second controller can pass through system configuration or some existing computing method, by the start address of target data, obtains the controller corresponding to storage space at described target data place.
Step S308: the 3rd controller is looked ahead described target data in its buffer memory.
Concrete, the 3rd controller can read described target data according to the start address of described target data and length from disk, is stored in the cache of the 3rd controller.
Step S309: the third reading request of data that the first controller Receiving Host sends, the data to be read of described third reading request of data are described target datas, or data to be read are parts of described target data.
The first controller receives after described third reading request of data, finds not store in its cache described target data, but in the cache of the 3rd controller, stores described target data, execution step S310.
Optionally, what receive described third reading request of data can be any one controller in storage system, if receive the 3rd controller of described third reading request of data, the 3rd controller can directly send to described main frame by target data in its buffer memory, does not need to perform step S310-S311; If what receive described third reading request of data is other controllers in storage system, it is similar that its operation steps and the first controller receive the situation of described third reading request of data.
Step S310: the first controller sends data read command to the 3rd controller, and described data read command comprises start address and the length of described target data, for requiring described the 3rd controller to send described target data.
Step S311: the 3rd controller sends described target data to the first controller.
It should be noted that, what the data channel between controller adopted is high speed data transfer passage, and according to statistics, the access speed of the cache data between controller is generally less than 1ms.Yet if cannot hit by cache, the speed that reads target data from disk is 6-10ms.Therefore,, even hit across controller, its speed is also much larger than the speed from disk reading out data.
Step S312: the first controller receives after the target data of the 3rd controller transmission, sends described target data to main frame, has realized cache hit.
In addition, in the present embodiment, the information that the first controller can also carry according to third reading request of data is determined the controller that third reading request of data is corresponding, to this controller, send the drop point information of third reading request of data, make the data to be read of the next read data request of administrative unit continuation prediction of looking ahead of this controller, carry out data cached operation.This process can repeating step S301-step S308, repeats no more here.
In embodiments of the present invention, at the first controller, receive after the read data request of main frame transmission, the address information of carrying according to described read data request is determined second controller, and described address information is sent to described second controller, by second controller, according to address information acquisition target data to be read, to carry out, read described target data to the operation of buffer memory.Owing to carrying out the controller of the operation that obtains target data to be read, be that the address information of being carried by read data request is determined, therefore can be by a controller collective analysis one section of read data request occurring above logical address, for this section of logical address, the information of the read data request obtaining is comprehensive, therefore can predict exactly target data to be read, and read in buffer memory.
Storage system
Introduce the storage system that the embodiment of the present invention provides below, as shown in Figure 8, the structural drawing of the storage system 80 providing for the embodiment of the present invention, comprises a plurality of controllers, and wherein, each controller comprises processor and buffer memory; Wherein:
The read data request that the first controller 801 sends for Receiving Host, described read data request is carried address information; The address information of carrying according to described read data request is determined second controller 802; To described second controller 802, send described address information.Concrete, what carry out aforesaid operations is the processor in the first controller 801.
Optionally, described address information comprises start address and the length of data to be read, can be according to the start address of data to be read, and according to the hashing algorithm of setting, obtaining controller corresponding to described read data request is second controller 802.
In embodiments of the present invention, in each controller, include at least one administrative unit of looking ahead, in embodiments of the present invention, the quantity of the administrative unit of looking ahead comprising in each controller about equally, each administrative unit of looking ahead is for managing the storage space of a sector address scope, for example one of LUN section of region.Concrete, can according to consistance hash algorithm or other hashing algorithms, obtain controller corresponding to the first read data request according to the LBA of the first data to be read, and then obtain the administrative unit of looking ahead in this controller.In controller, comprise one while looking ahead administrative unit, controller corresponding to the first read data request determined, the administrative unit of looking ahead that the first read data request is corresponding so has also just been determined; In controller, comprise a plurality of looking ahead during administrative unit, each administrative unit of looking ahead is managed the storage space of a sector address scope, therefore also can be according to the unique administrative unit of looking ahead of determining in a controller of LBA.For instance, controller corresponding to described the first data to be read is second controllers.
Optionally, can adopt hashing algorithm to carry out unique controller of determining by the input of start address, described hashing algorithm can be consistance hashing algorithm.
Optionally, in each controller of storage system, can preserve a default allocation list, described allocation list comprises the corresponding relation between start address and each controller, the controller that receives read data request can be inquired about according to the start address of carrying in read data request in allocation list, thereby obtains controller corresponding to described start address.
Optionally, above-mentioned allocation list can only be kept in a controller, when other controllers receive read data request, can send inquiry request to the controller of preserving allocation list, described inquiry request comprises the start address that described read data request is carried, the controller of described preservation allocation list can be inquired about according to described start address in allocation list, thereby obtain controller corresponding to described start address, and Query Result is sent to the controller that receives read data request.
Optionally, while breaking down in order to prevent from preserving the controller of allocation list, described allocation list is lost, and can in another controller of described storage system, preserve the copy of described allocation list.
Optionally, can also obtain by the mode of delivery controller corresponding to described start address, concrete, the number by described start address divided by controller, can obtain corresponding controller according to the mould calculating.
After the address information that the first controller 801 carries according to described read data request is determined second controller 802, can send described address information to described second controller 802.
Described second controller 802 is for obtain the address information of target data to be read according to described address information, described target data is read in buffer memory according to the address information of described target data.Concrete, what carry out aforesaid operations is the processor in the first controller 801.
Optionally, second controller 802 obtains according to described address information after the address information of target data to be read, can described target data be read in the buffer memory of second controller 802 according to the address information of described target data; Or according to the address information of described target data, to the 3rd controller 803, send prefetched command, the 3rd controller 803 reads described target data in the buffer memory of the 3rd controller 803 according to the address information of described target data.
Wherein, second controller 802 can utilize the administrative unit of looking ahead to obtain target data to be read.Concrete preparation method and above-described embodiment of the method are similar, repeat no more here.
Optionally, the next read data request that the first controller 801 Receiving Hosts send, the data to be read of described next read data request are described target datas, or data to be read are parts of described target data.The first controller 801 receives after described next read data request, find not store in its cache described target data, but in the cache of the 3rd controller 803, store described target data, can send data read command to the 3rd controller 803, described data read command comprises start address and the length of described target data, for requiring described the 3rd controller 803 to send described target data.The 3rd controller 803 receives after described data read command, to the first controller 801, sends described target data.
It should be noted that, next read data request is not limited to the read data request that and then described the first controller receives, so long as the read data request receiving after described read data request can be called next read data request.
What adopt due to data channel between controller is high speed data transfer passage, and according to statistics, the access speed of the cache data between controller is generally less than 1ms.Yet if cannot hit by cache, the speed that reads target data from disk is 6-10ms.Therefore,, even hit across controller, its speed is also much larger than the speed from disk reading out data.
In embodiments of the present invention, at the first controller, receive after the read data request of main frame transmission, the address information of carrying according to described read data request is determined second controller, and described address information is sent to described second controller, by second controller, according to address information acquisition target data to be read, to carry out, read described target data to the operation of buffer memory.Owing to carrying out the controller of the operation that obtains target data to be read, be that the address information of being carried by read data request is determined, therefore can be by a controller collective analysis one section of read data request occurring above logical address, for this section of logical address, the information of the read data request obtaining is comprehensive, therefore can predict exactly target data to be read, and read in buffer memory.
The possible implementation that one of ordinary skill in the art will appreciate that various aspects of the present invention or various aspects can be embodied as system, method or computer program.Therefore, the possible implementation of each aspect of the present invention or various aspects can adopt complete hardware implementation example, complete implement software example (comprising firmware, resident software etc.), or the form of the embodiment of integration software and hardware aspect, is all referred to as " circuit ", " module " or " system " here.In addition, the possible implementation of each aspect of the present invention or various aspects can adopt the form of computer program, and computer program refers to the computer readable program code being stored in computer-readable medium.
Computer-readable medium can be computer-readable signal media or computer-readable recording medium.Computer-readable recording medium is including but not limited to electronics, magnetic, optics, electromagnetism, infrared or semiconductor system, equipment or device, or aforesaid appropriately combined arbitrarily, as random access memory (RAM), ROM (read-only memory) (ROM), Erasable Programmable Read Only Memory EPROM (EPROM or flash memory), optical fiber, portable ROM (read-only memory) (CD-ROM).
Processor in computing machine reads the computer readable program code being stored in computer-readable medium, makes processor can carry out the function action of stipulating in the combination of in process flow diagram each step or each step; Generation is implemented in the device of the function action of stipulating in each piece of block diagram or the combination of each piece.
Computer readable program code can be completely carried out on user's computing machine, part is carried out on user's computing machine, as independent software package, part on user's computing machine and part on remote computer, or on remote computer or server, carry out completely.Also should be noted that in some alternate embodiment, in process flow diagram, in each step or block diagram, the dated function of each piece may be not according to occurring in sequence of indicating in figure.For example, depend on related function, in fact two steps or two pieces that illustrate in succession may be executed substantially concurrently, or these pieces sometimes may be carried out with reverse order.
Claims (16)
1. a data cached method, described method is applied in storage system, and described storage system comprises a plurality of controllers, and wherein, each controller comprises buffer memory; It is characterized in that, described method comprises:
The read data request that the first controller Receiving Host sends, described read data request is carried address information; The address information of carrying according to described read data request is determined second controller; To described second controller, send described address information;
Described second controller obtains the address information of target data to be read according to described address information, described target data is read in buffer memory according to the address information of described target data.
2. method according to claim 1, is characterized in that, the address information that described read data request is carried comprises the start address that described read data request is carried;
Described address information of carrying according to described read data request determines that second controller comprises:
The start address of carrying according to described read data request, according to the hashing algorithm of setting, determines second controller.
3. method according to claim 2, is characterized in that, the hashing algorithm of described setting comprises consistance hash algorithm.
4. method according to claim 1, is characterized in that, the address information that described read data request is carried comprises the start address that described read data request is carried;
Described address information of carrying according to described read data request determines that second controller comprises:
According to described start address, inquire about default allocation list, obtain second controller corresponding to described start address.
5. method according to claim 1, is characterized in that, describedly according to the address information of described target data, described target data is read to buffer memory and comprises:
Described second controller reads described target data in the buffer memory of described second controller according to the address information of described target data.
6. method according to claim 1, is characterized in that, describedly according to the address information of described target data, described target data is read to buffer memory and comprises:
Described second controller determines according to the address information of described target data the 3rd controller that described target data is corresponding; To described the 3rd controller, send prefetched command, described prefetched command comprises the address information of described target data;
Described the 3rd controller reads described target data in the buffer memory of described the 3rd controller according to the address information of described target data.
7. method according to claim 6, is characterized in that, also comprises:
Described the first controller receives the next read data request that described main frame sends, and described next read data request comprises the address information of described target data;
According to the address information of described target data, determine in the buffer memory of described the 3rd controller and store described target data;
To described the 3rd controller, send data read command, make described the 3rd controller that described target data is sent to described the first controller;
To described main frame, send described target data.
8. method according to claim 1, is characterized in that, the address information of described target data comprises start address and the length of described target data;
The address information that described second controller obtains target data to be read according to described address information comprises:
The address information that the read data request that described second controller sends according to described main frame is carried, determines the start address of described target data; And
The address information that the read data request sending according to described main frame is carried, corresponding the first data block in specified data piece table, and with continuous the second data block of described the first data block; According to the big or small summation of described the first data block and the second data block, obtain the length of described target data.
9. a storage system, is characterized in that, described system comprises:
The read data request that the first controller sends for Receiving Host, described read data request is carried address information; The address information of carrying according to described read data request is determined second controller; To described second controller, send described address information;
Described second controller is for obtain the address information of target data to be read according to described address information, described target data is read in buffer memory according to the address information of described target data.
10. system according to claim 9, is characterized in that, the address information that described read data request is carried comprises the start address that described read data request is carried;
Described the first controller is the start address for carrying according to described read data request specifically, according to the hashing algorithm of setting, determines second controller.
11. systems according to claim 10, is characterized in that, the hashing algorithm of described setting comprises consistance hash algorithm.
12. systems according to claim 9, is characterized in that, the address information that described read data request is carried comprises the start address that described read data request is carried;
Described the first controller, specifically for inquiring about default allocation list according to described start address, obtains second controller corresponding to described start address.
13. systems according to claim 9, is characterized in that, described second controller is also for reading described target data according to the address information of described target data the buffer memory of described second controller.
14. systems according to claim 9, is characterized in that, described system also comprises the 3rd controller;
Described second controller is the 3rd controller for determining that according to the address information of described target data described target data is corresponding also; To described the 3rd controller, send prefetched command, described prefetched command comprises the address information of described target data;
Described the 3rd controller is for reading described target data according to the address information of described target data the buffer memory of described the 3rd controller.
15. systems according to claim 14, is characterized in that,
The next read data request that described the first controller also sends for receiving described main frame, described next read data request comprises the address information of described target data;
According to the address information of described target data, determine in the buffer memory of described the 3rd controller and store described target data;
To described the 3rd controller, send data read command, make described the 3rd controller that described target data is sent to described the first controller;
To described main frame, send described target data.
16. systems according to claim 9, is characterized in that, the address information of described target data comprises start address and the length of described target data;
Described second controller is used for the address information of carrying according to the read data request of described main frame transmission, determines the start address of described target data; And
The address information that the read data request sending according to described main frame is carried, corresponding the first data block in specified data piece table, and with continuous the second data block of described the first data block; According to the big or small summation of described the first data block and the second data block, obtain the length of described target data.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2013/084024 WO2015039352A1 (en) | 2013-09-23 | 2013-09-23 | Data caching method and storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103635887A true CN103635887A (en) | 2014-03-12 |
| CN103635887B CN103635887B (en) | 2015-07-08 |
Family
ID=50215548
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201380001620.0A Active CN103635887B (en) | 2013-09-23 | 2013-09-23 | Data caching method and storage system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN103635887B (en) |
| WO (1) | WO2015039352A1 (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104461943A (en) * | 2014-12-29 | 2015-03-25 | 成都致云科技有限公司 | Data reading method, device and system |
| CN105574008A (en) * | 2014-10-11 | 2016-05-11 | 华为技术有限公司 | Task scheduling method and equipment applied to distributed file system |
| CN107844270A (en) * | 2014-12-31 | 2018-03-27 | 华为技术有限公司 | A kind of memory array system and data write request processing method |
| CN108345551A (en) * | 2017-01-23 | 2018-07-31 | 杭州海康威视数字技术股份有限公司 | A kind of method and device of storage data |
| WO2019127487A1 (en) * | 2017-12-29 | 2019-07-04 | 华为技术有限公司 | Data prefetching method and apparatus, and storage device |
| WO2020037625A1 (en) * | 2018-08-23 | 2020-02-27 | 袁振南 | Distributed storage system and data read-write method therefor, and storage terminal and storage medium |
| CN110928495A (en) * | 2019-11-12 | 2020-03-27 | 杭州宏杉科技股份有限公司 | Data processing method and device on multi-control storage system |
| CN111406251A (en) * | 2018-08-24 | 2020-07-10 | 华为技术有限公司 | Data prefetching method and device |
| CN112199304A (en) * | 2019-07-08 | 2021-01-08 | 华为技术有限公司 | Data prefetching method and device |
| CN112579479A (en) * | 2020-12-07 | 2021-03-30 | 海光信息技术股份有限公司 | Processor and method for maintaining transaction order while maintaining cache coherency |
| CN112799589A (en) * | 2021-01-14 | 2021-05-14 | 新华三大数据技术有限公司 | Data reading method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030195939A1 (en) * | 2002-04-16 | 2003-10-16 | Edirisooriya Samatha J. | Conditional read and invalidate for use in coherent multiprocessor systems |
| US6728258B1 (en) * | 1995-11-15 | 2004-04-27 | Hitachi, Ltd. | Multi-processor system and its network |
| CN1818855A (en) * | 2005-02-09 | 2006-08-16 | 国际商业机器公司 | Method and apparatus for performing data prefetch in a multiprocessor system |
| CN101311894A (en) * | 2007-03-30 | 2008-11-26 | 英特尔公司 | Method and apparatus for speculative prefetching in a multi-processor/multi-core message-passing machine |
| CN101630303A (en) * | 2009-08-24 | 2010-01-20 | 成都市华为赛门铁克科技有限公司 | Request message processing method, device and multi-processor memory system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101201723A (en) * | 2006-12-14 | 2008-06-18 | 英业达股份有限公司 | Virtual disc router system, virtual disc accesses system and method |
| CN101055511B (en) * | 2007-05-16 | 2010-05-26 | 华为技术有限公司 | Memory array system and its data operation method |
| CN101840309B (en) * | 2009-10-28 | 2011-10-26 | 创新科存储技术有限公司 | Access control method and system of double control disk array in multipath environment |
-
2013
- 2013-09-23 WO PCT/CN2013/084024 patent/WO2015039352A1/en active Application Filing
- 2013-09-23 CN CN201380001620.0A patent/CN103635887B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6728258B1 (en) * | 1995-11-15 | 2004-04-27 | Hitachi, Ltd. | Multi-processor system and its network |
| US20030195939A1 (en) * | 2002-04-16 | 2003-10-16 | Edirisooriya Samatha J. | Conditional read and invalidate for use in coherent multiprocessor systems |
| CN1818855A (en) * | 2005-02-09 | 2006-08-16 | 国际商业机器公司 | Method and apparatus for performing data prefetch in a multiprocessor system |
| CN101311894A (en) * | 2007-03-30 | 2008-11-26 | 英特尔公司 | Method and apparatus for speculative prefetching in a multi-processor/multi-core message-passing machine |
| CN101630303A (en) * | 2009-08-24 | 2010-01-20 | 成都市华为赛门铁克科技有限公司 | Request message processing method, device and multi-processor memory system |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105574008A (en) * | 2014-10-11 | 2016-05-11 | 华为技术有限公司 | Task scheduling method and equipment applied to distributed file system |
| CN105574008B (en) * | 2014-10-11 | 2020-01-31 | 华为技术有限公司 | Task scheduling method and device applied to distributed file system |
| CN104461943A (en) * | 2014-12-29 | 2015-03-25 | 成都致云科技有限公司 | Data reading method, device and system |
| CN104461943B (en) * | 2014-12-29 | 2017-10-27 | 成都致云科技有限公司 | Method for reading data, device and system |
| CN107844270A (en) * | 2014-12-31 | 2018-03-27 | 华为技术有限公司 | A kind of memory array system and data write request processing method |
| CN108345551A (en) * | 2017-01-23 | 2018-07-31 | 杭州海康威视数字技术股份有限公司 | A kind of method and device of storage data |
| CN108345551B (en) * | 2017-01-23 | 2020-05-12 | 杭州海康威视数字技术股份有限公司 | Data storage method and device |
| WO2019127487A1 (en) * | 2017-12-29 | 2019-07-04 | 华为技术有限公司 | Data prefetching method and apparatus, and storage device |
| US11099997B2 (en) | 2017-12-29 | 2021-08-24 | Huawei Technologies Co., Ltd. | Data prefetching method and apparatus, and storage device |
| WO2020037625A1 (en) * | 2018-08-23 | 2020-02-27 | 袁振南 | Distributed storage system and data read-write method therefor, and storage terminal and storage medium |
| CN111406251A (en) * | 2018-08-24 | 2020-07-10 | 华为技术有限公司 | Data prefetching method and device |
| US11669453B2 (en) | 2018-08-24 | 2023-06-06 | Huawei Technologies Co., Ltd. | Data prefetching method and apparatus |
| CN111406251B (en) * | 2018-08-24 | 2023-12-08 | 华为技术有限公司 | Data prefetching method and device |
| CN112199304A (en) * | 2019-07-08 | 2021-01-08 | 华为技术有限公司 | Data prefetching method and device |
| CN112199304B (en) * | 2019-07-08 | 2024-04-09 | 华为技术有限公司 | Data prefetching method and device |
| CN110928495A (en) * | 2019-11-12 | 2020-03-27 | 杭州宏杉科技股份有限公司 | Data processing method and device on multi-control storage system |
| CN110928495B (en) * | 2019-11-12 | 2023-09-22 | 杭州宏杉科技股份有限公司 | Data processing method and device on multi-control storage system |
| CN112579479A (en) * | 2020-12-07 | 2021-03-30 | 海光信息技术股份有限公司 | Processor and method for maintaining transaction order while maintaining cache coherency |
| CN112579479B (en) * | 2020-12-07 | 2022-07-08 | 成都海光微电子技术有限公司 | Processor and method for maintaining transaction order while maintaining cache coherency |
| CN112799589A (en) * | 2021-01-14 | 2021-05-14 | 新华三大数据技术有限公司 | Data reading method and device |
| CN112799589B (en) * | 2021-01-14 | 2023-07-14 | 新华三大数据技术有限公司 | Data reading method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103635887B (en) | 2015-07-08 |
| WO2015039352A1 (en) | 2015-03-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103635887B (en) | Data caching method and storage system | |
| CN108459826B (en) | Method and device for processing IO (input/output) request | |
| US20180349396A1 (en) | Managing I/O Operations in a Storage Network | |
| US9665485B2 (en) | Logical and physical block addressing for efficiently storing data to improve access speed in a data deduplication system | |
| US20170116136A1 (en) | Reducing data i/o using in-memory data structures | |
| TWI664541B (en) | Methods and systems for autonomous memory searching | |
| US10901619B2 (en) | Selecting pages implementing leaf nodes and internal nodes of a data set index for reuse | |
| US10877680B2 (en) | Data processing method and apparatus | |
| US9817865B2 (en) | Direct lookup for identifying duplicate data in a data deduplication system | |
| US20170124077A1 (en) | Flash module provided with database operation unit, and storage device | |
| CN102841854A (en) | Method and system for executing data reading based on dynamic hierarchical memory cache (hmc) awareness | |
| CN108363641B (en) | Main and standby machine data transmission method, control node and database system | |
| CN107329704B (en) | Cache mirroring method and controller | |
| JP6402647B2 (en) | Data arrangement program, data arrangement apparatus, and data arrangement method | |
| US20160147670A1 (en) | Page cache device and method for efficient mapping | |
| CN111694770B (en) | Method and device for processing IO (input/output) request | |
| WO2016041127A1 (en) | Data duplication method and storage array | |
| CN112114753B (en) | Data writing method, device and equipment | |
| CN112256599A (en) | Data prefetching method and device and storage device | |
| JP2017084338A (en) | Methods and systems for improving flash memory flushing | |
| CN111552692A (en) | Plus-minus cuckoo filter | |
| CN110765076A (en) | Data storage method and device, electronic equipment and storage medium | |
| US9652155B2 (en) | Computer system, cash data management method, and computer | |
| US8055815B2 (en) | Optimal paths with SCSI I/O referrals | |
| CN113835613B (en) | File reading method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |