CN103020151A - Large data quantity batch processing system and large data quantity batch processing method - Google Patents
Large data quantity batch processing system and large data quantity batch processing method Download PDFInfo
- Publication number
- CN103020151A CN103020151A CN2012104800632A CN201210480063A CN103020151A CN 103020151 A CN103020151 A CN 103020151A CN 2012104800632 A CN2012104800632 A CN 2012104800632A CN 201210480063 A CN201210480063 A CN 201210480063A CN 103020151 A CN103020151 A CN 103020151A
- Authority
- CN
- China
- Prior art keywords
- major key
- data
- paging
- cache device
- key set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention provides a large data quantity batch processing system which comprises a middleware unit, a first-order cache device and a second-order cache device, wherein the middleware unit is used for sending an information request to the first-order cache device, receiving a second-order paging primary key set from the second-order cache device, and sending a persistence data request to a database after querying to-be-processed data to the database according to the second-order paging primary key set and calculating the to-be-processed data; the first-order cache device is used for querying the primary key set according with the information request to the database, generating a first-order paging primary key set according to the primary key set and returning the first-order paging primary key set to the second-order cache device; and the second-order cache device is used for generating the second-order paging primary key set according to the first-order paging primary key set, and returning the second-order paging primary key set to the middleware unit. The invention further provides a large data quantity batch processing method. According to the technical scheme, the processing speed of mass data of the system can be increased greatly, the processing time of the system is shortened, and the combination property of the system is improved.
Description
Technical field
The present invention relates to field of computer technology, in particular to a kind of big data quantity batch processing system and a kind of big data quantity batch processing method.
Background technology
At present in the large-scale online transaction processing system (OLTP), weigh the index of its system performance quality, the processing speed of some key core algorithms under the big data quantity application scenarios often, and the speed of processing speed directly affects the performance of whole system.
A large-scale information system, some own more complicated business processing logic are often arranged, the business processing algorithm, when these complicated business processing efficiency under the small data quantity application scenarios often all out in the cold, because system response time is than faster under this scene, and may will occur in the big data quantity situation bottleneck of system's handling property, long-time without the serious situation such as response or the machine of directly delaying, so wherein relatively the problem of general character and core is exactly: first, if data volume is excessive, disposable the reading of program may cause Installed System Memory to overflow in the internal memory; The second, in internal memory, circulation is read one by one data and is being processed if not disposable reading out data, and then algorithm has become the single processing of circulation by batch processing, also must greatly affect the performance of system.To this, prior art uses the background paging technology to solve such problem.
Existing paging technique all is to realize paging technique in database side, a kind of is directly to utilize SQL statement to carry out paging, for example get for the first time 1-50 bar record, get for the second time 51-100 bar record etc., although this mode has reached and has read limited record and be loaded in the internal memory at every turn, but the pressure of database side is still very large, because the inquiry of each SQL statement all is the scanning to the result set complete record, processing speed is not optimized; Another is to realize paging by code, for example utilizes the ResultSet result set to carry out searching loop among the JAVA and realizes, travels through 1-50 bar record and taking-up for the first time.For the second time travel through 1-100 bar record, but only take out 51-100 bar record, still there is the shortcoming of at every turn inquiring about in advance all records in this mode; Next also has a kind of by finding in advance the major key PK with the result set that satisfies condition, then deposit temporary table in and be numbered with sequence number, gather by the many batches of PK that read of sequence number afterwards, inquire data in the database utilizing PK to gather, although this mode has solved the problem of front, but since want many batches of from the database temporary table reading out data, in the concurrent situation of height, the pressure of database side is still very large, and have repeatedly middleware unit to the connection of database, inquiry, data network transmission, in the arrowband environment, still there are some bottlenecks in efficient, does not reasonably utilize in addition the middleware unit resource.After last above-mentioned three kinds of schemes all do not have to propose to load data in the internal memory, the speed of how processing with a kind of general further optimization data of mode, all be to consider to solve the bottleneck that data load in the whole algorithm, and often the big data quantity batch algorithms often has inquiry to load and two processes of data processing persistence, and how paging processes the automatic adaptation multiple database, and these all are problems.
So, how to solve the reasonable use of middleware unit resource and database resource in the big data quantity loading procedure, how to make paging bottom self-adaptation multitype database, how a whole set of solution and system are proposed, prevent that the middleware unit internal memory from overflowing, alleviating the database side processing pressure, reducing transmitted data on network amount between middleware unit and the database, this is the technical matters that needs to be resolved hurrily.
Summary of the invention
The present invention just is being based on the problems referred to above, has proposed a kind of big data quantity batch system, can prevent that the middleware unit internal memory from overflowing, alleviating the processing pressure of database side.
According to an aspect of the present invention, the invention provides a kind of big data quantity batch processing system, comprise: middleware unit, level cache device and L2 cache device, wherein, described middleware unit is used for sending query requests to described level cache device, and reception is from the secondary paging major key set of described L2 cache device, to the pending data of data base querying and after described pending data are carried out computing, send perdurable data request to described database according to described secondary paging major key set; Described level cache device is used for meeting to described data base querying the major key set of described query requests, and generates the set of one-level paging major key and described one-level paging major key set is back to described L2 cache device according to described major key set; Described L2 cache device is used for generating the set of secondary paging major key and described secondary paging major key set being back to described middleware unit according to described one-level paging major key set.
By technique scheme, in the process of middleware reading out data, add the two-level cache structure, optimization data reads greatly, has solved the technical matters that the middleware internal memory overflows.
In technique scheme, preferred, can also comprise: the first setting unit arranges the level cache threshold value of described level cache device; When described level cache device also is used for data volume in described major key set less than or equal to described level cache threshold value, directly described one-level paging major key set is back to described L2 cache device, and in the data volume of described major key set during greater than described level cache threshold value, set up and the insertion temporary table, described temporary table is carried out paging and the major key that obtains is back to described L2 cache device.
If only have the level cache structure to solve the problem that the middleware internal memory overflows, then must do more fine-grained control to every page of major key data volume, after having adopted the two-level cache structure, because the just major key that level cache returns, each major key is the character string of a regular length, committed memory is less, so can greatly improve the major key data total amount of every page on level cache structure.
In technique scheme, preferred, can also comprise: the second setting unit arranges the L2 cache threshold value of described L2 cache device; When described L2 cache device also is used for data volume at described one-level paging major key less than or equal to described L2 cache threshold value, directly described secondary paging major key set is back to described middleware unit, and in the data volume of described major key set during greater than described L2 cache threshold value, described secondary paging major key set is temporary in internal memory, from described internal memory, take out every one page major key data, according to the described pending data of described every one page major key data query.
Based on the occupancy of middleware actual treatment data the L2 cache threshold value of L2 cache device is set, the storage threshold that buffer structures at different levels rationally are set is the treatment effeciency of elevator system to greatest extent.
In the technique scheme, preferred, described middleware unit comprises: affairs are set up subelement, are used for setting up standalone transaction; The subelement that locks is used for described pending data are added middleware unit rank major key lock, and described pending data are processed, and after processing finishes, described middleware unit rank is locked into capable release.
Each page data adopts standalone transaction to process, that is to say that every page data rear affairs that are disposed submit to immediately, rather than only play affairs at whole algorithm outermost layer, can be to the locking that locks for a long time of all data in the database, thereby promote the database integral concurrent processing capacity, reduced the pressure of database side.
In above-mentioned arbitrary technical scheme, preferred, can also comprise: identification device makes described level cache device-adaptive multi-type database.
According to a further aspect in the invention, a kind of big data quantity batch processing method also is provided, may further comprise the steps: step 402, middleware unit sends query requests to the level cache device, and database returns the major key that meets described query requests and gathers to described level cache device; Step 404, described level cache device generates the set of one-level paging major key and described one-level paging major key set is back to the L2 cache device according to described major key set; Step 406, described L2 cache device generates the set of secondary paging major key and described secondary paging major key set is back to described middleware unit according to described one-level paging major key set; Step 408, described middleware unit to the pending data of described data base querying and after more described pending data being carried out computing, sends perdurable data request to described database according to described secondary paging major key set.
By technique scheme, in the process of middleware reading out data, add the two-level cache structure, optimization data reads greatly, has solved the technical matters that the middleware internal memory overflows.
In technique scheme, preferred, described step 404 specifically comprises: the level cache threshold value that described level cache device is set; During less than or equal to described level cache threshold value, directly described one-level paging major key set is back to described L2 cache device in the data volume of described major key set; , set up and also insert temporary table during greater than described level cache threshold value in the data volume of described major key set, described temporary table is carried out paging and the major key that obtains is back to described L2 cache device.
If only have the level cache structure to solve the problem that the middleware internal memory overflows, then must do more fine-grained control to every page of major key data volume, after having adopted the two-level cache structure, because the just major key that level cache returns, each major key is the character string of a regular length, committed memory is less, so can greatly improve the major key data total amount of every page on level cache structure.
In technique scheme, preferred, described step 406 specifically comprises: the L2 cache threshold value that described L2 cache device is set; During less than or equal to described L2 cache threshold value, directly described secondary paging major key set is back to described middleware unit in the data volume of described one-level paging major key; During greater than described L2 cache threshold value, described secondary paging major key set is temporary in internal memory in the data volume of described major key set, from described internal memory, takes out every one page major key data, according to the described pending data of described every one page major key data query.
Based on the occupancy of middleware actual treatment data the L2 cache threshold value of L2 cache device is set, the storage threshold that buffer structures at different levels rationally are set is the treatment effeciency of elevator system to greatest extent.
In technique scheme, preferably, described step 408 specifically comprises: set up standalone transaction in described middleware unit, described pending data are added middleware unit rank major key lock, described pending data are processed, after processing finishes, described middleware unit rank is locked into capable release.
Each page data adopts standalone transaction to process, that is to say that every page data rear affairs that are disposed submit to immediately, rather than only play affairs at whole algorithm outermost layer, can be to the locking that locks for a long time of all data in the database, thereby promote the database integral concurrent processing capacity, reduced the pressure of database side.
In above-mentioned arbitrary technical scheme, preferred, described step 404 can also comprise, at described level cache device place, adopts identification device self-adaptation multi-type database.
Therefore, can greatly improve system to the processing speed of large-data operation according to big data quantity batch processing method of the present invention, balance is used middleware and database resource to the full extent, under the circumstances that reduces separately load, take full advantage of again separately resource, to reach the maximum lift of system performance
Description of drawings
Fig. 1 shows the schematic diagram of big data quantity batch processing in the correlation technique;
Fig. 2 shows the according to an embodiment of the invention block diagram of big data quantity batch processing system;
Fig. 3 shows the according to an embodiment of the invention schematic diagram of big data quantity batch processing;
Fig. 4 shows the according to an embodiment of the invention process flow diagram of big data quantity batch processing method;
Fig. 5 shows the according to an embodiment of the invention process flow diagram of big data quantity batch processing method.
Embodiment
In order more clearly to understand above-mentioned purpose of the present invention, feature and advantage, below in conjunction with the drawings and specific embodiments the present invention is further described in detail.
Set forth in the following description a lot of details so that fully understand the present invention, still, the present invention can also adopt other to be different from other modes described here and implement, and therefore, the present invention is not limited to the restriction of following public specific embodiment.
Before explanation is according to big data quantity batch processing system of the present invention, simply introduce first existing large Data Matching processing procedure.
As shown in Figure 1, general big data quantity batch processing services scene, all processing logics and algorithm all roughly are divided into following several process: middleware is initiated the request that inquiry loads data to database, middleware obtains data and in the internal memory computing, the most backward database was initiated the request of perdurable data after processing finished, database is finished the persistence operation, and such processing procedure is easy to cause the middleware internal memory to overflow.In order to solve this technical problem, disclose according to big data quantity batch processing system of the present invention.
Fig. 2 shows the according to an embodiment of the invention block diagram of big data quantity batch processing system.
As shown in Figure 2, big data quantity batch processing system 200 according to the embodiment of the invention comprises: middleware unit 202, level cache device 204 and L2 cache device 206, wherein, described middleware unit 202 is used for sending query requests to described level cache device 204, and reception is from the secondary paging major key set of described L2 cache device 206, to the pending data of data base querying and after described pending data are carried out computing, send perdurable data request to described database according to described secondary paging major key set; Described level cache device 204 is used for meeting to described data base querying the major key set of described query requests, and generates the set of one-level paging major key and described one-level paging major key set is back to described L2 cache device 206 according to described major key set; Described L2 cache device 206 is used for generating the set of secondary paging major key and described secondary paging major key set being back to described middleware unit 202 according to described one-level paging major key set.
By technique scheme, in the process of middleware reading out data, add the two-level cache structure, optimization data reads greatly, has solved the technical matters that the middleware internal memory overflows.
In the technique scheme, preferred, can also comprise: the first setting unit 208 arranges the level cache threshold value of described level cache device 204; When described level cache device 204 also is used for data volume in described major key set less than or equal to described level cache threshold value, directly described one-level paging major key set is back to described L2 cache device 206, and in the data volume of described major key set during greater than described level cache threshold value, set up and the insertion temporary table, described temporary table is carried out paging and the major key that obtains is back to described L2 cache device 206.
If only have the level cache structure to solve the problem that the middleware internal memory overflows, then must do more fine-grained control to every page of major key data volume, after having adopted the two-level cache structure, because the just major key that level cache returns, each major key is the character string of a regular length, committed memory is less, so can greatly improve the major key data total amount of every page on level cache structure.
Preferably, big data quantity batch processing system 200 can also comprise: the second setting unit 210 arranges the L2 cache threshold value of described L2 cache device 206; When described L2 cache device 206 also is used for data volume at described one-level paging major key less than or equal to described L2 cache threshold value, directly described secondary paging major key set is back to described middleware unit 202, and in the data volume of described major key set during greater than described L2 cache threshold value, described secondary paging major key set is temporary in internal memory, from described internal memory, take out every one page major key data, according to the described pending data of described every one page major key data query.
Based on the occupancy of middleware actual treatment data the L2 cache threshold value of L2 cache device 206 is set, the storage threshold that buffer structures at different levels rationally are set is the treatment effeciency of elevator system to greatest extent.
In the technique scheme, preferred, described middleware unit 202 comprises: affairs are set up subelement 2022, are used for setting up standalone transaction; The subelement 2024 that locks is used for described pending data are added middleware unit 202 rank major keys lock, and described pending data are processed, and after processing finishes, described middleware unit 202 ranks is locked into capable release.
Each page data adopts standalone transaction to process, that is to say that every page data rear affairs that are disposed submit to immediately, rather than only play affairs at whole algorithm outermost layer, can be to the locking that locks for a long time of all data in the database, thereby promote the database integral concurrent processing capacity, reduced the pressure of database side.
Preferably, big data quantity batch processing system 200 can also comprise: identification device 212 makes described level cache device 204 self-adaptation multi-type databases.
Comprehensively above-mentioned, whole big data quantity batch processing system can be divided into following several module and mutually transmit the data co-ordination: level cache device solves middleware memory bottleneck; Database identification device automatic adaptation multi-type database; Rationalize when the L2 cache device reduces database loads pressure and use the middleware resource; The standalone transaction treating apparatus further promotes database and many concurrent processing of middleware ability, thus Integral lifting system efficient.
Below in conjunction with the handling principle of Fig. 3 detailed description according to big data quantity batch processing system of the present invention.Fig. 3 shows the according to an embodiment of the invention schematic diagram of big data quantity batch processing.
As shown in Figure 3, add the two-level cache structure in the process of middleware (middleware unit in the corresponding diagram 2) reading out data, optimization data reads; Initiate to adopt standalone transaction in the persistence processing procedure at middleware, optimize and alleviate database side pressure.Whole process is as follows as we can see from the figure:
1. initiate query requests to the level cache device.
2. the level cache device satisfies the major key of querying condition to data base querying, and database returns all major keys and gathers the level cache structure.
3. the L2 cache device is gathered to level cache device acquisition request one-level paging major key.
4. the L2 cache device receives the secondary paging major key set of being returned by the level cache device.
5. middleware is to the combination of L2 cache device acquisition request secondary paging major key.
6. the L2 cache device returns the set of secondary paging major key to middleware.
7. middleware is inquired about pending data according to secondary paging major key to database combination.
8. adopt standalone transaction to process computational data.
9. persistence secondary paged data.
Wherein, the 2nd step to the 4th step is first order paging circulating treatment procedure; The 5th step to the 9th step is second level paging circulating treatment procedure, and the inner standalone transaction that adopts alleviates the database side processing pressure simultaneously.Level cache device inner utilization database temporary table is realized in the system, and the inner identification device 212 that comprises automatic adaptation underlying database is applicable to types of databases simultaneously.The L2 cache structure adopts and sets up internal memory level buffer memory in the middleware rank, temporal data major key information.
If only adopt the level cache device to prevent that the middleware internal memory from overflowing, must do more fine-grained control to every page of major key data volume, for example must process the data volume of total amount 1,000 ten thousand, for guaranteeing that internal memory does not overflow, must accomplish every page of maximum 5000 records, corresponding major key also is 5000, so just needs 2000 pages, namely 2000 paging queries.
And if adopt now the two-level cache structure, same every page of maximum 5000 data of L2 cache, corresponding major key also is 5000, but because the just major key that the level cache device returns, each major key is the character string of a regular length, committed memory is less, so its every page of major key data can reach for example 40,000 records, because L2 cache paging peek is to finish in internal memory fully, do not have remote inquiry, the query cost that brings of level cache only has so: 1,000 ten thousand/40,000=250, and that is to say and altogether only have paging query 250 times.Compare 2000 paging queries that only have the level cache device, the method of employing L2 cache structure can be less to the pressure of database side, also reduce simultaneously the Internet Transmission flow between middleware and the database, thereby further promoted the ability of whole system big data quantity batch processing.
Next describe in detail according to big data quantity batch processing method of the present invention in conjunction with Fig. 4 and Fig. 5.
Fig. 4 shows the according to an embodiment of the invention process flow diagram of big data quantity batch processing method.
As shown in Figure 4, big data quantity batch processing method according to an embodiment of the invention may further comprise the steps: step 402, and middleware unit sends query requests to the level cache device, and database returns the major key that meets query requests and gathers to the level cache device; Set generates the set of one-level paging major key and the set of one-level paging major key is back to the L2 cache device according to major key for step 404, level cache device; Set generates the set of secondary paging major key and the set of secondary paging major key is back to middleware unit according to one-level paging major key for step 406, L2 cache device; Step 408, middleware unit to the pending data of data base querying and after treating deal with data again and carrying out computing, sends perdurable data request to database according to secondary paging major key set.
By technique scheme, in the process of middleware reading out data, add the two-level cache structure, optimization data reads greatly, has solved the technical matters that the middleware internal memory overflows.
In the technique scheme, preferred, described step 404 specifically comprises: the level cache threshold value that described level cache device is set; During less than or equal to described level cache threshold value, directly described one-level paging major key set is back to described L2 cache device in the data volume of described major key set; , set up and also insert temporary table during greater than described level cache threshold value in the data volume of described major key set, described temporary table is carried out paging and the major key that obtains is back to described L2 cache device.
If only have the level cache structure to solve the problem that the middleware internal memory overflows, then must do more fine-grained control to every page of major key data volume, after having adopted the two-level cache structure, because the just major key that level cache returns, each major key is the character string of a regular length, committed memory is less, so can greatly improve the major key data total amount of every page on level cache structure.
In the technique scheme, preferred, described step 406 specifically comprises: the L2 cache threshold value that described L2 cache device is set; During less than or equal to described L2 cache threshold value, directly described secondary paging major key set is back to described middleware unit in the data volume of described one-level paging major key; During greater than described L2 cache threshold value, described secondary paging major key set is temporary in internal memory in the data volume of described major key set, from described internal memory, takes out every one page major key data, according to the described pending data of described every one page major key data query.
Based on the occupancy of middleware actual treatment data the L2 cache threshold value of L2 cache device is set, the storage threshold that buffer structures at different levels rationally are set is the treatment effeciency of elevator system to greatest extent.
In the technique scheme, preferably, described step 408 specifically comprises: set up standalone transaction in described middleware unit, described pending data are added middleware unit rank major key lock, described pending data are processed, after processing finishes, described middleware unit rank is locked into capable release.
Each page data adopts standalone transaction to process, that is to say that every page data rear affairs that are disposed submit to immediately, rather than only play affairs at whole algorithm outermost layer, can be to the locking that locks for a long time of all data in the database, thereby promote the database integral concurrent processing capacity, reduced the pressure of database side.
In above-mentioned arbitrary technical scheme, preferred, described step 404 can also comprise, at described level cache device place, adopts identification device self-adaptation multi-type database.
As shown in Figure 5, the big data quantity batch process can roughly be divided into 3 processes: 1) the level cache structure is processed query requests; 2) the L2 cache structure is processed query requests; 3) use standalone transaction to submit final data to
1) the level cache structure is processed query requests.
1. after the level cache structure receives query requests, at first carry out SQL statement and obtain result set.
2. traversing result collection is if the total size of the data volume of result set surpasses level cache threshold values, directly return results collection.
3. if the total size of the data volume of result set surpasses level cache threshold values, temporary table buffer storage treatment S QL statement.
4. the temporary table buffer storage automatically creates various type of database and inserts the temporary table SQL statement according to the bottom data Source Type.
The field of temporary table is as follows: number (self-propagation type), major key, import the field among the temporary table buffer memory SQL into.Wherein, numbering is to use for follow-up paging, and every kind of database self-propagation type-word section realizes that technology is variant, so can be according to the automatic differentiated treatment of type of database at this, the final SQL statement that forms the narration interspersed with flashbacks temporary table is similar: insert into temp (select rownumFrom ...).
The level cache structure internally in the temporary table paging take out pending data, the principle of paging is utilized number field in the temporary table exactly, since number field be the self-propagation type (for example, 12,3,4 ...), so the similar select pk of the SQL from temp where no of paging peek 〉=1and no＜=50 ...
6. the major key data acquisition that at last paging is taken out is passed to L2 cache structure (being the L2 cache device).
2) the L2 cache structure is processed query requests.
1. the L2 cache structure receives the major key data that the level cache structure is returned.
2. if the total size of the data volume of result set does not surpass the L2 cache threshold values, then directly return the major key result set.
3. if the total size of the data volume of result set surpasses the L2 cache threshold values, then the major key data are temporarily stored in the internal memory.
4. every one page major key data are taken out in the secondary paging from internal memory level buffer memory.
5. go to inquire in the database pending data according to every one page major key data.
3) use standalone transaction to submit data to.
1. create standalone transaction at middleware layer.
2. treat deal with data and add middleware rank major key lock.
To data calculate, last perdurable data.
4. the lock of feeling relieved.
4) each buffer structure threshold values arranges.
1. level cache structure acquiescence paged data threshold values 20000.
2. L2 cache structure acquiescence paged data threshold values 5000.
3. the threshold values of level cache structure and L2 cache structure can dynamically arrange according to the hardware condition of middleware.
4. level cache structure threshold values arranges main consideration major key character EMS memory occupation amount.
5. the occupancy that actual treatment data in the main consideration middleware are set of L2 cache structure threshold values.The treatment effeciency of the elevator system that the threshold values of buffer structures at different levels can be maximum rationally is set.
Therefore, can greatly improve system to the processing speed of large-data operation according to big data quantity batch processing method of the present invention, balance is used middleware and database resource to the full extent, under the circumstances that reduces separately load, take full advantage of again separately resource, to reach the maximum lift of system performance.To sum up, the method this so that infosystem can better adapt to the network environment more, that condition is harsher, big data quantity environment, can make client's operational system under larger business datum scene.
The above is the preferred embodiments of the present invention only, is not limited to the present invention, and for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any modification of doing, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a big data quantity batch processing system is characterized in that, comprising: middleware unit, level cache device and L2 cache device, wherein,
Described middleware unit is used for sending query requests to described level cache device, and reception is from the secondary paging major key set of described L2 cache device, to the pending data of data base querying and after described pending data are carried out computing, send perdurable data request to described database according to described secondary paging major key set;
Described level cache device is used for meeting to described data base querying the major key set of described query requests, and generates the set of one-level paging major key and described one-level paging major key set is back to described L2 cache device according to described major key set;
Described L2 cache device is used for generating the set of secondary paging major key and described secondary paging major key set being back to described middleware unit according to described one-level paging major key set.
2. big data quantity batch processing system according to claim 1 is characterized in that, also comprises: the first setting unit arranges the level cache threshold value of described level cache device;
When described level cache device also is used for data volume in described major key set less than or equal to described level cache threshold value, directly described one-level paging major key set is back to described L2 cache device, and in the data volume of described major key set during greater than described level cache threshold value, set up and the insertion temporary table, described temporary table is carried out paging and the major key that obtains is back to described L2 cache device.
3. big data quantity batch processing system according to claim 1 is characterized in that, also comprises:
The second setting unit arranges the L2 cache threshold value of described L2 cache device;
When described L2 cache device also is used for data volume at described one-level paging major key less than or equal to described L2 cache threshold value, directly described secondary paging major key set is back to described middleware unit, and in the data volume of described major key set during greater than described L2 cache threshold value, described secondary paging major key set is temporary in internal memory, from described internal memory, take out every one page major key data, according to the described pending data of described every one page major key data query.
4. big data quantity batch processing system according to claim 3 is characterized in that, described middleware unit comprises:
Affairs are set up subelement, are used for setting up standalone transaction;
The subelement that locks is used for described pending data are added middleware unit rank major key lock, and described pending data are processed, and after processing finishes, described middleware unit rank is locked into capable release.
5. each described big data quantity batch processing system in 4 according to claim 1 is characterized in that also comprise: identification device makes described level cache device-adaptive multi-type database.
6. a big data quantity batch processing method is characterized in that, may further comprise the steps:
Step 402, middleware unit sends query requests to the level cache device, and database returns the major key that meets described query requests and gathers to described level cache device;
Step 404, described level cache device generates the set of one-level paging major key and described one-level paging major key set is back to the L2 cache device according to described major key set;
Step 406, described L2 cache device generates the set of secondary paging major key and described secondary paging major key set is back to described middleware unit according to described one-level paging major key set;
Step 408, described middleware unit to the pending data of described data base querying and after more described pending data being carried out computing, sends perdurable data request to described database according to described secondary paging major key set.
7. big data quantity batch processing method according to claim 6 is characterized in that, described step 404 specifically comprises: the level cache threshold value that described level cache device is set;
During less than or equal to described level cache threshold value, directly described one-level paging major key set is back to described L2 cache device in the data volume of described major key set;
, set up and also insert temporary table during greater than described level cache threshold value in the data volume of described major key set, described temporary table is carried out paging and the major key that obtains is back to described L2 cache device.
8. big data quantity batch processing method according to claim 6 is characterized in that, described step 406 specifically comprises: the L2 cache threshold value that described L2 cache device is set;
During less than or equal to described L2 cache threshold value, directly described secondary paging major key set is back to described middleware unit in the data volume of described one-level paging major key;
During greater than described L2 cache threshold value, described secondary paging major key set is temporary in internal memory in the data volume of described major key set, from described internal memory, takes out every one page major key data, according to the described pending data of described every one page major key data query.
9. big data quantity batch processing method according to claim 6, it is characterized in that, described step 408 specifically comprises: set up standalone transaction in described middleware unit, described pending data are added middleware unit rank major key lock, described pending data are processed, after processing finishes, described middleware unit rank is locked into capable release.
10. each described big data quantity batch processing method in 9 according to claim 6 is characterized in that described step 404 also comprises, at described level cache device place, adopts identification device self-adaptation multi-type database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210480063.2A CN103020151B (en) | 2012-11-22 | 2012-11-22 | Big data quantity batch processing system and big data quantity batch processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210480063.2A CN103020151B (en) | 2012-11-22 | 2012-11-22 | Big data quantity batch processing system and big data quantity batch processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103020151A true CN103020151A (en) | 2013-04-03 |
| CN103020151B CN103020151B (en) | 2015-12-02 |
Family
ID=47968755
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210480063.2A Active CN103020151B (en) | 2012-11-22 | 2012-11-22 | Big data quantity batch processing system and big data quantity batch processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103020151B (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103218179A (en) * | 2013-04-23 | 2013-07-24 | 深圳市京华科讯科技有限公司 | Second-level system acceleration method based on virtualization |
| CN103888378A (en) * | 2014-04-09 | 2014-06-25 | 北京京东尚科信息技术有限公司 | Data exchange system and method based on cache mechanism |
| CN103886022A (en) * | 2014-02-24 | 2014-06-25 | 上海上讯信息技术股份有限公司 | Paging-query querying device and method based on primary key fields |
| CN104424319A (en) * | 2013-09-10 | 2015-03-18 | 镇江金钛软件有限公司 | Method for temporarily storing general data |
| CN104866434A (en) * | 2015-06-01 | 2015-08-26 | 北京圆通慧达管理软件开发有限公司 | Multi-application-oriented data storage system and data storage and calling method |
| CN106407020A (en) * | 2016-11-23 | 2017-02-15 | 青岛海信移动通信技术股份有限公司 | Database processing method of mobile terminal and mobile terminal thereof |
| CN106407019A (en) * | 2016-11-23 | 2017-02-15 | 青岛海信移动通信技术股份有限公司 | Database processing method of mobile terminal and mobile terminal thereof |
| CN107609068A (en) * | 2017-08-30 | 2018-01-19 | 苏州朗动网络科技有限公司 | A kind of noninductive moving method of data |
| CN108090086A (en) * | 2016-11-21 | 2018-05-29 | 迈普通信技术股份有限公司 | Paging query method and device |
| CN104111962B (en) * | 2013-04-22 | 2018-09-18 | Sap欧洲公司 | Enhanced affairs cache with batch operation |
| CN109710639A (en) * | 2018-11-26 | 2019-05-03 | 厦门市美亚柏科信息股份有限公司 | A kind of search method based on pair buffers, device and storage medium |
| CN109828834A (en) * | 2018-12-14 | 2019-05-31 | 泰康保险集团股份有限公司 | The method and system and its computer-readable intermediate value and electronic equipment of batch processing |
| CN109889336A (en) * | 2019-03-08 | 2019-06-14 | 浙江齐治科技股份有限公司 | A kind of middleware obtains the method, apparatus and system of password |
| CN109165090B (en) * | 2018-09-27 | 2019-07-05 | 苏宁消费金融有限公司 | Batch processing method and system based on statement |
| CN110457540A (en) * | 2019-06-28 | 2019-11-15 | 卓尔智联(武汉)研究院有限公司 | Querying method, service platform, terminal device and the storage medium of data |
| CN113312382A (en) * | 2021-05-31 | 2021-08-27 | 上海万物新生环保科技集团有限公司 | Method, device and system for database paging query |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050120181A1 (en) * | 2003-12-02 | 2005-06-02 | Oracle International Corporation | Invalidating cached data using secondary keys |
| US20060259479A1 (en) * | 2005-05-12 | 2006-11-16 | Microsoft Corporation | System and method for automatic generation of suggested inline search terms |
| CN101216840A (en) * | 2008-01-21 | 2008-07-09 | 金蝶软件（中国）有限公司 | Data enquiry method and data enquiry system |
| CN101860449A (en) * | 2009-04-09 | 2010-10-13 | 华为技术有限公司 | Data query method, device and system |
| CN201993755U (en) * | 2011-01-30 | 2011-09-28 | 上海振华重工(集团)股份有限公司 | Data filtration, compression and storage system of real-time database |
-
2012
- 2012-11-22 CN CN201210480063.2A patent/CN103020151B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050120181A1 (en) * | 2003-12-02 | 2005-06-02 | Oracle International Corporation | Invalidating cached data using secondary keys |
| US20060259479A1 (en) * | 2005-05-12 | 2006-11-16 | Microsoft Corporation | System and method for automatic generation of suggested inline search terms |
| CN101216840A (en) * | 2008-01-21 | 2008-07-09 | 金蝶软件（中国）有限公司 | Data enquiry method and data enquiry system |
| CN101860449A (en) * | 2009-04-09 | 2010-10-13 | 华为技术有限公司 | Data query method, device and system |
| CN201993755U (en) * | 2011-01-30 | 2011-09-28 | 上海振华重工(集团)股份有限公司 | Data filtration, compression and storage system of real-time database |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104111962B (en) * | 2013-04-22 | 2018-09-18 | Sap欧洲公司 | Enhanced affairs cache with batch operation |
| CN103218179A (en) * | 2013-04-23 | 2013-07-24 | 深圳市京华科讯科技有限公司 | Second-level system acceleration method based on virtualization |
| CN104424319A (en) * | 2013-09-10 | 2015-03-18 | 镇江金钛软件有限公司 | Method for temporarily storing general data |
| CN103886022B (en) * | 2014-02-24 | 2019-01-18 | 上海上讯信息技术股份有限公司 | A kind of query facility and its method carrying out paging query based on major key field |
| CN103886022A (en) * | 2014-02-24 | 2014-06-25 | 上海上讯信息技术股份有限公司 | Paging-query querying device and method based on primary key fields |
| CN103888378B (en) * | 2014-04-09 | 2017-08-25 | 北京京东尚科信息技术有限公司 | A kind of data exchange system and method based on caching mechanism |
| CN103888378A (en) * | 2014-04-09 | 2014-06-25 | 北京京东尚科信息技术有限公司 | Data exchange system and method based on cache mechanism |
| CN104866434A (en) * | 2015-06-01 | 2015-08-26 | 北京圆通慧达管理软件开发有限公司 | Multi-application-oriented data storage system and data storage and calling method |
| CN104866434B (en) * | 2015-06-01 | 2017-10-03 | 明算科技（北京）股份有限公司 | Towards data-storage system and data storage, the call method applied more |
| CN107273522A (en) * | 2015-06-01 | 2017-10-20 | 明算科技（北京）股份有限公司 | Towards the data-storage system and data calling method applied more |
| CN107273522B (en) * | 2015-06-01 | 2020-01-14 | 明算科技（北京）股份有限公司 | Multi-application-oriented data storage system and data calling method |
| CN108090086B (en) * | 2016-11-21 | 2022-02-22 | 迈普通信技术股份有限公司 | Paging query method and device |
| CN108090086A (en) * | 2016-11-21 | 2018-05-29 | 迈普通信技术股份有限公司 | Paging query method and device |
| CN106407019A (en) * | 2016-11-23 | 2017-02-15 | 青岛海信移动通信技术股份有限公司 | Database processing method of mobile terminal and mobile terminal thereof |
| CN106407020A (en) * | 2016-11-23 | 2017-02-15 | 青岛海信移动通信技术股份有限公司 | Database processing method of mobile terminal and mobile terminal thereof |
| CN107609068A (en) * | 2017-08-30 | 2018-01-19 | 苏州朗动网络科技有限公司 | A kind of noninductive moving method of data |
| CN107609068B (en) * | 2017-08-30 | 2021-03-16 | 企查查科技有限公司 | Data non-inductive migration method |
| CN109165090B (en) * | 2018-09-27 | 2019-07-05 | 苏宁消费金融有限公司 | Batch processing method and system based on statement |
| CN109710639A (en) * | 2018-11-26 | 2019-05-03 | 厦门市美亚柏科信息股份有限公司 | A kind of search method based on pair buffers, device and storage medium |
| CN109828834A (en) * | 2018-12-14 | 2019-05-31 | 泰康保险集团股份有限公司 | The method and system and its computer-readable intermediate value and electronic equipment of batch processing |
| CN109889336A (en) * | 2019-03-08 | 2019-06-14 | 浙江齐治科技股份有限公司 | A kind of middleware obtains the method, apparatus and system of password |
| CN109889336B (en) * | 2019-03-08 | 2022-06-14 | 浙江齐治科技股份有限公司 | Method, device and system for middleware to acquire password |
| CN110457540A (en) * | 2019-06-28 | 2019-11-15 | 卓尔智联(武汉)研究院有限公司 | Querying method, service platform, terminal device and the storage medium of data |
| CN113312382A (en) * | 2021-05-31 | 2021-08-27 | 上海万物新生环保科技集团有限公司 | Method, device and system for database paging query |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103020151B (en) | 2015-12-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103020151B (en) | Big data quantity batch processing system and big data quantity batch processing method | |
| US11126626B2 (en) | Massively parallel and in-memory execution of grouping and aggregation in a heterogeneous system | |
| CN104111958B (en) | A kind of data query method and device | |
| US5692182A (en) | Bufferpool coherency for identifying and retrieving versions of workfile data using a producing DBMS and a consuming DBMS | |
| CN1708757B (en) | A transparent edge-of-network data cache | |
| CN1300692C (en) | Dynamic and automatic memory management | |
| US6785675B1 (en) | Aggregation of resource requests from multiple individual requestors | |
| WO2012060889A1 (en) | Systems and methods for grouped request execution | |
| US20160140354A1 (en) | Dbfs permissions using user, role, and permissions flags | |
| US8949222B2 (en) | Changing the compression level of query plans | |
| CN104778270A (en) | Storage method for multiple files | |
| CN104679898A (en) | Big data access method | |
| CN113419823B (en) | Alliance chain system suitable for high concurrency transaction and design method thereof | |
| AU2005239366A1 (en) | Partial query caching | |
| CA2822900A1 (en) | Filtering queried data on data stores | |
| CN101216840A (en) | Data enquiry method and data enquiry system | |
| US10007800B2 (en) | Remote rule execution | |
| CN113420052B (en) | Multi-level distributed cache system and method | |
| CN106685902A (en) | User authority management method, client and server | |
| CN107562804B (en) | Data caching service system and method and terminal | |
| CN115712670A (en) | Data source management system | |
| WO2022127866A1 (en) | Data processing method and apparatus, and electronic device and storage medium | |
| CN114020779A (en) | Self-adaptive optimization retrieval performance database and data query method | |
| Franaszek et al. | Distributed concurrency control based on limited wait-depth | |
| CN116027982A (en) | Data processing method, device and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100094 Haidian District North Road, Beijing, No. 68 Applicant after: Yonyou Network Technology Co., Ltd. Address before: 100094 Beijing city Haidian District North Road No. 68, UFIDA Software Park Applicant before: UFIDA Software Co., Ltd. |
|
| COR | Change of bibliographic data | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |