CN103929472A - Data processing method, device and system - Google Patents
Data processing method, device and system Download PDFInfo
- Publication number
- CN103929472A CN103929472A CN201410107750.9A CN201410107750A CN103929472A CN 103929472 A CN103929472 A CN 103929472A CN 201410107750 A CN201410107750 A CN 201410107750A CN 103929472 A CN103929472 A CN 103929472A
- Authority
- CN
- China
- Prior art keywords
- data
- change event
- event
- data change
- caching system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a data processing method, device and system, and belongs to the field of data storage. The method comprises the steps that a data release platform obtains a log record of a back-end database from the back-end database; the data release platform obtains a data changing event from the log record; the data release platform provides the data changing event for a cache system, and the cache system processes cached data according to the data changing event. According to the data processing method, device and system, the problem that when the data cached in the cache system are changed in a MySQL database, data read by a client terminal from the cache system are old data or error data is solved, and the aims that the cache system can synchronously update the data or delete the changed data and cannot provide the old data or error data for the client terminal are achieved.
Description
Technical field
The present invention relates to field of data storage, particularly a kind of data processing method, Apparatus and system.
Background technology
MySQL database is a kind of at internet industry application database very widely.But because MySQL database is a kind of disk database, cannot provide quick high performance service, make simple MySQL cannot tackle the data access pressure of how much speed growths.
In order to solve data access pressure, in correlation technique, adopt caching system to carry out expedited data access speed.Please refer to Fig. 1, it shows the structural representation of a kind of Database Systems in correlation technique.These Database Systems comprise: caching system 120 and MySQL database 140.Wherein, MySQL database 140 is for storing total data in rear end.Caching system 120 is receiving after the data access of client, detects this locality and whether is cached with these data, if be not cached with this data, these data read to local cache from MySQL database 140, and data access is made to response.Also, caching system 120 can be buffered in this locality by the hot spot data of frequently access.Because the data in MySQL database 140 are stored in disk, and data in caching system 120 are stored in internal memory, so caching system 120 can provide quick high performance service.
Realizing in process of the present invention, inventor finds that prior art at least exists following problem: because the data in MySQL database 140 are to change at any time.If the data that have been buffered in caching system 120, in MySQL database 140, variation have occurred, the data that client reads from caching system 120 will be legacy data or misdata.Although the mechanism that caching system 120 provides the data of buffer memory to lose efficacy in the given time, this mechanism cannot address the above problem completely.
Summary of the invention
In MySQL database, there is variation in order to solve the data that are buffered in caching system, the data that client reads from caching system are by the problem that is legacy data or misdata, and the embodiment of the present invention provides a kind of data processing method, Apparatus and system.Described technical scheme is as follows:
First aspect, provides a kind of data processing method, and described method comprises:
Obtain the log recording of described back-end data base from back-end data base;
From described log recording, obtain data change event;
Described data change event is offered to caching system, so that described caching system is processed the data of buffer memory according to described data change event.
Alternatively, before the described log recording that obtains described back-end data base from described back-end data base, also comprise:
To described back-end data base login account, described account has the authority of the log recording that copies described back-end data base;
Send log recording by described account to described back-end data base and obtain request.
Alternatively, described described data change event is offered to caching system, comprising:
Described data change event is kept in relaying log recording;
Receive the subscribe request of described caching system to described data change event;
Issue described data change event according to described subscribe request to described caching system.
Alternatively, describedly issue described data change event according to described subscribe request to described caching system, comprising:
Described data change event is distributed in the subscription device setting in advance according to described subscribe request;
And/or,
According to described subscribe request, described data change event is distributed in universal information queue.
Second aspect, provides a kind of data processing method, and described method comprises:
Obtain data change event from data publishing platform, described data change event be described data publishing platform from back-end data base obtains the log recording of described back-end data base, from described log recording, obtain;
According to described data change event, the data of buffer memory are processed.
Alternatively, describedly obtain data change event from data publishing platform, comprising:
Subscribe to described data change event to described data publishing platform;
Obtain the described data change event that described data publishing platform is issued according to described subscribe request.
Alternatively, described in obtain the described data change record that described data publishing platform is issued according to described subscribe request, comprising:
From the subscription device setting in advance, obtain the described data change event that described data publishing platform is issued according to described subscribe request;
And/or,
From universal information queue, obtain the described data change event that described data publishing platform is issued according to described subscribe request.
Alternatively, describedly according to described data change event, the data of buffer memory are processed, being comprised:
According to described data change event, the data of buffer memory are upgraded to processing;
And/or,
According to described data change event, the data of buffer memory are deleted to processing.
The third aspect, provides a kind of data processing equipment, and described device comprises:
Record acquisition module, for obtain the log recording of described back-end data base from back-end data base;
Event extraction module, for obtaining data change event from described log recording;
Event provides module, for described data change event is offered to caching system, so that described caching system is processed the data of buffer memory according to described data change event.
Alternatively, described device, also comprises:
Account Registration module and request sending module;
Described Account Registration module, for to described back-end data base login account, described account has the authority of the log recording that copies described back-end data base;
Described request sending module, obtains request for sending log recording by described account to described back-end data base.
Alternatively, described event provides module, comprising:
Event storage unit, subscription receiving element and event release unit;
Described event storage unit, for being kept at relaying log recording by described data change event;
Described subscription receiving element, for receiving the subscribe request of described caching system to described data change event;
Described event release unit, for issuing described data change event according to described subscribe request to described caching system.
Alternatively, described event release unit, comprising:
First issues subelement, and/or second issues subelement;
Described first issues subelement, for described data change event being distributed on according to described subscribe request to the subscription device setting in advance;
Described second issues subelement, for described data change event being distributed on to universal information queue according to described subscribe request.
Fourth aspect, provides a kind of data processing equipment, and described device, comprising:
Event acquisition module, for obtain data change event from data publishing platform, described data change record be described data publishing platform from back-end data base obtains the log recording of described back-end data base, from described log recording, obtain;
Data processing module, for processing the data of buffer memory according to described data change event.
Alternatively, described event acquisition module, comprising:
Event subscription unit and event acquiring unit;
Described event subscription unit, for subscribing to described data change event to described data publishing platform;
Described event acquiring unit, the described data change event of issuing according to described subscribe request for obtaining described data publishing platform.
Alternatively, described event acquiring unit, comprising:
First obtains subelement, and/or second obtains subelement;
Described first obtains subelement, obtains for the subscription device from setting in advance the described data change event that described data publishing platform is issued according to described subscribe request;
Described second obtains subelement, the described data change event of issuing according to described subscribe request for obtain described data publishing platform from universal information queue.
Alternatively, described data processing module, comprising: upgrade processing unit, and/or, delete processing unit;
Described renewal processing unit, for upgrading processing according to described data change event to the data of buffer memory;
Described deletion processing unit, for deleting processing according to described data change event to the data of buffer memory.
The 5th aspect, provides a kind of data handling system, and described system comprises: data publishing platform, caching system and back-end data base; Described data publishing platform is connected with described back-end data base with described caching system respectively, and described caching system is connected with described back-end data base;
Described data publishing platform, comprises the data processing equipment described in the as above third aspect;
Described caching system, comprises the data processing equipment described in as above fourth aspect.
The beneficial effect that the technical scheme that the embodiment of the present invention provides is brought is:
By obtained the log recording of back-end data base from back-end data base by data publishing platform, from log recording, obtain data change event, data change event is offered to caching system, the data of buffer memory are processed according to data change event by caching system; Solved the data that are buffered in caching system, in MySQL database, variation has occurred, the data that client reads from caching system are by the problem that is legacy data or misdata; Reached caching system can be synchronously new data or delete the data that change more, the effect of legacy data or misdata can be provided to client.
Brief description of the drawings
In order to be illustrated more clearly in the technical scheme in the embodiment of the present invention, below the accompanying drawing of required use during embodiment is described is briefly described, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, do not paying under the prerequisite of creative work, can also obtain according to these accompanying drawings other accompanying drawing.
Fig. 1 is the structural representation of the Database Systems that provide of background technology;
Fig. 2 is the structural representation of the related implementation environment of the data processing method that provides of the embodiment of the present invention;
Fig. 3 is the method flow diagram of the data processing method that provides of one embodiment of the invention;
Fig. 4 is the method flow diagram of the data processing method that provides of another embodiment of the present invention;
Fig. 5 is the block diagram of the data handling system that provides of one embodiment of the invention;
Fig. 6 is the block diagram of the data handling system that provides of another embodiment of the present invention.
Embodiment
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing, embodiment of the present invention is described further in detail.
Please refer to Fig. 2, it shows the structural representation of the related implementation environment of data processing method that the embodiment of the present invention provides.This implementation environment comprises data publishing platform 220, caching system 240 and back-end data base 260.
Data publishing platform 220 is systems newly-increased in the embodiment of the present invention, for obtaining the data variation event of back-end data base 260.Data publishing platform 220 is connected with described caching system 240 by cable network or wireless network.
Caching system 240 improves performance for the data of buffer memory back-end data base 260.This caching system 240 can be distributed memory target cache system Memcached.Caching system 240 is connected with described back-end data base 260 by cable network or wireless network.
Back-end data base 260 is for storing data.Back-end data base 260 need to have log recording and daily record reproduction function.This back-end data base 260 can be MySQL database.
Please refer to Fig. 3, it shows the method flow diagram of the data processing method that one embodiment of the invention provides.This data processing method is applied in the implementation environment shown in Fig. 2, and described method comprises:
Step 301, data publishing platform is obtained the log recording of back-end data base from back-end data base;
Step 302, data publishing platform is obtained data change event from log recording;
Data change event comprises: data are inserted event, data deletion event and data update event.
Step 303, data change event is offered caching system by data publishing platform;
Step 304, caching system obtains data change event from data publishing platform;
Wherein, data change event be data publishing platform from back-end data base obtains the log recording of back-end data base, from log recording, get.
Step 305, caching system is processed the data of buffer memory according to data change event.
In sum, the data processing method that the present embodiment provides, by obtained the log recording of back-end data base from back-end data base by data publishing platform, from log recording, obtain data change event, data change event is offered to caching system, the data of buffer memory are processed according to data change event by caching system; Solved the data that are buffered in caching system, in MySQL database, variation has occurred, the data that client reads from caching system are by the problem that is legacy data or misdata; Reached caching system can be synchronously new data or delete the data that change more, the effect of legacy data or misdata can be provided to client.
It should be added that, above-mentioned steps 301 can realize separately to step 303 data processing method that becomes data publishing platform side, and above-mentioned steps 304 and step 305 can realize separately the data processing method that becomes caching system side.
Please refer to Fig. 4, it shows the method flow diagram of the data processing method that another embodiment of the present invention provides.This data processing method is applied in the implementation environment shown in Fig. 2, and described method comprises:
Step 401, data publishing platform is database registration account to the back-end, and the account has the authority of the log recording that copies back-end data base;
Back-end data base can produce log recording in running.This log recording has comprised data change event.Data change event comprises: data are inserted event, data deletion event and data update event.Such as, in table, insert data, data are inserted event and will be recorded mark and the concrete value of these data.Again such as, in table, deleted data, data deletion event can record mark and the concrete value of these data.For another example, upgraded data in table, data update event can record the mark of these data, value before changing and value after changing.Certainly, log recording has also recorded other event outside data change event.
Data publishing platform is in order to obtain the log recording of back-end data base.First database registration account to the back-end, the account has the authority of the log recording that copies back-end data base.Data publishing platform can disguise oneself as back-end data base from database, replication log record from back-end data base.
Taking back-end data base as MySQL database is as example, MySQL database can generate binary log record in running, utilizes Binlog function that MySQL provides can copy the binary log record of MySQL database.For this reason, can start a Binlog and capture thread in data publishing platform, this Binlog captures thread and is used for being connected to MySQL database crawl log record.First this Binlog captures thread and copies user to one of MySQL database initialize, and this copies user and has been awarded REPLICATION SLAVE authority, also copies the authority of the log recording of MySQL database.
Step 402, data publishing platform is obtained request by account database transmission to the back-end log recording.
Log recording obtains in request and can carry: the side-play amount of the mark of back-end data base, the filename of journal file and journal file.If data publishing platform is to obtain for the first time log recording, this side-play amount can be 0; If data publishing platform is not to obtain for the first time log recording, continue the position of the log recording that this side-play amount can have been obtained from data publishing platform.Such as, data publishing platform had been obtained 1000 log recordings, and database obtains the log recording of the 1001st article of beginning to the back-end.
Step 403, data publishing platform is obtained the log recording of back-end data base from back-end data base;
According to log recording, log recording is sent to data publishing platform by the request of obtaining to back-end data base, and data publishing platform receives this log recording.
Step 404, data publishing platform is obtained data change event from log recording;
Due to other event of also having recorded in log recording outside data change event, so data publishing platform is only extracted data change event from log recording, and abandon other event.
Step 405, data publishing platform is kept at data change event in relaying log recording;
Data publishing platform is kept at the data change event of extracting in relaying log recording.Relaying log recording is created by data publishing platform, and relaying log recording can have more than one.Data publishing platform is in the time writing relaying log recording, first create new relaying log recording, then data writing change event in this relaying log recording, if this relaying log recording record number or size of data has reached predetermined threshold value, newly-built another relaying log recording of data publishing platform continues record.
Step 406, caching system is to data publishing platform subscription data change event;
Data publishing platform need to offer caching system by data change event.Particularly, data publishing platform can offer caching system in the mode of subscribing to by data change event.Caching system is to data publishing platform subscription data change event, and accordingly, data publishing platform receives the subscribe request of caching system to data change event.
Because caching system may be multiple, a part of data in each caching system buffer memory back-end data base., caching system is not all interested in all data change events yet.So caching system is in the time sending subscribe request to data publishing platform, subscribe request can be carried the data change event that caching system is paid close attention to.Such as, in subscribe request, specify log recording side-play amount, caching system is only subscribed to the data change event after this log recording side-play amount.Again such as, in subscribe request, specified a certain table, caching system is only subscribed to the corresponding data change event of this table.For another example, specified a part of key assignments in subscribe request, caching system is only subscribed to the corresponding data change event of this part key assignments.Even, specified a part of field in subscribe request, caching system is only subscribed to the corresponding data change event of this part field.
Step 407, data publishing platform is changed event according to subscribe request to caching system distributing data.
Data publishing platform can start respectively one for each caching system and issue thread, and this issue thread is used for as corresponding caching system distributing data change event.
Particularly, this step can adopt any one realization in following two kinds of modes:
The first, data publishing platform is distributed on data change event in the subscription device setting in advance according to subscribe request;
Such as, data publishing platform self provides subscription device, then by this subscription device, data change event is offered to caching system.
The second, data publishing platform is distributed on data change event in universal information queue according to subscribe request.
Such as, data change event is distributed on rabbitMQ(rabbit Message Queue, rabbit message queue by data publishing platform) in.
Because the programming language that universal information queue is supported is comparatively extensive, can support the demand of different caching systems.
Step 408, caching system obtains the data change event that data publishing platform is issued according to subscribe request;
Accordingly, this step also can adopt any one realization in following two kinds of modes:
The first, the subscription device that caching system sets in advance from data publishing platform, obtain the data change event that data publishing platform is issued according to subscribe request;
The second, caching system obtains the data change event that data publishing platform is issued according to subscribe request from universal information queue.
Step 409, caching system is processed the data of buffer memory according to data change event.
After caching system obtains data change event, there are 2 kinds of different processing modes:
The first, caching system upgrades processing according to data change event to the data of buffer memory;
Owing to there being the mark of the data that relate in each data change event, so whether caching system can be cached with these data according to this mark scanning of home.
If scanning result, for not being cached with this data, shows that these data were never buffered, or these data have ceased to be in force automatically after being buffered one section, and the system that has been buffered is removed automatically.
If scanning result for being cached with this data, show these data buffer memory crossed, caching system, in the time that data change event is data deletion event, is deleted these data from local cache; In the time that data change event is data update event, be value after changing by this Data Update.
The second, caching system is deleted processing according to data change event to the data of buffer memory.
Owing to there being the mark of the data that relate in each data change event, so whether caching system can be cached with these data according to this mark scanning of home.
If scanning result, for not being cached with this data, shows that these data were never buffered, or these data have ceased to be in force automatically after being buffered one section, and the system that has been buffered is removed automatically.
If scanning result for being cached with this data, show these data buffer memory crossed, no matter which kind of event is caching system data change event be, all these data are deleted from local cache.In the time that receive client for the access request of these data next time, caching system again to the back-end database obtain the up-to-date value of these data.
In sum, the data processing method that the present embodiment provides, by obtained the log recording of back-end data base from back-end data base by data publishing platform, from log recording, obtain data change event, data change event is offered to caching system, the data of buffer memory are processed according to data change event by caching system; Solved the data that are buffered in caching system, in MySQL database, variation has occurred, the data that client reads from caching system are by the problem that is legacy data or misdata; Reached caching system can be synchronously new data or delete the data that change more, the effect of legacy data or misdata can be provided to client.
The data processing method that the present embodiment provides, also by the dissemination method of two kinds of different data change events is provided, makes to be applicable to the subscription of different caching systems, has improved the applicability of this data processing method.
The data processing method that the present embodiment provides, also, by providing caching system obtaining after data change event, two kinds of different data processing methods of the data of caching system to buffer memory, can adapt to different concrete scenes.Wherein, the first data processing method makes caching system be cached to up-to-date data, the performance can ensure client-access data time and accuracy.The processing logic of the second data processing method is comparatively simple, also can use preferably in the time that the key assignments manifestation mode of caching system and back-end data base is not identical data.
It should be added that, above-mentioned steps 401, step 402, step 403, step 404, step 405 and step 407 can realize separately the data processing method that becomes data publishing platform side, and above-mentioned steps 406, step 408 and step 409 can realize separately the data processing method that becomes caching system side.
Please refer to Fig. 5, it shows the block diagram of the data handling system that one embodiment of the invention provides.This data handling system comprises data publishing platform 520, caching system 540 and back-end data base 560.Data publishing platform 520 is connected with back-end data base 560 with caching system 540 respectively, and caching system 540 is connected with back-end data base 560.
Data publishing platform 520 comprises a data processing equipment, and this data processing equipment can be realized and be become the whole or a part of of data publishing platform by software, hardware or both combinations.This data processing equipment, comprising: record acquisition module 522, event acquisition module 524 and event module 526 is provided.
Record acquisition module 522, for obtain the log recording of described back-end data base from back-end data base;
Event extraction module 524, for obtaining data change event from described log recording;
Event provides module 526, for described data change event is offered to caching system 540, so that described caching system 540 is processed the data of buffer memory according to described data change event.
Caching system 540 comprises a data processing equipment, and this data processing equipment can be realized and be become the whole or a part of of data publishing platform by software, hardware or both combinations.This data processing equipment, comprising: event acquisition module 542 and data processing module 544;
Event acquisition module 542, for obtain data change event from data publishing platform, described data change record be described data publishing platform from back-end data base obtains the log recording of described back-end data base, from described log recording, obtain;
Data processing module 544, for processing the data of buffer memory according to described data change event.
In sum, the data handling system that the present embodiment provides, by obtained the log recording of back-end data base from back-end data base by data publishing platform, from log recording, obtain data change event, data change event is offered to caching system, the data of buffer memory are processed according to data change event by caching system; Solved the data that are buffered in caching system, in MySQL database, variation has occurred, the data that client reads from caching system are by the problem that is legacy data or misdata; Reached caching system can be synchronously new data or delete the data that change more, the effect of legacy data or misdata can be provided to client.
Please refer to Fig. 6, it shows the block diagram of the data distribution system that another embodiment of the present invention provides.This data handling system comprises data publishing platform 520, caching system 540 and back-end data base 560.Data publishing platform 520 is connected with back-end data base 560 with caching system 540 respectively, and caching system 540 is connected with back-end data base 560.
Data publishing platform 520 comprises a data processing equipment, and this data processing equipment can be realized and be become the whole or a part of of data publishing platform by software, hardware or both combinations.This data processing equipment, comprising: record acquisition module 522, event acquisition module 524 and event module 526 is provided.
Record acquisition module 522, for obtain the log recording of described back-end data base from back-end data base;
Event extraction module 524, for obtaining data change event from described log recording;
Event provides module 526, for described data change event is offered to caching system 540, so that described caching system 540 is processed the data of buffer memory according to described data change event.
Alternatively, data publishing platform 520 also comprises: Account Registration module 521a and request sending module 521b;
Described Account Registration module 521a, for to described back-end data base login account, described account has the authority of the log recording that copies described back-end data base;
Described request sending module 521b, obtains request for sending log recording by described account to described back-end data base.
Alternatively, described event provides module 526, comprising: event storage unit 526a, subscription receiving element 526b and event release unit 526c;
Described event storage unit 526a, for being kept at relaying log recording by described data change event;
Described subscription receiving element 526b, for receiving the subscribe request of described caching system to described data change event;
Described event release unit 526c, for issuing described data change event according to described subscribe request to described caching system.
Alternatively, described event release unit 526c, comprising: first issues subelement, and/or second issues subelement (not shown);
Described first issues subelement, for described data change event being distributed on according to described subscribe request to the subscription device setting in advance;
Described second issues subelement, for described data change event being distributed on to universal information queue according to described subscribe request.
Caching system 540 comprises a data processing equipment, and this data processing equipment can be realized and be become the whole or a part of of data publishing platform by software, hardware or both combinations.This data processing equipment, comprising: event acquisition module 542 and data processing module 544;
Event acquisition module 542, for obtain data change event from data publishing platform, described data change record be described data publishing platform from back-end data base obtains the log recording of described back-end data base, from described log recording, obtain;
Data processing module 544, for processing the data of buffer memory according to described data change event.
Alternatively, described event acquisition module 542, comprising: event subscription unit 542a and event acquiring unit 542b;
Described event subscription unit 542a, for subscribing to described data change event to described data publishing platform;
Described event acquiring unit 542b, the described data change event of issuing according to described subscribe request for obtaining described data publishing platform.
Alternatively, described event acquiring unit 542, comprising: first obtains subelement, and/or second obtains subelement (not shown);
Described first obtains subelement, obtains for the subscription device from setting in advance the described data change event that described data publishing platform is issued according to described subscribe request;
Described second obtains subelement, the described data change event of issuing according to described subscribe request for obtain described data publishing platform from universal information queue.
Alternatively, described data processing module 544, comprising: upgrade processing unit 544a, and/or, delete processing unit 544b;
Described renewal processing unit 544a, for upgrading processing according to described data change event to the data of buffer memory;
Described deletion processing unit 544b, for deleting processing according to described data change event to the data of buffer memory.
In sum, the data handling system that the present embodiment provides, by obtained the log recording of back-end data base from back-end data base by data publishing platform, from log recording, obtain data change event, data change event is offered to caching system, the data of buffer memory are processed according to data change event by caching system; Solved the data that are buffered in caching system, in MySQL database, variation has occurred, the data that client reads from caching system are by the problem that is legacy data or misdata; Reached caching system can be synchronously new data or delete the data that change more, the effect of legacy data or misdata can be provided to client.
The data handling system that the present embodiment provides, also by the published method of two kinds of different data change events is provided, makes to be applicable to the subscription of different caching systems, has improved the applicability of this data handling system.
The data handling system that the present embodiment provides, also, by providing caching system obtaining after data change event, two kinds of different data processing methods of the data of caching system to buffer memory, can adapt to different concrete scenes.Wherein, the first data processing method makes caching system be cached to up-to-date data, the performance can ensure client-access data time and accuracy.The processing logic of the second data processing method is comparatively simple, also can use preferably in the time that the key assignments manifestation mode of caching system and back-end data base is not identical data.
It should be noted that: the data processing equipment that above-described embodiment provides is in the time processing the data of buffer memory, only be illustrated with the division of above-mentioned each functional module, in practical application, can above-mentioned functions be distributed and completed by different functional modules as required, be divided into different functional modules by the internal structure of equipment, to complete all or part of function described above.In addition, the data processing equipment that above-described embodiment provides and data processing method embodiment belong to same design, and its specific implementation process refers to embodiment of the method, repeats no more here.
The invention described above embodiment sequence number, just to describing, does not represent the quality of embodiment.
One of ordinary skill in the art will appreciate that all or part of step that realizes above-described embodiment can complete by hardware, also can carry out the hardware that instruction is relevant by program completes, described program can be stored in a kind of computer-readable recording medium, the above-mentioned storage medium of mentioning can be read-only memory, disk or CD etc.
The foregoing is only preferred embodiment of the present invention, in order to limit the present invention, within the spirit and principles in the present invention not all, any amendment of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (17)
1. a data processing method, is characterized in that, described method comprises:
Obtain the log recording of described back-end data base from back-end data base;
From described log recording, obtain data change event;
Described data change event is offered to caching system, so that described caching system is processed the data of buffer memory according to described data change event.
2. method according to claim 1, is characterized in that, before the described log recording that obtains described back-end data base from described back-end data base, also comprises:
To described back-end data base login account, described account has the authority of the log recording that copies described back-end data base;
Send log recording by described account to described back-end data base and obtain request.
3. method according to claim 1, is characterized in that, described described data change event is offered to caching system, comprising:
Described data change event is kept in relaying log recording;
Receive the subscribe request of described caching system to described data change event;
Issue described data change event according to described subscribe request to described caching system.
4. method according to claim 3, is characterized in that, describedly issues described data change event according to described subscribe request to described caching system, comprising:
Described data change event is distributed in the subscription device setting in advance according to described subscribe request;
And/or,
According to described subscribe request, described data change event is distributed in universal information queue.
5. a data processing method, is characterized in that, described method comprises:
Obtain data change event from data publishing platform, described data change event be described data publishing platform from back-end data base obtains the log recording of described back-end data base, from described log recording, obtain;
According to described data change event, the data of buffer memory are processed.
6. method according to claim 5, is characterized in that, describedly obtains data change event from data publishing platform, comprising:
Subscribe to described data change event to described data publishing platform;
Obtain the described data change event that described data publishing platform is issued according to described subscribe request.
7. method according to claim 6, is characterized in that, described in obtain the described data change record that described data publishing platform is issued according to described subscribe request, comprising:
From the subscription device setting in advance, obtain the described data change event that described data publishing platform is issued according to described subscribe request;
And/or,
From universal information queue, obtain the described data change event that described data publishing platform is issued according to described subscribe request.
8. according to the arbitrary described method of claim 5 to 7, it is characterized in that, describedly according to described data change event, the data of buffer memory processed, comprising:
According to described data change event, the data of buffer memory are upgraded to processing;
And/or,
According to described data change event, the data of buffer memory are deleted to processing.
9. a data processing equipment, is characterized in that, described device comprises:
Record acquisition module, for obtain the log recording of described back-end data base from back-end data base;
Event extraction module, for obtaining data change event from described log recording;
Event provides module, for described data change event is offered to caching system, so that described caching system is processed the data of buffer memory according to described data change event.
10. device according to claim 9, is characterized in that, described device, also comprises:
Account Registration module and request sending module;
Described Account Registration module, for to described back-end data base login account, described account has the authority of the log recording that copies described back-end data base;
Described request sending module, obtains request for sending log recording by described account to described back-end data base.
11. devices according to claim 9, is characterized in that, described event provides module, comprising:
Event storage unit, subscription receiving element and event release unit;
Described event storage unit, for being kept at relaying log recording by described data change event;
Described subscription receiving element, for receiving the subscribe request of described caching system to described data change event;
Described event release unit, for issuing described data change event according to described subscribe request to described caching system.
12. devices according to claim 11, is characterized in that, described event release unit, comprising:
First issues subelement, and/or second issues subelement;
Described first issues subelement, for described data change event being distributed on according to described subscribe request to the subscription device setting in advance;
Described second issues subelement, for described data change event being distributed on to universal information queue according to described subscribe request.
13. 1 kinds of data processing equipments, is characterized in that, described device, comprising:
Event acquisition module, for obtain data change event from data publishing platform, described data change record be described data publishing platform from back-end data base obtains the log recording of described back-end data base, from described log recording, obtain;
Data processing module, for processing the data of buffer memory according to described data change event.
14. devices according to claim 13, is characterized in that, described event acquisition module, comprising:
Event subscription unit and event acquiring unit;
Described event subscription unit, for subscribing to described data change event to described data publishing platform;
Described event acquiring unit, the described data change event of issuing according to described subscribe request for obtaining described data publishing platform.
15. devices according to claim 14, is characterized in that, described event acquiring unit, comprising:
First obtains subelement, and/or second obtains subelement;
Described first obtains subelement, obtains for the subscription device from setting in advance the described data change event that described data publishing platform is issued according to described subscribe request;
Described second obtains subelement, the described data change event of issuing according to described subscribe request for obtain described data publishing platform from universal information queue.
16. according to claim 13 to 15 arbitrary described devices, it is characterized in that, described data processing module, comprising: upgrade processing unit, and/or, delete processing unit;
Described renewal processing unit, for upgrading processing according to described data change event to the data of buffer memory;
Described deletion processing unit, for deleting processing according to described data change event to the data of buffer memory.
17. 1 kinds of data handling systems, is characterized in that, described system comprises: data publishing platform, caching system and back-end data base; Described data publishing platform is connected with described back-end data base with described caching system respectively, and described caching system is connected with described back-end data base;
Described data publishing platform, comprise as claim 9 to claim 12 data processing equipment as described in arbitrary;
Described caching system, comprises the data processing equipment as described in as arbitrary in claim 13 to 16.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410107750.9A CN103929472A (en) | 2014-03-21 | 2014-03-21 | Data processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410107750.9A CN103929472A (en) | 2014-03-21 | 2014-03-21 | Data processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103929472A true CN103929472A (en) | 2014-07-16 |
Family
ID=51147544
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410107750.9A Pending CN103929472A (en) | 2014-03-21 | 2014-03-21 | Data processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103929472A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106933703A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | A kind of method of database data backup, device and electronic equipment |
| CN107273047A (en) * | 2017-06-08 | 2017-10-20 | 郑州云海信息技术有限公司 | A kind of buffer memory management method, cache manager and storage management software |
| CN107528894A (en) * | 2017-08-16 | 2017-12-29 | 郑州云海信息技术有限公司 | A kind of storage system Real time data acquisition method and platform |
| CN107534792A (en) * | 2015-04-30 | 2018-01-02 | 索尼公司 | Receiving device, send equipment and data processing method |
| CN107995242A (en) * | 2016-10-27 | 2018-05-04 | 北京京东尚科信息技术有限公司 | A kind of method for processing business and system |
| CN108628901A (en) * | 2017-03-23 | 2018-10-09 | 深圳市优朋普乐传媒发展有限公司 | A kind of data-updating method and device |
| CN109284066A (en) * | 2017-07-19 | 2019-01-29 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device, equipment and system |
| CN109597817A (en) * | 2018-11-14 | 2019-04-09 | 阿里巴巴集团控股有限公司 | A kind of caching data processing method, device, equipment and medium |
| CN110928911A (en) * | 2019-12-10 | 2020-03-27 | 北大方正集团有限公司 | System, method and device for processing checking request and computer readable storage medium |
| WO2023138185A1 (en) * | 2022-01-24 | 2023-07-27 | 华为技术有限公司 | Information pushing method and server |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5848241A (en) * | 1996-01-11 | 1998-12-08 | Openframe Corporation Ltd. | Resource sharing facility functions as a controller for secondary storage device and is accessible to all computers via inter system links |
| CN101710903A (en) * | 2009-11-17 | 2010-05-19 | 广州从兴电子开发有限公司 | Method, system and platform for publishing information |
| CN101719149A (en) * | 2009-12-03 | 2010-06-02 | 联动优势科技有限公司 | Data synchronization method and device |
| CN102023973A (en) * | 2009-09-14 | 2011-04-20 | 中国移动通信集团公司 | Method, device and system for synchronizing database with application cache server |
| CN103237045A (en) * | 2013-02-22 | 2013-08-07 | 北方工业大学 | Parallel processing system and parallel processing method for large-scale real-time traffic data |
-
2014
- 2014-03-21 CN CN201410107750.9A patent/CN103929472A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5848241A (en) * | 1996-01-11 | 1998-12-08 | Openframe Corporation Ltd. | Resource sharing facility functions as a controller for secondary storage device and is accessible to all computers via inter system links |
| CN102023973A (en) * | 2009-09-14 | 2011-04-20 | 中国移动通信集团公司 | Method, device and system for synchronizing database with application cache server |
| CN101710903A (en) * | 2009-11-17 | 2010-05-19 | 广州从兴电子开发有限公司 | Method, system and platform for publishing information |
| CN101719149A (en) * | 2009-12-03 | 2010-06-02 | 联动优势科技有限公司 | Data synchronization method and device |
| CN103237045A (en) * | 2013-02-22 | 2013-08-07 | 北方工业大学 | Parallel processing system and parallel processing method for large-scale real-time traffic data |
Non-Patent Citations (1)
| Title |
|---|
| 詹自熬 等: "基于XML和MSMQ的数据同步系统设计与实现", 《河南科学》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107534792A (en) * | 2015-04-30 | 2018-01-02 | 索尼公司 | Receiving device, send equipment and data processing method |
| CN106933703A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | A kind of method of database data backup, device and electronic equipment |
| CN107995242A (en) * | 2016-10-27 | 2018-05-04 | 北京京东尚科信息技术有限公司 | A kind of method for processing business and system |
| CN108628901A (en) * | 2017-03-23 | 2018-10-09 | 深圳市优朋普乐传媒发展有限公司 | A kind of data-updating method and device |
| CN107273047A (en) * | 2017-06-08 | 2017-10-20 | 郑州云海信息技术有限公司 | A kind of buffer memory management method, cache manager and storage management software |
| CN109284066A (en) * | 2017-07-19 | 2019-01-29 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device, equipment and system |
| CN109284066B (en) * | 2017-07-19 | 2022-09-30 | 阿里巴巴集团控股有限公司 | Data processing method, device, equipment and system |
| CN107528894A (en) * | 2017-08-16 | 2017-12-29 | 郑州云海信息技术有限公司 | A kind of storage system Real time data acquisition method and platform |
| CN109597817A (en) * | 2018-11-14 | 2019-04-09 | 阿里巴巴集团控股有限公司 | A kind of caching data processing method, device, equipment and medium |
| CN110928911A (en) * | 2019-12-10 | 2020-03-27 | 北大方正集团有限公司 | System, method and device for processing checking request and computer readable storage medium |
| WO2023138185A1 (en) * | 2022-01-24 | 2023-07-27 | 华为技术有限公司 | Information pushing method and server |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103929472A (en) | Data processing method, device and system | |
| CN107004032B (en) | Mirror image file conversion method and device | |
| CN107977396B (en) | Method and device for updating data table of KeyValue database | |
| CN102495772B (en) | Characteristic-based terminal program cloud backup and recovery methods | |
| CN103179185A (en) | Method and system for creating files in cache of distributed file system client | |
| CN104778129B (en) | Method and device for realizing virtual storage of mobile terminal | |
| CN101656751A (en) | Method and system for accelerating file uploading and downloading | |
| CN111447248A (en) | File transmission method and device | |
| EP3627768A1 (en) | Service configuration method and apparatus for sdn | |
| CN102360321A (en) | Terminal program quick backup and recovery method based on cloud architecture | |
| US11301489B2 (en) | Synchronizing online and offline transfer of data to cloud storage system | |
| CN102279857B (en) | Method and system for realizing data reproduction | |
| US9678670B2 (en) | Method for compute element state replication | |
| CN105141672A (en) | Method, device and system for data storage | |
| CN104092726A (en) | Automatic replacing method and apparatus of same-name file | |
| CN111124603A (en) | CD-ROM mounting method, device and related equipment | |
| CN104361034A (en) | File management method and device based on cloud storage | |
| CN103246729A (en) | Method and system for processing multi-media files of android mobile terminal | |
| CN103973772B (en) | File read/write method and device | |
| US11157462B2 (en) | Namespace data object name persistence after decoupling of transportable storage device from data server | |
| CN102164152A (en) | Method and system for remotely processing files | |
| JP6610189B2 (en) | Synchronize collaborative documents with online document management systems | |
| CN103259847A (en) | File pushing method, desktop cloud platform and server | |
| CN114443677A (en) | Object storage service system | |
| CN110300140A (en) | For the method for content update, refreshing client and network node in cloud distribution network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140716 |
|
| RJ01 | Rejection of invention patent application after publication |