CN101364897A - System for historical data archiving and implementing method - Google Patents
System for historical data archiving and implementing method Download PDFInfo
- Publication number
- CN101364897A CN101364897A CNA2008102161958A CN200810216195A CN101364897A CN 101364897 A CN101364897 A CN 101364897A CN A2008102161958 A CNA2008102161958 A CN A2008102161958A CN 200810216195 A CN200810216195 A CN 200810216195A CN 101364897 A CN101364897 A CN 101364897A
- Authority
- CN
- China
- Prior art keywords
- filing
- historical data
- archiving
- thread
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a historical data archiving system and the implementation method. The historical data archiving system comprises an archiving parameter configuration module, an archiving data loading module, and an archiving operation module, wherein the archiving parameter configuration module is used for configuring an archiving list name for the historical data, and configuring archiving parameters such as the archiving mode, the archiving output directory, the archiving start time, the archiving operation, the filter condition and the archiving fields in the archiving list name; the archiving data loading module is used for loading all of the archiving parameters as the input of the historical data archiving; and the archiving operation module is used for detecting whether historical data meeting the archiving condition exists according to the archiving parameters, and archiving the historical data meeting the archiving condition. The system and the implementation method have simpler operation, and can improve the operation flexibility and the data outputting visualization and greatly reduce data redundancy.
Description
Technical field
The present invention relates to historical data archiving field in the database, relate in particular to a kind of system and implementation method of historical data archiving.
Background technology
State the historical data that needs in the network management system a large amount of and handle, performance data for example is owing to need gather for example 5 minutes/24 hours to the different performance item cycle of carrying out of the different operating end points of distinct device.Described network management system all can produce very many performance datas every day, such as greater than 1,000 ten thousand, need file also i.e. backup to these data, for uses such as inquiry later on, statistics, report form processing, current filing way commonly used is that the backup recovery instrument that utilizes the Database Systems of described network management system itself to provide carries out, but the method may can have influence on the efficient of follow-up use because filing data is too much when especially redundant data is many.Just can use these data after also needing behind the described historical data archiving to recover, cause operation very inconvenient, and these operation requirements are familiar to database, general personnel are not easy to finish.
Second kind of archiving method of prior art is the filing that own developing instrument carries out historical data.Current when carrying out business operation such as report form processing, need earlier filing data is loaded in the volatile data base of described Database Systems, again the data in the described volatile data base are inquired about.Though filing all is that instrument is finished automatically with loading, operate simple relatively, and described volatile data base service efficiency is also than customer data base height, but when the filing data amount is very large, the loading of data and the performance bottleneck that the inquiry of described volatile data base is become easily system.
But existing described network management system is except that performance can not be met consumers' demand preferably, also has a significant disadvantages, data after the filing all are the data that described network management system generates exactly, it is a kind of initial data, be the data that the developer faces, also need to consult the concrete meaning that related data could be understood data when the general user need use these data, this just makes archive file not transport out in a kind of definite mode.As seen, prior art awaits further developing.
Summary of the invention
The present invention provides a kind of historical data archiving System and method for for solving above-mentioned defective of the prior art, filing parameter configuration module, filing data load-on module and archive operation module are set on the prior art basis, to increase the simplicity and the flexibility of operation, improve the efficient of network management system deal with data.
For solving the problems of the technologies described above, the present invention program comprises:
A kind of system of historical data archiving, wherein, described system comprises filing parameter configuration module, a filing data load-on module and an archive operation module, described filing parameter configuration module and described filing data load-on module communicate to connect, described filing data load-on module is connected with described archive operation module communication, and described filing parameter configuration module is used to historical data configuration filing parameter; Described filing data load-on module is used to load all described filing parameters of described filing parameter configuration module configuration; Described archive operation module is used for described historical data is carried out archive operation.
Described system, wherein, described archive operation module also comprises a main thread, described main thread is used to create a filing thread, detect and whether have the described historical data that satisfies the filing condition and wake described filing thread up and carry out archive operation, and described filing thread is in resting state and waits for that described main thread wakes up and carries out archive operation.
The method of described historical data archiving system may further comprise the steps:
A, described filing parameter configuration module are that described historical data disposes described filing table name, and will file parameter configuration under described filing table name;
B, described filing data load-on module load the input of all described filing parameters as described historical data archiving;
Whether C, described archive operation module exist the described historical data that satisfies the filing condition according to all described filing parameter detecting, and the described historical data that satisfies described filing condition is carried out archive operation.
Described implementation method, wherein, described filing parameter comprises: archive mode, filing output directory, filing time started, filing action, filtercondition and filing field.
Described implementation method, wherein, described archive mode comprises cycle filing and thresholding filing, described cycle filing is also disposed the filing cycle; Threshold value and the thresholding cycle of detecting described historical data thresholding are also disposed in described thresholding filing.
Described implementation method, wherein, described filing action comprises judges whether described archive mode and described filing table name mate, whether delete former described historical data and whether delete described filing table name when described historical data is sky.
Described implementation method, wherein, described filing field comprises the value after value, field value and the described field value after field name and described field name transform transforms.
Described implementation method, wherein, described step C also comprises: when not satisfying the described historical data of described filing condition, then described filing thread is in resting state.
Described implementation method, wherein, described step C is further comprising the steps of:
C1, described main thread are created a described filing thread for each described filing table name, and described filing thread is in resting state;
Whether C2, described main thread exist the described historical data that satisfies described filing condition according to described filtercondition and described filing cycle or the regular detection of described thresholding cycle, if exist, then described main thread wakes described filing thread up and carries out archive operation.
Described implementation method, wherein, described step C2 is further comprising the steps of:
C21, described filing thread are searched all the described filing table names that need file, and obtain all the described filing parameters under the described filing table name;
C22, described filing thread are searched the described historical data that satisfies the filing condition according to described filtercondition;
The described historical data original value that C23, described filing thread will satisfy described filing condition transforms and outputs in the archive file of assigned catalogue.
Described implementation method, wherein, described step C2 is further comprising the steps of:
C24, described filing thread with described historical data archiving after, former described historical data and described table name are carried out other operation of described filing parameter appointment.
The invention provides a kind of system and implementation method of historical data archiving, filing parameter configuration module, filing data load-on module and archive operation module are set on the prior art basis, described network management system can be finished by configuration according to business demand, reduced complexity, improved the intuitive of simplicity, flexibility and the dateout of user's operation database manipulation; Can also only file according to user's filtercondition, reduce data redundancy, improve the efficient of network management system deal with data greatly the data that satisfy condition.
Description of drawings
Fig. 1 is the system configuration and the schematic flow sheet of the historical numerical control filing of the present invention;
Fig. 2 is the schematic flow sheet of historical data archiving specific embodiment of the present invention.
Embodiment
Below with reference to accompanying drawing, comparatively detailed explanation is carried out in each preferred embodiment of the present invention.
The invention provides a kind of system and implementation method of historical data archiving, as shown in Figure 1, filing parameter configuration module, filing data load-on module and archive operation module are set on the prior art basis, described network management system can be finished by configuration according to business demand, reduced complexity, improved the simplicity and the flexibility of user's operation database manipulation; Can also only file according to user's filtercondition, reduce data redundancy, improve the efficient of network management system deal with data greatly the historical data that satisfies condition.
As shown in Figure 1, described historical data archiving system comprises filing parameter configuration module, a filing data load-on module and an archive operation module, described filing parameter configuration module and described filing data load-on module communicate to connect, described filing data load-on module is connected with described archive operation module communication, and described filing parameter configuration module is used to filing parameters such as historical data configuration filing table name, archive mode, filing output directory, filing time started, filing action, filtercondition and filing field; Described filing data load-on module is used to load all described filing parameters of described filing parameter configuration module configuration; Described archive operation module is used for described historical data is carried out archive operation; Described archive operation module also comprises a described main thread, described main thread is used to create a described filing thread, detect and whether have the described historical data that satisfies the filing condition and wake described filing thread up and carry out archive operation, and described filing thread is in resting state and waits for that described main thread wakes up and carries out archive operation.
The historical data archiving system flow of the invention described above as shown in Figure 1, may further comprise the steps:
Step 101: described filing parameter configuration module is that described historical data disposes described filing table name, and described filing parameter configuration such as described archive mode, described filing output directory, described filing time started, described filing action, described filtercondition and described filing field are arrived under the described filing table name;
Step 102: described filing data load-on module loads the input of all described filing parameters of configuration in the described step 101 as described historical data archiving;
Step 103: whether described archive operation module exists the described historical data that satisfies the filing condition according to all described filing parameter detecting, and the described historical data that satisfies described filing condition is carried out archive operation.
Wherein, also comprise in the described step 101: described archive mode comprises cycle filing and thresholding filing, and described cycle filing is also disposed the filing cycle; Threshold value and the thresholding cycle of detecting described historical data thresholding are also disposed in described thresholding filing; Described filing action comprises judges whether described archive mode and described filing table name mate, whether delete former described historical data and whether delete described filing table name when described historical data is sky; Described filing field comprises the value after value, field value and the described field value after field name and described field name transform transforms.
Wherein, also comprise in the described step 103: when not satisfying the described historical data of described filing condition, then described filing thread is in resting state.Described main thread is created a described filing thread for each described filing table name, and described filing thread is in resting state; Whether described main thread exists the described historical data that satisfies described filing condition according to described filtercondition and described filing cycle or the regular detection of described thresholding cycle, if exist, then described main thread wakes described filing thread up and carries out archive operation.Described filing thread is searched all the described filing table names that need file, and obtains all the described filing parameters under the described filing table name; Described filing thread is searched the described historical data that satisfies the filing condition according to described filtercondition; The described historical data original value that described filing thread will satisfy described filing condition transforms and outputs in the archive file of assigned catalogue.Described filing thread with described historical data archiving after, former described historical data and described table name are carried out other operation of described filing parameter appointment.As seen, through the present invention as shown in Figure 1 the processing of flow process reduced complexity to database manipulation, increased the simplicity and the flexibility of user's operation.
In order further to set forth the present invention, describe at a specific embodiment.
As Fig. 1 and shown in Figure 2, the present embodiment parameter value leaves in the exterior arrangement file, dispose the value of described filing parameter by manual configuration this document, detailed process is as follows: set up data structure DUMP_INFO_TABLE, one of member is the described filing table name that will file, resolves the table name of filing described in the configuration file process and can leave in here; Set up data structure DUMP_MODE_INFO then, the member comprises described archive mode, described filing output directory, described filing time started and described filing periodic quantity, these members' value can be filled in resolving the configuration file process, and defines the member of a member of this data structure as described data structure DUMP_INFO_TABLE; Set up data structure DUMP_RULE_INFO subsequently, whether the member comprises pattern matching, whether delete legacy data, whether delete described filing table name and described filtercondition, these members' value can be filled in resolving the configuration file process, and defines the member of a member of this data structure as described data structure DUMP_INFO_TABLE; Set up data structure TRANS_INFO_FIELD again, the member comprises that title, the various values of described field and the described field value after described field name and described field transform transforms value afterwards, these values can be filled in resolving the configuration file process, and define the member of a queue object of this data structure as described data structure DUMP_INFO_TABLE; Create the queue object DumpInfoTables of a DUMP_INFO_TABLE at last, all described filing parameters of above-mentioned configuration are finished the object assignment of a described data structure DUMP_INFO_TABLE, and add among the described formation DumpInfoTables, described queue object DumpInfoTables is delivered to described archive operation module as parameter.
As shown in Figure 2, present embodiment is further comprising the steps of:
Step 201: described archive operation module is created a filing thread for each the described filing table name that will file among the described queue object DumpInfoTables that transmits, and described filing thread creation is laggard goes into the sleep state; Described main thread goes at set intervals to check according to described filtercondition, described thresholding cycle or filing cycle whether each the described filing table name among the DumpInfoTables satisfies the filing condition, if satisfy described filing condition, then wake described filing thread up and enter archive operation; Otherwise described filing thread continues sleep a period of time; Described main thread goes to check whether each the described filing table name among the described queue object DumpInfoTables satisfies the filing condition again;
Step 202: the described filing table name to each satisfied filing condition among the described queue object DumpInfoTables is filed;
Step 203: described filing thread checks whether described queue object DumpInfoTables is pattern matching, if, then construct sql statement inquiry might need the described filing table name filed;
Step 204: described filing thread is set up a data structure FIELDINFO, the member comprises the title of field, the type of field, set up a mapping simultaneously, deposit described filing table name and FIELDINFO, set up the map information of described filing table name and respective field, obtain all information that previous step inquires;
Step 205: the filtercondition according to this table among the described queue object DumpInfoTables is inquired about, and obtains a result set;
Step 206: described filing thread reads each field value of each bar record in the described result set and transforms and outputs in the archive file;
Step 207: after the described historical data of described filing thread output filing is finished, original described historical data and described filing table name deleted wait other operations.For making things convenient for inquiry, statistics and report form processing etc., the present embodiment archive file is the Excel file.
In sum, adopt the system and the implementation method of a kind of historical data archiving provided by the invention, filing parameter configuration module, filing data load-on module and archive operation module are set on the prior art basis, can finish by manual configuration according to business demand, do not relate to complicated database manipulation, thereby make operation easier, satisfied user's request preferably; Can be according to described filing cycle or described thresholding cycle, carry out the filing of appointment according to the described filtercondition of user's appointment, be the form output that the user understands easily with data conversion, thereby improved the flexibility of operation and the intuitive of dateout greatly; Can also only carry out archive operation according to user's instruction, significantly reduce data redundancy, further improve the efficient of described network management system deal with data the described historical data that satisfies condition.
Should be understood that above-mentioned description at preferred embodiment is comparatively detailed, can not therefore think the restriction to scope of patent protection of the present invention, the scope of asking for protection of the present invention should be as the criterion with claims.
Claims (11)
1, a kind of system of historical data archiving, it is characterized in that, described system comprises filing parameter configuration module, a filing data load-on module and an archive operation module, described filing parameter configuration module and described filing data load-on module communicate to connect, described filing data load-on module is connected with described archive operation module communication, and described filing parameter configuration module is used to historical data configuration filing parameter; Described filing data load-on module is used to load all described filing parameters of described filing parameter configuration module configuration; Described archive operation module is used for described historical data is carried out archive operation.
2, system according to claim 1, it is characterized in that, described archive operation module also comprises a main thread, described main thread is used to create a filing thread, detect and whether have the described historical data that satisfies the filing condition and wake described filing thread up and carry out archive operation, and described filing thread is in resting state and waits for that described main thread wakes up and carries out archive operation.
3, a kind of realization may further comprise the steps according to the method for the described system of claim 1:
A, described filing parameter configuration module are that described historical data disposes described filing table name, and will file parameter configuration under described filing table name;
B, described filing data load-on module load the input of all described filing parameters as described historical data archiving;
Whether C, described archive operation module exist the described historical data that satisfies the filing condition according to all described filing parameter detecting, and the described historical data that satisfies described filing condition is carried out archive operation.
4, implementation method according to claim 3 is characterized in that, described filing parameter comprises: archive mode, filing output directory, filing time started, filing action, filtercondition and filing field.
5, implementation method according to claim 4 is characterized in that, described archive mode comprises cycle filing and thresholding filing, and described cycle filing is also disposed the filing cycle; Threshold value and the thresholding cycle of detecting described historical data thresholding are also disposed in described thresholding filing.
6, implementation method according to claim 4, it is characterized in that described filing action comprises judges whether described archive mode and described filing table name mate, whether delete former described historical data and whether delete described filing table name when described historical data is sky.
7, implementation method according to claim 4 is characterized in that, described filing field comprises the value after value, field value and the described field value after field name and described field name transform transforms.
8, implementation method according to claim 3 is characterized in that, described step C also comprises: when not satisfying the described historical data of described filing condition, then described filing thread is in resting state.
9, implementation method according to claim 3 is characterized in that, described step C is further comprising the steps of:
C1, described main thread are created a described filing thread for each described filing table name, and described filing thread is in resting state;
Whether C2, described main thread exist the described historical data that satisfies described filing condition according to described filtercondition and described filing cycle or the regular detection of described thresholding cycle, if exist, then described main thread wakes described filing thread up and carries out archive operation.
10, implementation method according to claim 9 is characterized in that, described step C2 is further comprising the steps of:
C21, described filing thread are searched all the described filing table names that need file, and obtain all the described filing parameters under the described filing table name;
C22, described filing thread are searched the described historical data that satisfies the filing condition according to described filtercondition;
The described historical data original value that C23, described filing thread will satisfy described filing condition transforms and outputs in the archive file of assigned catalogue.
11, implementation method according to claim 9 is characterized in that, described step C2 is further comprising the steps of:
C24, described filing thread with described historical data archiving after, former described historical data and described table name are carried out other operation of described filing parameter appointment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2008102161958A CN101364897A (en) | 2008-09-17 | 2008-09-17 | System for historical data archiving and implementing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2008102161958A CN101364897A (en) | 2008-09-17 | 2008-09-17 | System for historical data archiving and implementing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101364897A true CN101364897A (en) | 2009-02-11 |
Family
ID=40391070
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2008102161958A Pending CN101364897A (en) | 2008-09-17 | 2008-09-17 | System for historical data archiving and implementing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101364897A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103514295A (en) * | 2013-10-10 | 2014-01-15 | 中国电子科技集团公司第十五研究所 | Method and device for archiving historical data |
| CN105279261A (en) * | 2015-10-23 | 2016-01-27 | 北京京东尚科信息技术有限公司 | Dynamic extensible database filing method and system |
| CN106126526A (en) * | 2016-06-13 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of data managing method and device |
| CN107370773A (en) * | 2016-05-11 | 2017-11-21 | 北京京东尚科信息技术有限公司 | The method and system of the telescopic distributed data filing of High Availabitity |
| CN107870981A (en) * | 2017-09-30 | 2018-04-03 | 平安科技（深圳）有限公司 | Electronic installation, the method and storage medium of tables of data filing processing |
| WO2018058983A1 (en) * | 2016-09-28 | 2018-04-05 | 平安科技（深圳）有限公司 | Database capacity calculation method, apparatus, server, and storage device |
| CN109583842A (en) * | 2018-12-04 | 2019-04-05 | 北京诺士诚国际工程项目管理有限公司 | A kind of management monthly magazine generation method, device, electronic equipment and computer storage medium |
| CN111506749A (en) * | 2020-04-15 | 2020-08-07 | 北京金山云网络技术有限公司 | Monitoring data archiving method and device, electronic equipment and storage medium |
| CN113032406A (en) * | 2021-05-26 | 2021-06-25 | 四川新网银行股份有限公司 | Data archiving method for centralized management of sub-tables through metadata database |
-
2008
- 2008-09-17 CN CNA2008102161958A patent/CN101364897A/en active Pending
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103514295A (en) * | 2013-10-10 | 2014-01-15 | 中国电子科技集团公司第十五研究所 | Method and device for archiving historical data |
| CN103514295B (en) * | 2013-10-10 | 2016-09-28 | 中国电子科技集团公司第十五研究所 | historical data archiving method and historical data archiving device |
| CN105279261A (en) * | 2015-10-23 | 2016-01-27 | 北京京东尚科信息技术有限公司 | Dynamic extensible database filing method and system |
| CN105279261B (en) * | 2015-10-23 | 2019-07-02 | 北京京东尚科信息技术有限公司 | Dynamic scalable database filing method and system |
| CN107370773A (en) * | 2016-05-11 | 2017-11-21 | 北京京东尚科信息技术有限公司 | The method and system of the telescopic distributed data filing of High Availabitity |
| CN106126526A (en) * | 2016-06-13 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of data managing method and device |
| WO2018058983A1 (en) * | 2016-09-28 | 2018-04-05 | 平安科技（深圳）有限公司 | Database capacity calculation method, apparatus, server, and storage device |
| CN107870981A (en) * | 2017-09-30 | 2018-04-03 | 平安科技（深圳）有限公司 | Electronic installation, the method and storage medium of tables of data filing processing |
| CN107870981B (en) * | 2017-09-30 | 2021-10-22 | 平安科技（深圳）有限公司 | Electronic device, data sheet filing method and storage medium |
| CN109583842A (en) * | 2018-12-04 | 2019-04-05 | 北京诺士诚国际工程项目管理有限公司 | A kind of management monthly magazine generation method, device, electronic equipment and computer storage medium |
| CN111506749A (en) * | 2020-04-15 | 2020-08-07 | 北京金山云网络技术有限公司 | Monitoring data archiving method and device, electronic equipment and storage medium |
| CN111506749B (en) * | 2020-04-15 | 2024-01-02 | 北京金山云网络技术有限公司 | Monitoring data archiving method and device, electronic equipment and storage medium |
| CN113032406A (en) * | 2021-05-26 | 2021-06-25 | 四川新网银行股份有限公司 | Data archiving method for centralized management of sub-tables through metadata database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101364897A (en) | System for historical data archiving and implementing method | |
| CN102027457B (en) | Managing storage of individually accessible data units | |
| CN103036956B (en) | A kind of distributed configuration mass data filing system and implementation method | |
| CN1959717B (en) | System and method for preprocessing mass remote sensing data collection driven by order form | |
| CN105808633A (en) | Data archiving method and system | |
| CN102193917A (en) | Method and device for processing and querying data | |
| CN105243528A (en) | Financial IT system graphical centralized reconciliation system and method under big data environment | |
| CN101937474A (en) | Mass data query method and device | |
| CN101631041B (en) | Service configuration processing method and configuration relation management module | |
| CN111857777B (en) | System updating method and device based on double-core intelligent ammeter and computer equipment | |
| CN103294652A (en) | Data conversion method and system | |
| CN103927314A (en) | Data batch processing method and device | |
| CN109766206A (en) | A kind of log collection method and system | |
| CN102521218A (en) | File combining method and file combining device | |
| CN104519078A (en) | A method and a device for realizing cloud storage | |
| CN104679500A (en) | Automatic generation realizing method and device for entity classes | |
| CN103577048A (en) | Desktop icon intelligent arranging system and method | |
| CN110209736A (en) | Device, method and the storage medium of block chain data processing | |
| CN103257971A (en) | Terminal and file management method | |
| CN114281757A (en) | Database migration method and system and computer readable storage medium | |
| CN102932476B (en) | Network storage synchro system | |
| CN108574718A (en) | A kind of cloud host creation method and device | |
| CN101286886A (en) | Method and device to recover configuring information of network appliance | |
| CN102323943A (en) | Virtual-machine-based data storage method and system | |
| CN102752138A (en) | Asynchronous configuration management method and network device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Open date: 20090211 |