CN103888321A - Dataflow detecting method and multi-core processing device - Google Patents
Dataflow detecting method and multi-core processing device Download PDFInfo
- Publication number
- CN103888321A CN103888321A CN201410148517.5A CN201410148517A CN103888321A CN 103888321 A CN103888321 A CN 103888321A CN 201410148517 A CN201410148517 A CN 201410148517A CN 103888321 A CN103888321 A CN 103888321A
- Authority
- CN
- China
- Prior art keywords
- data flow
- chain table
- ltsh chain
- treatment facility
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention provides a dataflow detecting method based on a multi-core processing device. The multi-core processing device has the ability of concurrently processing million-level dataflow. The method comprises the step of establishing Hash chain tables corresponding all core processing units in the multi-core processing device according to the processing ability of the multi-core processing device and the current detecting task, wherein nodes in the Hash chain tables are used for buffering dataflow in a backbone network corresponding to the current task in real time, and the dataflow buffered by all nodes is different; the step of executing the aging process on the Hash chain tables corresponding to the core processing units in a concurrent processing mode according to the LRU mechanism; the step of determining the dataflow buffered by the aged Hash chain tables as the active flow. According to the dataflow detecting method, the processing speed is improved, and therefore the active flow can be detected in real time, and the requirement for the real-time performance in detecting the active flow in the backbone network can be met.
Description
Technical field
The application relates to computer network field, particularly a kind of data-flow detection method and multinuclear treatment facility.
Background technology
In high-speed backbone network, data arrive in the mode of one or more stream, here the definition of stream is still the stream that five-tuple defines in the general sense, is defined by source IP, object IP, source port, destination interface and protocol type, and identical five-tuple data are considered as same data flow.Because online dynamic dataflow arrives continually, and if can not process active data stream in time, this data flow can disappear at once, and the requirement of real-time need to design a kind of data-flow detection algorithm running on internal memory.In backbone network, small sample data flow life period is shorter, does not possess researching value, and high amount of traffic life period is long, be the major part of network service, to enlivening the real-time detection of high amount of traffic, have very important significance for optimization of network performance and network information processing.
At present, enliven and in the real-time detection method of high amount of traffic, have a kind of data-flow detection method based on filtering.Based on the data-flow detection method of filtering, it is general in the situation that laboratory flow is less, can packet-by-packet adjudicate packet, but backbone network is generally 10G express network, have the demand of real-time, the data-flow detection method based on filtering is but difficult to meet the real-time demand of backbone network.
Summary of the invention
For solving the problems of the technologies described above, the embodiment of the present application provides a kind of data-flow detection method and multinuclear treatment facility, to reach raising processing speed, realizes and detects in real time active stream, can meet the object that detects the requirement of real-time of active stream in backbone network, technical scheme is as follows:
A kind of data-flow detection method, based on multinuclear treatment facility, described multinuclear treatment facility possesses the disposal ability of concurrent processing 1,000,000 DBMS streams, and described method comprises:
For disposal ability and the current detection task of described multinuclear treatment facility, build the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility, node in described ltsh chain table is for the data flow of the corresponding backbone network of current detection task described in real-time buffer memory, and the data flow of each node institute buffer memory is different;
In the mode of concurrent processing, carry out each kernel processes unit and adopt least recently used LRU mechanism ltsh chain table separately to be carried out to the process of burin-in process;
The data flow of determining the ltsh chain table institute buffer memory after burin-in process is the corresponding active stream of current detection task.
Preferably, in the described disposal ability for described multinuclear treatment facility and current detection task, build in described multinuclear treatment facility before the each self-corresponding ltsh chain table in each kernel processes unit, also comprise:
In the mode of concurrent processing, carry out the process that in described multinuclear treatment facility, each kernel processes unit adopts combined filtration rule to carry out active stream detection;
Wherein, the process that any one kernel processes unit adopts combined filtration rule to carry out active stream detection comprises:
Whether described kernel processes unit judges newly meets preset length scope to the length of data flow;
If so, determine that described is newly active stream to data flow;
If not, described in described kernel processes unit judges, newly whether meet five-tuple filtering rule to data flow;
If so, determine that described is newly active stream to data flow;
If not, described in described kernel processes unit judges, newly whether meet keyword filtering rule to data flow;
If so, determine that described is newly active stream to data flow;
If not, trigger described multinuclear treatment facility and carry out the disposal ability for described multinuclear treatment facility, build the step of the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility.
Preferably, any one kernel processes unit adopts LRU mechanism its ltsh chain table to be carried out to the process of burin-in process, comprising:
A, to newly carrying out cryptographic Hash calculating to data flow, using the cryptographic Hash obtaining as keyword, in described ltsh chain table, search the node that whether exists this cryptographic Hash corresponding, if so, perform step B, if not, execution step C;
B, upgrade the nodal information of this cryptographic Hash corresponding node, and this cryptographic Hash corresponding node is placed in to described ltsh chain table foremost;
C, judge in described ltsh chain table whether have idle node, if so, perform step D, otherwise, execution step E;
D, choose an idle node and deposit this cryptographic Hash, and this idle node is placed in to described ltsh chain table foremost;
E, deletion are positioned at the node of the afterbody of described ltsh chain table, discharge chain table space and deposit this cryptographic Hash, and node corresponding this cryptographic Hash is placed in to described ltsh chain table foremost.
Preferably, before whether described kernel processes unit judges newly meets preset length scope to the length of data flow, also comprise:
Described kernel processes unit adopts mono-kind of 10G the data transfer model POS(Packet Over SDH of optical fiber) the new of form be converted to 10G Ethernet ETH(Ethernet, Ethernet to data flow) form newly arrive data flow.
Preferably, to before newly carrying out cryptographic Hash calculating to data flow, also comprise:
Described kernel processes unit is newly converted to the new for data flow of 10G ETH form to data flow by 10G POS form.
Preferably, described to the process of newly carrying out cryptographic Hash calculating to data flow, comprising:
Adopt the strong hash function of randomness to newly carrying out cryptographic Hash calculating to data flow.
A kind of multinuclear treatment facility, described multinuclear treatment facility possesses the disposal ability of concurrent processing 1,000,000 DBMS streams, and described multinuclear treatment facility comprises:
Build module, for disposal ability and current detection task for described multinuclear treatment facility, build the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility, node in described ltsh chain table is for the corresponding backbone network data flow of current detection task described in real-time buffer memory, and the data flow of each node institute buffer memory is different;
The first control module, for the mode with concurrent processing, carries out each kernel processes unit and adopts LRU mechanism ltsh chain table separately to be carried out to the process of burin-in process;
Determination module is the corresponding active stream of current detection task for the data flow of determining the ltsh chain table institute buffer memory after burin-in process;
Multiple kernel processes unit, described kernel processes unit is for adopting LRU mechanism to carry out burin-in process to ltsh chain table separately.
Preferably, also comprise:
The second control module, for the mode with concurrent processing, carries out the process that in described multinuclear treatment facility, each kernel processes unit adopts combined filtration rule to carry out active stream detection;
Wherein, described kernel processes unit comprises:
Whether the first judgment sub-unit, newly meet preset length scope to the length of data flow for judging, if so, carries out and determine subelement, if not, carries out the second judgment sub-unit;
Described definite subelement, for determining that described is newly active stream to data flow;
Whether described the second judgment sub-unit, describedly newly meet five-tuple filtering rule to data flow for judging, if so, carries out and determine subelement, if not, carries out the 3rd judgment sub-unit;
Whether described the 3rd judgment sub-unit, describedly newly meet keyword filtering rule to data flow for judging, if so, carries out described definite subelement, if not, carries out and trigger subelement;
Described triggering subelement, carries out described structure module for triggering described multinuclear treatment facility.
Preferably, described kernel processes unit comprises:
Search subelement, for to newly carrying out cryptographic Hash calculating to data flow, using the cryptographic Hash obtaining as keyword, in described ltsh chain table, search the node that whether exists this cryptographic Hash corresponding, if so, carry out and upgrade subelement, if not, carry out the 4th judgment sub-unit;
Described renewal subelement, for upgrading the nodal information of this cryptographic Hash corresponding node, and is placed in described ltsh chain table foremost by this cryptographic Hash corresponding node;
Described the 4th judgment sub-unit, for judging whether described ltsh chain table exists idle node, if so, carries out and chooses subelement, if not, carries out and deletes subelement;
The described subelement of choosing, deposits this cryptographic Hash for choosing an idle node, and this idle node is placed in to described ltsh chain table foremost;
Described deletion subelement, for deleting the node of the afterbody that is positioned at described ltsh chain table, discharges chain table space and deposits this cryptographic Hash, and node corresponding this cryptographic Hash is placed in to described ltsh chain table foremost.
Preferably, described kernel processes unit comprises:
Conversion subelement, for being converted to the new for data flow of 10G ETH form by the new of 10G POS form to data flow.
Compared with prior art, the application's beneficial effect is:
The data-flow detection method that the application provides relies on the disposal ability that possesses concurrent processing 1,000,000 DBMS streams, specific implementation is: for disposal ability and the current detection task of described multinuclear treatment facility, build the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility, node in described ltsh chain table is for the corresponding data flow of current detection task described in real-time buffer memory, and the data flow of each node institute buffer memory is different; In the mode of concurrent processing, each kernel processes unit ltsh chain table is separately carried out to LRU mechanism burin-in process; The data flow of determining the ltsh chain table institute buffer memory after LRU mechanism burin-in process is active stream.
That the mode of concurrent processing is carried out LRU mechanism burin-in process to each kernel processes unit ltsh chain table separately due to what adopt, therefore can concurrently carry out LRU mechanism burin-in process to multiple ltsh chain tables, the concurrent data flow of determining multiple ltsh chain table institute buffer memory is active stream, than the mode of processing one by one, improve processing speed, detect in real time active stream thereby realized, can meet the requirement of real-time that detects active stream in backbone network.
Accompanying drawing explanation
In order to be illustrated more clearly in the technical scheme in the embodiment of the present application, below the accompanying drawing of required use during embodiment is described is briefly described, apparently, accompanying drawing in the following describes is only some embodiment of the application, for those of ordinary skills, do not paying under the prerequisite of creative work, can also obtain according to these accompanying drawings other accompanying drawing.
Fig. 1 is a kind of flow chart of the data-flow detection method that provides of the application;
Fig. 2 is the another kind of flow chart of the data-flow detection method that provides of the application;
Fig. 3 is a kind of sub-process figure of the data-flow detection method that provides of the application;
Fig. 4 is the another kind of sub-process figure of the data-flow detection method that provides of the application;
Fig. 5 is a kind of structural representation of the multinuclear treatment facility that provides of the application;
Fig. 6 is the another kind of structural representation of the multinuclear treatment facility that provides of the application;
Fig. 7 is a kind of structural representation of the kernel processes unit that provides of the application;
Fig. 8 is the another kind of structural representation of the kernel processes unit that provides of the application.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present application, the technical scheme in the embodiment of the present application is clearly and completely described, obviously, described embodiment is only the application's part embodiment, rather than whole embodiment.Based on the embodiment in the application, those of ordinary skills are not making the every other embodiment obtaining under creative work prerequisite, all belong to the scope of the application's protection.
The data-flow detection method that the application provides, based on multinuclear treatment facility, wherein, described multinuclear treatment facility possesses the disposal ability of concurrent processing 1,000,000 DBMS streams.In this application, described multiple treatment facility can but be not limited to 32 core treatment facilities.
Embodiment mono-
Refer to Fig. 1, show a kind of flow chart of the data-flow detection method that the application provides, can comprise the following steps:
Step S11: for disposal ability and the current detection task of described multinuclear treatment facility, build the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility.
In the present embodiment, the node in described ltsh chain table is for the data flow of the corresponding backbone network of current detection task described in real-time buffer memory, and the data flow of each node institute buffer memory is different.
When the data flow of the node institute buffer memory in ltsh chain table, change with the variation of the data flow in the corresponding backbone network of current detection task.
Described multinuclear equipment, for itself disposal ability and current detection task, builds the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility.
Wherein, current detection task is the Detection task in some real networks.Detection task in different real networks is often different.
Because the process of the each self-corresponding ltsh chain table in structure each kernel processes unit is identical, therefore, in the present embodiment, only the process that builds any one each self-corresponding ltsh chain table in kernel processes unit is described.
Concrete, the process that builds any one each self-corresponding ltsh chain table in kernel processes unit is:
Steps A 11: generate ltsh chain table.
The length of the ltsh chain table generating is fixed.While generating ltsh chain table, each node of ltsh chain table is all sky nodes.Wherein, adopt doubly linked list to build the Hash node data structure in ltsh chain table.The Hash node data structure that adopts doubly linked list to build in ltsh chain table has been alleviated hash-collision problem to a certain extent.
Steps A 12: the empty node in described ltsh chain table is carried out to assignment.
The detailed process of empty node being carried out to assignment is: first described multinuclear treatment facility carries out cryptographic Hash calculating to the data flow receiving from backbone network, obtain cryptographic Hash, then determine described data flow corresponding node in described ltsh chain table according to cryptographic Hash, complete data flow described in this nodal cache, thereby complete the assignment of node.
Each node in ltsh chain table completes assignment, and structure completes.It should be noted that, each node of the ltsh chain table while the structure separately data flow of institute's buffer memory all can be used as active stream.
Concrete, cryptographic Hash is calculated and is input as five-tuple (when the data flow receiving is carried out cryptographic Hash calculating, being the form of five-tuple), is output as cryptographic Hash.The Hash calculation function selection principle calculating for cryptographic Hash is for requiring randomness strong, and value is obeyed and is uniformly distributed.And adopt the dynamic bucket of a zip mode degree of depth Hash table to carry out index to traffic flow information, realize based on five-tuple stream information node fast finding.
Before carrying out structure ltsh chain table, need to determine Hash barrelage.Concrete, realize a dynamically bucket degree of depth, avoid overflowing because of the node that conflict causes.
Step S12: in the mode of concurrent processing, carry out each kernel processes unit and adopt LRU (least recently used, least recently used) mechanism ltsh chain table separately to be carried out to the process of burin-in process.
Multinuclear treatment facility, in the mode of concurrent processing, is carried out each kernel processes unit and is adopted LRU mechanism ltsh chain table separately to be carried out to the process of burin-in process.
The process that each kernel processes unit adopts LRU mechanism to carry out burin-in process to ltsh chain table is separately concurrent carrying out, and has improved greatly processing speed.
In the present embodiment, in the mode of concurrent processing, carrying out each kernel processes unit adopts process that LRU mechanism carries out burin-in process to ltsh chain table separately, in the mode of concurrent processing, carry out each kernel processes unit and adopt LRU mechanism the chain table space of ltsh chain table separately to be carried out to the process of burin-in process.
Step S13: the data flow of determining the ltsh chain table institute buffer memory after burin-in process is the corresponding active stream of current detection task.
In the present embodiment, multinuclear treatment facility determine each kernel processes unit separately the data flow of the institute of the ltsh chain table after burin-in process buffer memory be the corresponding active stream of current detection task.
Due to the data flow in the corresponding backbone network of the real-time buffer memory current detection of the node in ltsh chain table task, therefore, through the data flow of the ltsh chain table institute buffer memory of burin-in process be the corresponding active stream of current detection task be data flow in the corresponding backbone network of current detection task after real-time buffer memory and burin-in process, the data flow being cached on ltsh chain table is the corresponding active stream of current detection task.
Concrete, through each node in the ltsh chain table of burin-in process separately the data flow of institute's buffer memory be active stream.
The data-flow detection method that the application provides relies on the disposal ability that possesses concurrent processing 1,000,000 DBMS streams, specific implementation is: for disposal ability and the current detection task of described multinuclear treatment facility, build the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility, node in described ltsh chain table is for the corresponding data flow of current detection task described in real-time buffer memory, and the data flow of each node institute buffer memory is different; In the mode of concurrent processing, each kernel processes unit ltsh chain table is separately carried out to LRU mechanism burin-in process; The data flow of determining the ltsh chain table institute buffer memory after LRU mechanism burin-in process is active stream.
That the mode of concurrent processing is carried out LRU mechanism burin-in process to each kernel processes unit ltsh chain table separately due to what adopt, therefore can concurrently carry out LRU mechanism burin-in process to multiple ltsh chain tables, the concurrent data flow of determining multiple ltsh chain table institute buffer memory is active stream, than the mode of processing one by one, improve processing speed, detect in real time active stream thereby realized, can meet the requirement of real-time that detects active stream in backbone network.
Embodiment bis-
In the present embodiment, expand another data-flow detection method on the basis of the data-flow detection method shown in Fig. 1, refer to Fig. 2, Fig. 2 shows the another kind of flow chart of the data-flow detection method that the application provides, and can comprise the following steps:
Step S21: in the mode of concurrent processing, carry out the process that in described multinuclear treatment facility, each kernel processes unit adopts combined filtration rule to carry out active stream detection.
In the present embodiment, when the data flow of particular type has demand in to backbone network, adopt combined filtration rule to carry out active stream detection, be about to meet the data flow (being the data flow of particular type) of combined filtration rule as active stream, thereby obtain the active stream of respective type.
In the present embodiment, combined filtration rule is specifically made up of length filtration rule, five-tuple filtering rule and keyword filtering rule.
The process that each kernel processes unit adopts combined filtration rule to carry out active stream detection is concurrent carrying out.
Whether each kernel processes unit directly judgement newly meets filtering rule to data flow, if meet, is describedly newly active stream to data flow, if do not meet, is describedly newly non-active stream to data flow.
It should be noted that, the data flow newly newly receiving from backbone network to data flow, what in present specification, occur is newly to data flow the data flow newly receiving from backbone network, follow-up repeating no more.
Step S22: for disposal ability and the current detection task of described multinuclear treatment facility, build the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility.
Step S23: in the mode of concurrent processing, carry out each kernel processes unit and adopt LRU mechanism ltsh chain table separately to be carried out to the process of burin-in process.
Step S24: the data flow of determining the ltsh chain table institute buffer memory after burin-in process is active stream.
Step S22, step S23 are identical with step S13 with step S11, step S12 in the data-flow detection method shown in Fig. 1 with step S24, do not repeat them here.
The process of carrying out active stream detection due to each kernel processes unit employing combined filtration rule is identical, therefore the process that the present embodiment only adopts combined filtration rule to carry out active stream detection to any one kernel processes unit is described, specifically refer to Fig. 3, Fig. 3 shows a kind of sub-process figure of the data-flow detection method that the application provides, and can comprise the following steps:
Step S31: whether described kernel processes unit judges newly meets preset length scope to the length of data flow.
In the present embodiment, newly whether to meet preset length scope to the length of data flow be newly whether to meet length filtration rule to data flow described in described kernel processes unit judges to described kernel processes unit judges.
Wherein, preset length scope can dynamically change its zone of reasonableness according to the actual requirements.
If judged result is that newly the length to data flow meets preset length scope, perform step S32, if judged result is that newly the length violation to data flow closes preset length scope, perform step S33.
It should be noted that, before execution step S31, need described kernel processes unit by 10G POS(Packet Over SDH, a kind of data transfer model that adopts optical fiber) the new of form be converted to 10G ETH(Ethernet, Ethernet to data flow) form newly arrive data flow.
What kernel processes unit was processed in the present embodiment is newly the new for data flow of 10G ETH ethernet frame format to data flow.
Step S32: determine that described is newly active stream to data flow.
Step S33: newly whether meet five-tuple filtering rule to data flow described in described kernel processes unit judges.
If judged result is described new for data stream conforms five-tuple filtering rule, return to execution step S32, otherwise, execution step S34.
Step S34: newly whether meet keyword filtering rule to data flow described in described kernel processes unit judges.
If judged result is described new for data stream conforms keyword filtering rule, return to execution step S32, otherwise, execution step S25.
Step S35: trigger described multinuclear treatment facility and carry out the disposal ability for described multinuclear treatment facility, build the step of the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility.
In the present embodiment, each kernel processes unit often newly receives a data flow, just carries out a step S31 to step S35.
Embodiment tri-
What illustrate in the present embodiment, is that each kernel processes unit adopts LRU mechanism ltsh chain table separately to be carried out to the process of burin-in process.
The process that adopts LRU mechanism to carry out burin-in process to ltsh chain table separately due to each kernel processes unit is identical, therefore the process that the present embodiment only adopts LRU mechanism to carry out burin-in process to its ltsh chain table to any one kernel processes unit is described, specifically refer to Fig. 4, Fig. 4 shows the another kind of sub-process figure of the data-flow detection method that the application provides, and can comprise the following steps:
Step S41: to newly carrying out cryptographic Hash calculating to data flow, using the cryptographic Hash obtaining as keyword, search the node that whether exists this cryptographic Hash corresponding in described ltsh chain table.
Concrete; whether exist in order to locate and to search a node in chained list fast; adopt related Hash calculation function in the data-flow detection method shown in Fig. 1; first to newly carrying out Hash buffer memory to data flow according to five-tuple; using the cryptographic Hash obtaining as keyword, in described ltsh chain table, search the node that whether exists this cryptographic Hash corresponding.Wherein, in described ltsh chain table, search and whether exist node corresponding to this cryptographic Hash to be specially to utilize Hash comparison function assignment in described ltsh chain table, to search the node that whether exists this cryptographic Hash corresponding.
If find, perform step S42, if do not find, execution step S43.
It should be noted that, for backbone network, before execution step S41, need described kernel processes unit that the new of 10G POS form is converted to the new for data flow of 10G ETH form to data flow.
What kernel processes unit was processed in the present embodiment is newly the new for data flow of 10G ETH ethernet frame format to data flow.
Step S42: upgrade the nodal information of this cryptographic Hash corresponding node, and this cryptographic Hash corresponding node is placed in to described ltsh chain table foremost.
Step S43: judge whether there is idle node in described ltsh chain table.
If judged result, for there is idle node, performs step S44, otherwise, execution step S45.
Step S44: choose an idle node and deposit this cryptographic Hash, and this idle node is placed in to described ltsh chain table foremost.
Step S45: delete the node of the afterbody that is positioned at described ltsh chain table, discharge chain table space and deposit this cryptographic Hash, and node corresponding this cryptographic Hash is placed in to described ltsh chain table foremost.
In the time newly receiving data flow, if ltsh chain table is full,, the knot removal of ltsh chain table afterbody " the oldest ", vacate chain table space new for node corresponding to data flow for depositing, and by (being top) foremost that be newly placed in ltsh chain table to node corresponding to data flow.
Adopt LRU mechanism, because the rill duration is short, arrival rate is low, be always likely replaced away; And flow greatly, the duration is long, access cache is frequent, so tend to be buffered in the forward position of stem of ltsh chain table.
In the present embodiment, each kernel processes unit often newly receives a data flow, just carries out a step S41 to step S45.
For aforesaid each embodiment of the method, for simple description, therefore it is all expressed as to a series of combination of actions, but those skilled in the art should know, the application is not subject to the restriction of described sequence of movement, because according to the application, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in specification all belongs to preferred embodiment, and related action and module might not be that the application is necessary.
Embodiment tetra-
In the present embodiment, a kind of multinuclear treatment facility is provided, described multinuclear treatment facility possesses the disposal ability of concurrent processing 1,000,000 DBMS streams, refer to Fig. 5, Fig. 5 shows a kind of structural representation of the multinuclear treatment facility that the application provides, and multinuclear treatment facility comprises: build module 51, the first control module 52, determination module 53 and multiple kernel processes unit 54.
The first control module 52, for the mode with concurrent processing, carries out each kernel processes unit and adopts LRU mechanism ltsh chain table separately to be carried out to the process of burin-in process.
In the present embodiment, the another kind of multinuclear treatment facility of the multinuclear treatment facility being different from shown in Fig. 5 is also provided, refer to Fig. 6, Fig. 6 shows the another kind of structural representation of the multinuclear treatment facility that the application provides, and also comprises: the second control module 61 on the basis of Fig. 5.
The second control module 61, for the mode with concurrent processing, carries out each kernel processes unit 54 in described multinuclear treatment facility and adopts combined filtration rule to carry out the process of active stream detection.
In kernel processes unit 54, refer to Fig. 7 for realizing the concrete structure that adopts combined filtration rule to carry out the process of active stream detection, Fig. 7 shows a kind of structural representation of the kernel processes unit that the application provides, and kernel processes unit comprises: the first judgment sub-unit 71, determine subelement 72, the second judgment sub-unit 73, the 3rd judgment sub-unit 74 and trigger subelement 75.
Whether the first judgment sub-unit 71, newly meet preset length scope to the length of data flow for judging, if so, carries out and determine subelement 72, if not, carries out the second judgment sub-unit 73.
Described definite subelement 72, for determining that described is newly active stream to data flow.
Whether described the second judgment sub-unit 73, describedly newly meet five-tuple filtering rule to data flow for judging, if so, carries out and determine subelement 72, if not, carries out the 3rd judgment sub-unit 74.
Whether described the 3rd judgment sub-unit 74, describedly newly meet keyword filtering rule to data flow for judging, if so, carries out described definite subelement 72, if not, carries out and trigger subelement 75.
Described triggering subelement 75, carries out described structure module 51 for triggering described multinuclear treatment facility.
In addition, in kernel processes unit 54, refer to Fig. 8 for realizing the concrete structure that adopts LRU mechanism to carry out the process of burin-in process to its ltsh chain table, Fig. 8 shows the another kind of structural representation of the kernel processes unit that the application provides, and kernel processes unit comprises: search subelement 81, upgrade subelement 82, the 4th judgment sub-unit 83, choose subelement 84 and delete subelement 85.
Described renewal subelement 82, for upgrading the nodal information of this cryptographic Hash corresponding node, and is placed in described ltsh chain table foremost by this cryptographic Hash corresponding node.
Described the 4th judgment sub-unit 83, for judging whether described ltsh chain table exists idle node, if so, carries out and chooses subelement 84, if not, carries out and deletes subelement 85.
The described subelement 84 of choosing, deposits this cryptographic Hash for choosing an idle node, and this idle node is placed in to described ltsh chain table foremost.
Described deletion subelement 85, for deleting the node of the afterbody that is positioned at described ltsh chain table, discharges chain table space and deposits this cryptographic Hash, and node corresponding this cryptographic Hash is placed in to described ltsh chain table foremost.
In the present embodiment, kernel processes unit includes conversion subelement, for the new of 10G POS form is converted to the new for data flow of 10G ETH form to data flow.
Conversion subelement was carried out before carrying out the first judgment sub-unit 71, and carried out before subelement 81 is searched in execution.
Certainly, the multinuclear treatment facility that the present embodiment provides includes 10G input interface, for receiving 10G POS light form new for data flow of backbone network.
10G input interface is sent to conversion subelement by the new of 10G POS form to data flow, by conversion subelement, the new of 10G POS light form is converted to the new for data flow of 10G ETH form to data flow.
The multinuclear treatment facility that the present embodiment provides includes 10G output interface equally, for the active stream detecting is exported.
It should be noted that, each embodiment in this specification all adopts the mode of going forward one by one to describe, and what each embodiment stressed is and the difference of other embodiment, between each embodiment identical similar part mutually referring to.For device class embodiment, because it is substantially similar to embodiment of the method, so description is fairly simple, relevant part is referring to the part explanation of embodiment of the method.
Finally, also it should be noted that, in this article, relational terms such as the first and second grades is only used for an entity or operation to separate with another entity or operating space, and not necessarily requires or imply and between these entities or operation, have the relation of any this reality or sequentially.And, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thereby the process, method, article or the equipment that make to comprise a series of key elements not only comprise those key elements, but also comprise other key elements of clearly not listing, or be also included as the intrinsic key element of this process, method, article or equipment.The in the situation that of more restrictions not, the key element being limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment that comprises described key element and also have other identical element.
For convenience of description, while describing above device, being divided into various unit with function describes respectively.Certainly, in the time implementing the application, the function of each unit can be realized in same or multiple software and/or hardware.
As seen through the above description of the embodiments, those skilled in the art can be well understood to the mode that the application can add essential general hardware platform by software and realizes.Based on such understanding, the part that the application's technical scheme contributes to prior art in essence in other words can embody with the form of software product, this computer software product can be stored in storage medium, as ROM/RAM, magnetic disc, CD etc., comprise that some instructions (can be personal computers in order to make a computer equipment, server, or the network equipment etc.) carry out the method described in some part of each embodiment of the application or embodiment.
A kind of data-flow detection method and the multinuclear treatment facility that above the application are provided are described in detail, applied principle and the execution mode of specific case to the application herein and set forth, the explanation of above embodiment is just for helping to understand the application's method and core concept thereof; , for one of ordinary skill in the art, according to the application's thought, all will change in specific embodiments and applications, in sum, this description should not be construed as the restriction to the application meanwhile.
Claims (10)
1. a data-flow detection method, is characterized in that, based on multinuclear treatment facility, described multinuclear treatment facility possesses the disposal ability of concurrent processing 1,000,000 DBMS streams, and described method comprises:
Described multinuclear treatment facility is for disposal ability and the current detection task of described multinuclear treatment facility, build the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility, node in described ltsh chain table is for the data flow of the corresponding backbone network of current detection task described in real-time buffer memory, and the data flow of each node institute buffer memory is different; And,
In the mode of concurrent processing, carry out each kernel processes unit and adopt least recently used LRU mechanism ltsh chain table separately to be carried out to the process of burin-in process; And,
The data flow of determining the ltsh chain table institute buffer memory after burin-in process is the corresponding active stream of current detection task.
2. method according to claim 1, is characterized in that, in the described disposal ability for described multinuclear treatment facility and current detection task, builds in described multinuclear treatment facility before the each self-corresponding ltsh chain table in each kernel processes unit, also comprises:
In the mode of concurrent processing, carry out the process that in described multinuclear treatment facility, each kernel processes unit adopts combined filtration rule to carry out active stream detection;
Wherein, the process that any one kernel processes unit adopts combined filtration rule to carry out active stream detection comprises:
Whether described kernel processes unit judges newly meets preset length scope to the length of data flow;
If so, determine that described is newly active stream to data flow;
If not, described in described kernel processes unit judges, newly whether meet five-tuple filtering rule to data flow;
If so, determine that described is newly active stream to data flow;
If not, described in described kernel processes unit judges, newly whether meet keyword filtering rule to data flow;
If so, determine that described is newly active stream to data flow;
If not, trigger described multinuclear treatment facility and carry out the disposal ability for described multinuclear treatment facility, build the step of the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility.
3. method according to claim 1 and 2, is characterized in that, any one kernel processes unit adopts LRU mechanism its ltsh chain table to be carried out to the process of burin-in process, comprising:
A, to newly carrying out cryptographic Hash calculating to data flow, using the cryptographic Hash obtaining as keyword, in described ltsh chain table, search the node that whether exists this cryptographic Hash corresponding, if so, perform step B, if not, execution step C;
B, upgrade the nodal information of this cryptographic Hash corresponding node, and this cryptographic Hash corresponding node is placed in to described ltsh chain table foremost;
C, judge in described ltsh chain table whether have idle node, if so, perform step D, otherwise, execution step E;
D, choose an idle node and deposit this cryptographic Hash, and this idle node is placed in to described ltsh chain table foremost;
E, deletion are positioned at the node of the afterbody of described ltsh chain table, discharge chain table space and deposit this cryptographic Hash, and node corresponding this cryptographic Hash is placed in to described ltsh chain table foremost.
4. method according to claim 2, is characterized in that, before whether described kernel processes unit judges newly meets preset length scope to the length of data flow, also comprises:
Described kernel processes unit adopts mono-kind of 10G the data transfer model POS(Packet Over SDH of optical fiber) the new of form be converted to 10G Ethernet ETH(Ethernet, Ethernet to data flow) form newly arrive data flow.
5. method according to claim 3, is characterized in that, to before newly carrying out cryptographic Hash calculating to data flow, also comprises:
Described kernel processes unit is newly converted to the new for data flow of 10G ETH form to data flow by 10G POS form.
6. method according to claim 3, is characterized in that, described to the process of newly carrying out cryptographic Hash calculating to data flow, comprising:
Adopt the strong hash function of randomness to newly carrying out cryptographic Hash calculating to data flow.
7. a multinuclear treatment facility, is characterized in that, described multinuclear treatment facility possesses the disposal ability of concurrent processing 1,000,000 DBMS streams, and described multinuclear treatment facility comprises:
Build module, for disposal ability and current detection task for described multinuclear treatment facility, build the each self-corresponding ltsh chain table in each kernel processes unit in described multinuclear treatment facility, node in described ltsh chain table is for the corresponding backbone network data flow of current detection task described in real-time buffer memory, and the data flow of each node institute buffer memory is different;
The first control module, for the mode with concurrent processing, carries out each kernel processes unit and adopts LRU mechanism ltsh chain table separately to be carried out to the process of burin-in process;
Determination module is the corresponding active stream of current detection task for the data flow of determining the ltsh chain table institute buffer memory after burin-in process;
Multiple kernel processes unit, described kernel processes unit is for adopting LRU mechanism to carry out burin-in process to ltsh chain table separately.
8. multinuclear treatment facility according to claim 7, is characterized in that, also comprises:
The second control module, for the mode with concurrent processing, carries out the process that in described multinuclear treatment facility, each kernel processes unit adopts combined filtration rule to carry out active stream detection;
Wherein, described kernel processes unit comprises:
Whether the first judgment sub-unit, newly meet preset length scope to the length of data flow for judging, if so, carries out and determine subelement, if not, carries out the second judgment sub-unit;
Described definite subelement, for determining that described is newly active stream to data flow;
Whether described the second judgment sub-unit, describedly newly meet five-tuple filtering rule to data flow for judging, if so, carries out and determine subelement, if not, carries out the 3rd judgment sub-unit;
Whether described the 3rd judgment sub-unit, describedly newly meet keyword filtering rule to data flow for judging, if so, carries out described definite subelement, if not, carries out and trigger subelement;
Described triggering subelement, carries out described structure module for triggering described multinuclear treatment facility.
9. according to the multinuclear treatment facility described in claim 7 or 8, it is characterized in that, described kernel processes unit comprises:
Search subelement, for to newly carrying out cryptographic Hash calculating to data flow, using the cryptographic Hash obtaining as keyword, in described ltsh chain table, search the node that whether exists this cryptographic Hash corresponding, if so, carry out and upgrade subelement, if not, carry out the 4th judgment sub-unit;
Described renewal subelement, for upgrading the nodal information of this cryptographic Hash corresponding node, and is placed in described ltsh chain table foremost by this cryptographic Hash corresponding node;
Described the 4th judgment sub-unit, for judging whether described ltsh chain table exists idle node, if so, carries out and chooses subelement, if not, carries out and deletes subelement;
The described subelement of choosing, deposits this cryptographic Hash for choosing an idle node, and this idle node is placed in to described ltsh chain table foremost;
Described deletion subelement, for deleting the node of the afterbody that is positioned at described ltsh chain table, discharges chain table space and deposits this cryptographic Hash, and node corresponding this cryptographic Hash is placed in to described ltsh chain table foremost.
10. multinuclear treatment facility according to claim 9, is characterized in that, described kernel processes unit comprises:
Conversion subelement, for being converted to the new for data flow of 10G ETH form by the new of 10G POS form to data flow.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410148517.5A CN103888321B (en) | 2014-04-14 | 2014-04-14 | Dataflow detecting method and multi-core processing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410148517.5A CN103888321B (en) | 2014-04-14 | 2014-04-14 | Dataflow detecting method and multi-core processing device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103888321A true CN103888321A (en) | 2014-06-25 |
| CN103888321B CN103888321B (en) | 2017-05-24 |
Family
ID=50957048
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410148517.5A Active CN103888321B (en) | 2014-04-14 | 2014-04-14 | Dataflow detecting method and multi-core processing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103888321B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105162663A (en) * | 2015-09-25 | 2015-12-16 | 中国人民解放军信息工程大学 | Online traffic identification method based on flow set |
| CN105187436A (en) * | 2015-09-25 | 2015-12-23 | 中国航天科工集团第二研究院七〇六所 | Packet filtering host network control method based on hash table |
| CN106254134A (en) * | 2016-08-29 | 2016-12-21 | 上海斐讯数据通信技术有限公司 | A kind of network equipment and the method that data are flow to line pipe control thereof |
| CN106357451A (en) * | 2016-09-30 | 2017-01-25 | 广东电网有限责任公司电力科学研究院 | Fuzzing test data stream management method |
| CN108170489A (en) * | 2016-12-07 | 2018-06-15 | 腾讯科技（深圳）有限公司 | A kind of method and device of quick loading resource |
| CN108255590A (en) * | 2017-12-07 | 2018-07-06 | 深圳比特微电子科技有限公司 | A kind of method of data flow control and device |
| CN112468365A (en) * | 2020-11-26 | 2021-03-09 | 上海阅维科技股份有限公司 | Data quality detection method, system and medium for network mirror flow |
| CN114020657A (en) * | 2021-11-03 | 2022-02-08 | 无锡沐创集成电路设计有限公司 | Message searching method, system, storage medium and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7406522B2 (en) * | 2001-09-26 | 2008-07-29 | Packeteer, Inc. | Dynamic partitioning of network resources |
| CN101834763A (en) * | 2010-06-25 | 2010-09-15 | 山东大学 | Multiple-category large-flow parallel measuring method under high speed network environment |
| CN102025563A (en) * | 2010-11-30 | 2011-04-20 | 东南大学 | Network flow identification method based on Hash collision compensation |
| CN102497297A (en) * | 2011-12-13 | 2012-06-13 | 曙光信息产业(北京)有限公司 | System and method for realizing deep packet inspection technology based on multi-core and multi-thread |
| CN102546299A (en) * | 2012-01-09 | 2012-07-04 | 北京锐安科技有限公司 | Method for detecting deep packet under large flow |
-
2014
- 2014-04-14 CN CN201410148517.5A patent/CN103888321B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7406522B2 (en) * | 2001-09-26 | 2008-07-29 | Packeteer, Inc. | Dynamic partitioning of network resources |
| CN101834763A (en) * | 2010-06-25 | 2010-09-15 | 山东大学 | Multiple-category large-flow parallel measuring method under high speed network environment |
| CN102025563A (en) * | 2010-11-30 | 2011-04-20 | 东南大学 | Network flow identification method based on Hash collision compensation |
| CN102497297A (en) * | 2011-12-13 | 2012-06-13 | 曙光信息产业(北京)有限公司 | System and method for realizing deep packet inspection technology based on multi-core and multi-thread |
| CN102546299A (en) * | 2012-01-09 | 2012-07-04 | 北京锐安科技有限公司 | Method for detecting deep packet under large flow |
Non-Patent Citations (1)
| Title |
|---|
| 赵小欢 等: "互联网流采用技术综述", 《小型微型计算机系统》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105162663A (en) * | 2015-09-25 | 2015-12-16 | 中国人民解放军信息工程大学 | Online traffic identification method based on flow set |
| CN105187436A (en) * | 2015-09-25 | 2015-12-23 | 中国航天科工集团第二研究院七〇六所 | Packet filtering host network control method based on hash table |
| CN105162663B (en) * | 2015-09-25 | 2019-02-19 | 中国人民解放军信息工程大学 | A kind of online method for recognizing flux based on adfluxion |
| CN105187436B (en) * | 2015-09-25 | 2019-03-08 | 中国航天科工集团第二研究院七〇六所 | A kind of packet filtering mainframe network control method based on hash table |
| CN106254134A (en) * | 2016-08-29 | 2016-12-21 | 上海斐讯数据通信技术有限公司 | A kind of network equipment and the method that data are flow to line pipe control thereof |
| CN106357451A (en) * | 2016-09-30 | 2017-01-25 | 广东电网有限责任公司电力科学研究院 | Fuzzing test data stream management method |
| CN108170489A (en) * | 2016-12-07 | 2018-06-15 | 腾讯科技（深圳）有限公司 | A kind of method and device of quick loading resource |
| CN108255590A (en) * | 2017-12-07 | 2018-07-06 | 深圳比特微电子科技有限公司 | A kind of method of data flow control and device |
| CN112468365A (en) * | 2020-11-26 | 2021-03-09 | 上海阅维科技股份有限公司 | Data quality detection method, system and medium for network mirror flow |
| CN114020657A (en) * | 2021-11-03 | 2022-02-08 | 无锡沐创集成电路设计有限公司 | Message searching method, system, storage medium and electronic equipment |
| CN114020657B (en) * | 2021-11-03 | 2023-03-17 | 无锡沐创集成电路设计有限公司 | Message searching method, system, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103888321B (en) | 2017-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103888321A (en) | Dataflow detecting method and multi-core processing device | |
| CN103873712B (en) | Voip gateway detection method and multinuclear treatment facility | |
| CN101296114B (en) | Parallel pattern matching method and system based on stream | |
| Xu et al. | Stela: Enabling stream processing systems to scale-in and scale-out on-demand | |
| CN102752198B (en) | Multi-core message forwarding method, multi-core processor and network equipment | |
| Balkesen et al. | Adaptive input admission and management for parallel stream processing | |
| US20160188376A1 (en) | Push/Pull Parallelization for Elasticity and Load Balance in Distributed Stream Processing Engines | |
| CN105897921B (en) | A kind of data block method for routing of the sampling of combination fingerprint and reduction fragmentation of data | |
| Vaquero et al. | xDGP: A dynamic graph processing system with adaptive partitioning | |
| CN105045723A (en) | Processing method, apparatus and system for cached data | |
| CN101604261B (en) | Task scheduling method for supercomputer | |
| CN104915717A (en) | Data processing method, knowledge base reasoning method and related device | |
| WO2017107812A1 (en) | User log storage method and device | |
| US20090282217A1 (en) | Horizontal Scaling of Stream Processing | |
| CN103412858A (en) | Method for large-scale feature matching of text content or network content analyses | |
| US9846599B1 (en) | Adaptive query cursor management | |
| US20220327412A1 (en) | Dynamic quantum data output post-processing | |
| Tan et al. | On resource pooling and separation for LRU caching | |
| Weigert et al. | Mining large distributed log data in near real time | |
| CN106909624B (en) | Real-time sequencing optimization method for mass data | |
| EP3011456B1 (en) | Sorted event monitoring by context partition | |
| US20240078235A1 (en) | Task-execution planning using machine learning | |
| US10209763B2 (en) | Power aware switching using analytics | |
| CN105335530B (en) | A method of promoting long data block data de-duplication performance | |
| CN110245130A (en) | Data duplicate removal method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |