RapidMiner

Evaluating Anomaly and signature detection methods

Contributor II fwood201
Contributor II

Re: Evaluating Anomaly and signature detection methods

Thats pretty cool! so here's my dataset - as you can see some data is labelled normal others the name of the attack. Would this generate attribute operator recognise the difference between the attacks and the normal ones? How could i put this in an intrusion detection process?

 

Cheers

F

RM Staff
RM Staff

Re: Evaluating Anomaly and signature detection methods

Hi,

 

look like a equation like:

if(label!="normal","attack","normal")

would to the trick.

 

~Martin

--------------------------------------------------------------------------
Head of Data Science Services at RapidMiner
Contributor II fwood201
Contributor II

Re: Evaluating Anomaly and signature detection methods

Thanks that worked perfectly! My last question is - do you think it would be possible to somehow feed the model a novel attack that isnt in the signature database to demonstrate that the IDS' wont detect them if the signature pattern isnt there?

 

F.

RM Staff
RM Staff

Re: Evaluating Anomaly and signature detection methods

Hi,

 

arguably yes. In the area of fraud you often run into the issue of a tradeoff between being good in detecting known patterns (low false positive rate) and being good in also finding unknown patterns.

I usually work in environments where these fraud detections are handed to a human experts who decides. In this case I recommend to populate the list with 3 different sources of potential frauds/attacks:

 

1. A list generated by a supervised algorithm, which is very good in finding "known" (= already seen) patterns

2. A list generated by a unsupervised algorithm, which is good in also finding new patterns

3. Randomly selected instances as control group

 

The results of 2 and 3 serve as new tagged examples for 1. in a feedback loop.

 

Best,

Martin

--------------------------------------------------------------------------
Head of Data Science Services at RapidMiner
Contributor II fwood201
Contributor II

Re: Evaluating Anomaly and signature detection methods

Can you suggest a process for implementing this?

 

F.

Highlighted
RM Staff
RM Staff

Re: Evaluating Anomaly and signature detection methods

Hi,

 

i think the parts in itself are fairly straight forward to built. It's what you already do + a standard supervised method like we explain it in getting started. The tricky thing is how to merge the results and how to built the feedback loop. This is usually customer depended.

 

Best,

Martin

--------------------------------------------------------------------------
Head of Data Science Services at RapidMiner
RM Staff
RM Staff

Re: Evaluating Anomaly and signature detection methods

Dear @fwood201,

 

I fully agree with @mschmitz on "

1. A list generated by a supervised algorithm, which is very good in finding "known" (= already seen) patterns

2. A list generated by a unsupervised algorithm, which is good in also finding new patterns

3. Randomly selected instances as control group

 

The results of 2 and 3 serve as new tagged examples for 1. in a feedback loop."

Some fraud detection templates are built in RapidMiner studio. You can run it for a quick demo of supversied algo (as suggest in Martin's list #1) on medical fraud instances. I also installed the anomaly detection extension from Marketplace and run HBOS, for instance, to get the risk scores from unsupervised algo (Martin's list #2). Actually the input example set has negative (non-fraud control group) data randomly selected from a big population. So we can later look into the prediction results for false postives (pre-labeled as 'false' but predicted as 'true' fraud), and manually correct (feedback) the label after some invesitigations.

template.pngaddHBOS.png