rloess in Rapiminer
MavenHi.
I'm trying to make a predictive regression model and are currently looking at my raw data. I can see that my raw data got a lot of noise in the signal, which means that it got alot of variance in a short time period. My goal is to make data windows of 10 mins where I calculate the mean, variance and linear regression coefficient and then use these windows in my model.
But if I use the noisy data, the variance will be larger than i actually is. That makes me think that I need to make a smooth fitted line to the noisy data before I use them? I've seen another project doing this by using the rloess technique, but this is not implemented in Rapidminer?
Instead I've tryed to use "Moving average", "Exponential smoothing" and "Fit trend" with an "Local Polynomial Regression inside. Moving average and Exponential smoothing are both making a fitted line, but its not possible to weight the outliers like you can do in rloess. Fit trend is taking to long to process because of the amount of data.
Anyone have an idea for the best approach for this?
Picture of the noisy data attached.
Answers
Hi Mathias,
have you thought about using the outlier detection score as a weight in the regression?
~Martin
Dortmund, Germany
I've tried the "k-NN Global Anomaly Score" but it's taking far too long to process.
Try one of the clustering ones,they are at O(N**2).
~Martin
Dortmund, Germany