How to solve

Bazlur

Dear Community Members,

I am trying to execute a process for a dataset with CMGOS anomaly detection technique. However, every time I try to execute with different parameters, the process is failed with "Matrix is Singular" error. From the discussion, I tried with a covariance matrix before the example set is passed to the clustering process. It still fails. Do you have any idea how to solve this error?

Bazlur

Thanks, jacobcybulski. Yes, obviously I have to use clustering, such as k-means or x-means before CMGOS. However, even after doing that I was facing the "Matrix is Singular" error. Later on, I solved the error by adding a Normalization operator to prepreocess the data before it passes to the clustering. Accordingly, I solved the problem.

jacobcybulski

Excellent, normalisation is always recommended before any operator that measures distances between data points to ensure that all attributes are of equal importance. In your case, it seems there was such a huge difference between the units of your attributes that one of them virtually vanished in the process, thus resulting in its (near) zero interquartile range, which can also result in a "singular" matrix error.
Jacob

jacobcybulski

The error you are getting is common for data with linear dependencies between attributes. Note that first you need to create a centroid-based clustering system (such as k-means)  and only then apply CMGOS. Both k-means and CMGOS are sensitive to dependencies between attributes. So investigate attribute multicollinearities first, e.g. you can build a regression model without feature selection and no removal of collinearities and check for any coefficients with low tolerance. Alternatively, you can apply PCA to convert your attributes set to orthogonal uncorrelated attributes.
Jacob