"Association Rules with Thousands of Products"
I am trying to create association rules.
My dataset have 30496 transactions and 18270 products and it is stored in a MySQL database like this:
transaction | product
1 | 0987
1 | 6543
2 | 1234
... | ...
Before use FP-Growth operator I convert data with Nominal to Numerical operator and aggregate the rows of the same transaction. After that I convert data again with Numerical to Binomial operator.
I run the process with RapidAnalytics but it returns the error "java.lang.OutOfMemoryError: GC overhead limit exceeded ". And I used Free Memory operator.
How can I solve this problem? It is possible to create association rules with this number of products?
Other question, it is possible to convert data to a sparse matrix?
I am studing other approach in other database. I installed in a PostgreSQL database the MADlib library. And with the same dataset, MADlib function create the association rules in 30 seconds without any transformation of the data!!!