Loading each csv file related to images into Deep learning during cross validation

varunm1

Hi

I have dataset with 4700 images with dimensions 2000*102. All these images are converted to seperate CSV files (pixel values). I think I can't load all these csv files at a time as this will crash my RAM (32GB). Is it possible for RM to access each image(.csv) file during training and testing from storage location rather than placing it on RAM. This is to train and test CNN in deep learning extension.

@mschmitz @hughesfleming68

Thanks,
Varun

hughesfleming68

Hi Varun, interesting question. I have read about progressive loading of image data with Keras with flow from dataframe and flow from directory but I am not sure how that might work in Rapidminer. I have never tried personally. I am wondering if you could store the image data in a database.

varunm1

Hi @hughesfleming68

Thanks for responding, yes that is what I was wondering as we can load chunks of data from directory using keras in python and it will be useful in RM as well if we have that option (not sure if its there). As sometimes rescaling and downsampling are not good option due to huge loss of data. I will load into database in the form of multiple tables, but I am not aware if it takes table by table when I apply the algorithms.

Thanks
Varun

Telcontar120

Seems like a another use case of the old "stream database" operator which is unfortunately now deprecated.  I believe @land has been working on a new streaming extension, but I am not sure whether it is adapted for this particular use case.