Loading each csv file related to images into Deep learning during cross validation

varunm1varunm1 Moderator, Member Posts: 1,041   Unicorn
edited January 2019 in Help

I have dataset with 4700 images with dimensions 2000*102. All these images are converted to seperate CSV files (pixel values). I think I can't load all these csv files at a time as this will crash my RAM (32GB). Is it possible for RM to access each image(.csv) file during training and testing from storage location rather than placing it on RAM. This is to train and test CNN in deep learning extension.

@mschmitz @hughesfleming68



  • hughesfleming68hughesfleming68 Member Posts: 261   Unicorn
    Hi Varun, interesting question. I have read about progressive loading of image data with Keras with flow from dataframe and flow from directory but I am not sure how that might work in Rapidminer. I have never tried personally. I am wondering if you could store the image data in a database.
  • varunm1varunm1 Moderator, Member Posts: 1,041   Unicorn
    Hi @hughesfleming68

    Thanks for responding, yes that is what I was wondering as we can load chunks of data from directory using keras in python and it will be useful in RM as well if we have that option (not sure if its there). As sometimes rescaling and downsampling are not good option due to huge loss of data. I will load into database in the form of multiple tables, but I am not aware if it takes table by table when I apply the algorithms.

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,308   Unicorn
    Seems like a another use case of the old "stream database" operator which is unfortunately now deprecated.  I believe @land has been working on a new streaming extension, but I am not sure whether it is adapted for this particular use case.  
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
Sign In or Register to comment.