"Extract information from weblog (how to handle 31 text files for 3GB)"

Answers

• Options

  THW_Mark Member Posts: 11 Contributor II

  September 2010

  i'm not too familiar with the inner processes of rapidminer, but i assume that if you read a 3GB log file, it will occupy at least that amount in the memory. given that you are doing some manipulations/selections on the data, i would guess that you would need more RAM available then 3GB, more like 6. If you don't have 6GB RAM, you can split up the processes by splitting your log files.
  
  Save the extracted data in to the repository (part1, part2, ... partn), and when you are done, combine the repository files to make the final entry containing your extracted data.

  1
• Options

  makchishing Member Posts: 6 Contributor II

  October 2010

  THW Mark wrote:

  i'm not too familiar with the inner processes of rapidminer, but i assume that if you read a 3GB log file, it will occupy at least that amount in the memory. given that you are doing some manipulations/selections on the data, i would guess that you would need more RAM available then 3GB, more like 6. If you don't have 6GB RAM, you can split up the processes by splitting your log files. Save the extracted data in to the repository (part1, part2, ... partn), and when you are done, combine the repository files to make the final entry containing your extracted data.

  Thanks Mark ::)
  
  Actually, I want to do a very simple work, i.e. to read thorugh the 3GB text, extract and aggregate some substring in it.
  If the inner process of rapidminer is should be run process by process in RAM....(I mean..must read all text into RAM first),
  I would rather to read the 3GB text into database first and aggregate myself.

  1