Term frequency vs TF-IDF

Answers

• Options

  IngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder

  March 2012

  Hi,
  
  term frequency is simply the normalized number of occurrences of a term divided by the number of terms of the document (from the souce code):
  
          int numTerms = wordList.size();
          double totalTermNumber = 0;
          for (float value: frequencies)
          totalTermNumber += value;
         
          // Create the result structure
          double[] wv = new double[numTerms];
  
          // If document contains at least one term
          if (totalTermNumber > 0) {
              // Create the vector
              double length = 0.0;
              for (int i = 0; i < wv.length; i++) {
                  wv = frequencies / totalTermNumber;
                  length += wv * wv;
              }
  
              length = Math.sqrt(length);
  
              // Normalize the vector
              if (length > 0.0)
                  for (int i = 0; i < wv.length; i++)
                      wv = wv / length;
          }
  
  
  
  In case of TDIDF, the term frequency of each term is normalized with the inverse document frequency (from the source code):
  
        // Obtain the total number of documents and the document frequencies
          int numDocuments = wordList.getNumberOfDocuments();
          int[] docFrequencies = wordList.getDocumentFrequencies();
          double totalTermNumber = 0;
          for (float value: frequencies)
          totalTermNumber += value;
         
         
          // Create the result structure
          double[] wv = new double[docFrequencies.length];
  
          // Create the vector
  
          // If the document contains at least one term
          if (totalTermNumber > 0) {
              double length = 0.0;
              for (int i = 0; i < wv.length; i++) {
                  // Note: docFrequencies is always > 0 as otherwise the word
                  // would not be in the word list, it is also always smaller as
                  // the total number of documents
  
                  double idf = Math.log(((double) numDocuments) / ((double) docFrequencies));
                  wv = (frequencies / totalTermNumber) * idf;
                  length += wv * wv;
              }
              length = Math.sqrt(length);
  
              // Normalize the vector
              if (length > 0.0)
                  for (int i = 0; i < wv.length; i++)
                      wv = wv / length;
          }
  
  
  You can find a discussion about the normalization somewhere in the forum if this is interesting for you.
  
  Cheers,
  Ingo
  
  

  0
• Options

  ferandi Member Posts: 9 Contributor II

  March 2012

  Thank you so much Ingo!
  
  On Text classification with one are more suitable? Thanks in advance!!

  0
• Options

  IngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder

  March 2012

  Hi,
  
  as always in data mining: just test it. If one of both preprocessings works better for your data then just go for it. In general, however, TFIDF has outperformed term frequency alone on all practical settings I have experienced so far.
  
  Cheers,
  Ingo

  0