Document (#35316)

Author
Roitblat, H.L.
Kershaw, A.
Oot, P.
Title
Document categorization in legal electronic discovery : computer classification vs. manual review
Source
Journal of the American Society for Information Science and Technology. 61(2010) no.1, S.70-80
Year
2009
Abstract
In litigation in the US, the parties are obligated to produce to one another, when requested, those documents that are potentially relevant to issues and facts of the litigation (called discovery). As the volume of electronic documents continues to grow, the expense of dealing with this obligation threatens to surpass the amounts at issue and the time to identify these relevant documents can delay a case for months or years. The same holds true for government investigations and third-parties served with subpoenas. As a result, litigants are looking for ways to reduce the time and expense of discovery. One approach is to supplant or reduce the traditional means of having people, usually attorneys, read each document, with automated procedures that use information retrieval and machine categorization to identify the relevant documents. This study compared an original categorization, obtained as part of a response to a Department of Justice Request and produced by having one or more of 225 attorneys review each document with automated categorization systems provided by two legal service providers. The goal was to determine whether the automated systems could categorize documents at least as well as human reviewers could, thereby saving time and expense. The results support the idea that machine categorization is no less accurate at identifying relevant/responsive documents than employing a team of reviewers. Based on these results, it would appear that using machine categorization can be a reasonable substitute for human review.
Field
Rechtswissenschaft

Similar documents (content)

  1. Yang, Y.; Wilbur, J.: Using corpus statistics to remove redundant words in text categorization (1996) 0.23
    0.23201913 = sum of:
      0.23201913 = product of:
        0.82863975 = sum of:
          0.0077282586 = weight(abstract_txt:that in 4268) [ClassicSimilarity], result of:
            0.0077282586 = score(doc=4268,freq=1.0), product of:
              0.04113416 = queryWeight, product of:
                1.0754405 = boost
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.015904764 = queryNorm
              0.18787934 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.012674671 = weight(abstract_txt:with in 4268) [ClassicSimilarity], result of:
            0.012674671 = score(doc=4268,freq=2.0), product of:
              0.045404196 = queryWeight, product of:
                1.129882 = boost
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.015904764 = queryNorm
              0.27915198 = fieldWeight in 4268, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.030317524 = weight(abstract_txt:time in 4268) [ClassicSimilarity], result of:
            0.030317524 = score(doc=4268,freq=1.0), product of:
              0.09296035 = queryWeight, product of:
                1.4001179 = boost
                4.1745143 = idf(docFreq=1768, maxDocs=42306)
                0.015904764 = queryNorm
              0.32613394 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.1745143 = idf(docFreq=1768, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.07045774 = weight(abstract_txt:reduce in 4268) [ClassicSimilarity], result of:
            0.07045774 = score(doc=4268,freq=1.0), product of:
              0.14248112 = queryWeight, product of:
                1.4153008 = boost
                6.3296742 = idf(docFreq=204, maxDocs=42306)
                0.015904764 = queryNorm
              0.4945058 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.3296742 = idf(docFreq=204, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.0741491 = weight(abstract_txt:automated in 4268) [ClassicSimilarity], result of:
            0.0741491 = score(doc=4268,freq=1.0), product of:
              0.16874823 = queryWeight, product of:
                1.8864068 = boost
                5.624406 = idf(docFreq=414, maxDocs=42306)
                0.015904764 = queryNorm
              0.4394067 = fieldWeight in 4268, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.624406 = idf(docFreq=414, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.08218255 = weight(abstract_txt:documents in 4268) [ClassicSimilarity], result of:
            0.08218255 = score(doc=4268,freq=2.0), product of:
              0.18072642 = queryWeight, product of:
                2.7608418 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.015904764 = queryNorm
              0.45473453 = fieldWeight in 4268, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
          0.55112994 = weight(abstract_txt:categorization in 4268) [ClassicSimilarity], result of:
            0.55112994 = score(doc=4268,freq=5.0), product of:
              0.47354138 = queryWeight, product of:
                4.468993 = boost
                6.6622515 = idf(docFreq=146, maxDocs=42306)
                0.015904764 = queryNorm
              1.1638474 = fieldWeight in 4268, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                6.6622515 = idf(docFreq=146, maxDocs=42306)
                0.078125 = fieldNorm(doc=4268)
        0.28 = coord(7/25)
    
  2. Kim, J.-H.; Choi, K.-S.: Patent document categorization based on semantic structural information (2007) 0.21
    0.21216473 = sum of:
      0.21216473 = product of:
        0.7577312 = sum of:
          0.008743526 = weight(abstract_txt:that in 2934) [ClassicSimilarity], result of:
            0.008743526 = score(doc=2934,freq=2.0), product of:
              0.04113416 = queryWeight, product of:
                1.0754405 = boost
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.015904764 = queryNorm
              0.2125612 = fieldWeight in 2934, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.0625 = fieldNorm(doc=2934)
          0.0071698767 = weight(abstract_txt:with in 2934) [ClassicSimilarity], result of:
            0.0071698767 = score(doc=2934,freq=1.0), product of:
              0.045404196 = queryWeight, product of:
                1.129882 = boost
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.015904764 = queryNorm
              0.15791221 = fieldWeight in 2934, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.0625 = fieldNorm(doc=2934)
          0.02425402 = weight(abstract_txt:time in 2934) [ClassicSimilarity], result of:
            0.02425402 = score(doc=2934,freq=1.0), product of:
              0.09296035 = queryWeight, product of:
                1.4001179 = boost
                4.1745143 = idf(docFreq=1768, maxDocs=42306)
                0.015904764 = queryNorm
              0.26090714 = fieldWeight in 2934, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.1745143 = idf(docFreq=1768, maxDocs=42306)
                0.0625 = fieldNorm(doc=2934)
          0.036948945 = weight(abstract_txt:document in 2934) [ClassicSimilarity], result of:
            0.036948945 = score(doc=2934,freq=2.0), product of:
              0.09768622 = queryWeight, product of:
                1.4352659 = boost
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.015904764 = queryNorm
              0.37824112 = fieldWeight in 2934, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.0625 = fieldNorm(doc=2934)
          0.045054797 = weight(abstract_txt:relevant in 2934) [ClassicSimilarity], result of:
            0.045054797 = score(doc=2934,freq=1.0), product of:
              0.15461403 = queryWeight, product of:
                2.0850167 = boost
                4.662428 = idf(docFreq=1085, maxDocs=42306)
                0.015904764 = queryNorm
              0.29140174 = fieldWeight in 2934, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.662428 = idf(docFreq=1085, maxDocs=42306)
                0.0625 = fieldNorm(doc=2934)
          0.11387547 = weight(abstract_txt:documents in 2934) [ClassicSimilarity], result of:
            0.11387547 = score(doc=2934,freq=6.0), product of:
              0.18072642 = queryWeight, product of:
                2.7608418 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.015904764 = queryNorm
              0.63009864 = fieldWeight in 2934, product of:
                2.4494898 = tf(freq=6.0), with freq of:
                  6.0 = termFreq=6.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.0625 = fieldNorm(doc=2934)
          0.5216845 = weight(abstract_txt:categorization in 2934) [ClassicSimilarity], result of:
            0.5216845 = score(doc=2934,freq=7.0), product of:
              0.47354138 = queryWeight, product of:
                4.468993 = boost
                6.6622515 = idf(docFreq=146, maxDocs=42306)
                0.015904764 = queryNorm
              1.1016662 = fieldWeight in 2934, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                6.6622515 = idf(docFreq=146, maxDocs=42306)
                0.0625 = fieldNorm(doc=2934)
        0.28 = coord(7/25)
    
  3. Goren-Bar, D.; Kuflik, T.: Supporting user-subjective categorization with self-organizing maps and learning vector quantization (2005) 0.21
    0.20651022 = sum of:
      0.20651022 = product of:
        0.86045927 = sum of:
          0.023789674 = weight(abstract_txt:human in 4326) [ClassicSimilarity], result of:
            0.023789674 = score(doc=4326,freq=1.0), product of:
              0.080168515 = queryWeight, product of:
                1.0616271 = boost
                4.7479334 = idf(docFreq=996, maxDocs=42306)
                0.015904764 = queryNorm
              0.29674584 = fieldWeight in 4326, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.7479334 = idf(docFreq=996, maxDocs=42306)
                0.0625 = fieldNorm(doc=4326)
          0.010708589 = weight(abstract_txt:that in 4326) [ClassicSimilarity], result of:
            0.010708589 = score(doc=4326,freq=3.0), product of:
              0.04113416 = queryWeight, product of:
                1.0754405 = boost
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.015904764 = queryNorm
              0.26033324 = fieldWeight in 4326, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.0625 = fieldNorm(doc=4326)
          0.010139737 = weight(abstract_txt:with in 4326) [ClassicSimilarity], result of:
            0.010139737 = score(doc=4326,freq=2.0), product of:
              0.045404196 = queryWeight, product of:
                1.129882 = boost
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.015904764 = queryNorm
              0.22332159 = fieldWeight in 4326, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.0625 = fieldNorm(doc=4326)
          0.0522537 = weight(abstract_txt:document in 4326) [ClassicSimilarity], result of:
            0.0522537 = score(doc=4326,freq=4.0), product of:
              0.09768622 = queryWeight, product of:
                1.4352659 = boost
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.015904764 = queryNorm
              0.5349137 = fieldWeight in 4326, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.0625 = fieldNorm(doc=4326)
          0.08052212 = weight(abstract_txt:documents in 4326) [ClassicSimilarity], result of:
            0.08052212 = score(doc=4326,freq=3.0), product of:
              0.18072642 = queryWeight, product of:
                2.7608418 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.015904764 = queryNorm
              0.445547 = fieldWeight in 4326, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.0625 = fieldNorm(doc=4326)
          0.68304545 = weight(abstract_txt:categorization in 4326) [ClassicSimilarity], result of:
            0.68304545 = score(doc=4326,freq=12.0), product of:
              0.47354138 = queryWeight, product of:
                4.468993 = boost
                6.6622515 = idf(docFreq=146, maxDocs=42306)
                0.015904764 = queryNorm
              1.4424198 = fieldWeight in 4326, product of:
                3.4641016 = tf(freq=12.0), with freq of:
                  12.0 = termFreq=12.0
                6.6622515 = idf(docFreq=146, maxDocs=42306)
                0.0625 = fieldNorm(doc=4326)
        0.24 = coord(6/25)
    
  4. Collins-Thompson, K.; Callan, J.: Predicting reading difficulty with statistical language models (2005) 0.17
    0.17325695 = sum of:
      0.17325695 = product of:
        0.54142797 = sum of:
          0.013824729 = weight(abstract_txt:that in 580) [ClassicSimilarity], result of:
            0.013824729 = score(doc=580,freq=5.0), product of:
              0.04113416 = queryWeight, product of:
                1.0754405 = boost
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.015904764 = queryNorm
              0.33608878 = fieldWeight in 580, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.0625 = fieldNorm(doc=580)
          0.027588356 = weight(abstract_txt:identify in 580) [ClassicSimilarity], result of:
            0.027588356 = score(doc=580,freq=1.0), product of:
              0.08849025 = queryWeight, product of:
                1.115367 = boost
                4.988275 = idf(docFreq=783, maxDocs=42306)
                0.015904764 = queryNorm
              0.3117672 = fieldWeight in 580, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.988275 = idf(docFreq=783, maxDocs=42306)
                0.0625 = fieldNorm(doc=580)
          0.010139737 = weight(abstract_txt:with in 580) [ClassicSimilarity], result of:
            0.010139737 = score(doc=580,freq=2.0), product of:
              0.045404196 = queryWeight, product of:
                1.129882 = boost
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.015904764 = queryNorm
              0.22332159 = fieldWeight in 580, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.0625 = fieldNorm(doc=580)
          0.02612685 = weight(abstract_txt:document in 580) [ClassicSimilarity], result of:
            0.02612685 = score(doc=580,freq=1.0), product of:
              0.09768622 = queryWeight, product of:
                1.4352659 = boost
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.015904764 = queryNorm
              0.26745686 = fieldWeight in 580, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.0625 = fieldNorm(doc=580)
          0.059319284 = weight(abstract_txt:automated in 580) [ClassicSimilarity], result of:
            0.059319284 = score(doc=580,freq=1.0), product of:
              0.16874823 = queryWeight, product of:
                1.8864068 = boost
                5.624406 = idf(docFreq=414, maxDocs=42306)
                0.015904764 = queryNorm
              0.35152537 = fieldWeight in 580, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.624406 = idf(docFreq=414, maxDocs=42306)
                0.0625 = fieldNorm(doc=580)
          0.045054797 = weight(abstract_txt:relevant in 580) [ClassicSimilarity], result of:
            0.045054797 = score(doc=580,freq=1.0), product of:
              0.15461403 = queryWeight, product of:
                2.0850167 = boost
                4.662428 = idf(docFreq=1085, maxDocs=42306)
                0.015904764 = queryNorm
              0.29140174 = fieldWeight in 580, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.662428 = idf(docFreq=1085, maxDocs=42306)
                0.0625 = fieldNorm(doc=580)
          0.08052212 = weight(abstract_txt:documents in 580) [ClassicSimilarity], result of:
            0.08052212 = score(doc=580,freq=3.0), product of:
              0.18072642 = queryWeight, product of:
                2.7608418 = boost
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.015904764 = queryNorm
              0.445547 = fieldWeight in 580, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.115787 = idf(docFreq=1875, maxDocs=42306)
                0.0625 = fieldNorm(doc=580)
          0.27885213 = weight(abstract_txt:categorization in 580) [ClassicSimilarity], result of:
            0.27885213 = score(doc=580,freq=2.0), product of:
              0.47354138 = queryWeight, product of:
                4.468993 = boost
                6.6622515 = idf(docFreq=146, maxDocs=42306)
                0.015904764 = queryNorm
              0.5888654 = fieldWeight in 580, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.6622515 = idf(docFreq=146, maxDocs=42306)
                0.0625 = fieldNorm(doc=580)
        0.32 = coord(8/25)
    
  5. Mearman, D.: Managing electronic records (1993) 0.16
    0.16139917 = sum of:
      0.16139917 = product of:
        0.5043724 = sum of:
          0.0077282586 = weight(abstract_txt:that in 1054) [ClassicSimilarity], result of:
            0.0077282586 = score(doc=1054,freq=1.0), product of:
              0.04113416 = queryWeight, product of:
                1.0754405 = boost
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.015904764 = queryNorm
              0.18787934 = fieldWeight in 1054, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.4048555 = idf(docFreq=10381, maxDocs=42306)
                0.078125 = fieldNorm(doc=1054)
          0.034485444 = weight(abstract_txt:identify in 1054) [ClassicSimilarity], result of:
            0.034485444 = score(doc=1054,freq=1.0), product of:
              0.08849025 = queryWeight, product of:
                1.115367 = boost
                4.988275 = idf(docFreq=783, maxDocs=42306)
                0.015904764 = queryNorm
              0.389709 = fieldWeight in 1054, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.988275 = idf(docFreq=783, maxDocs=42306)
                0.078125 = fieldNorm(doc=1054)
          0.008962346 = weight(abstract_txt:with in 1054) [ClassicSimilarity], result of:
            0.008962346 = score(doc=1054,freq=1.0), product of:
              0.045404196 = queryWeight, product of:
                1.129882 = boost
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.015904764 = queryNorm
              0.19739026 = fieldWeight in 1054, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.5265954 = idf(docFreq=9191, maxDocs=42306)
                0.078125 = fieldNorm(doc=1054)
          0.030317524 = weight(abstract_txt:time in 1054) [ClassicSimilarity], result of:
            0.030317524 = score(doc=1054,freq=1.0), product of:
              0.09296035 = queryWeight, product of:
                1.4001179 = boost
                4.1745143 = idf(docFreq=1768, maxDocs=42306)
                0.015904764 = queryNorm
              0.32613394 = fieldWeight in 1054, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.1745143 = idf(docFreq=1768, maxDocs=42306)
                0.078125 = fieldNorm(doc=1054)
          0.068879195 = weight(abstract_txt:legal in 1054) [ClassicSimilarity], result of:
            0.068879195 = score(doc=1054,freq=1.0), product of:
              0.14034498 = queryWeight, product of:
                1.4046513 = boost
                6.2820463 = idf(docFreq=214, maxDocs=42306)
                0.015904764 = queryNorm
              0.49078488 = fieldWeight in 1054, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.2820463 = idf(docFreq=214, maxDocs=42306)
                0.078125 = fieldNorm(doc=1054)
          0.032658566 = weight(abstract_txt:document in 1054) [ClassicSimilarity], result of:
            0.032658566 = score(doc=1054,freq=1.0), product of:
              0.09768622 = queryWeight, product of:
                1.4352659 = boost
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.015904764 = queryNorm
              0.33432108 = fieldWeight in 1054, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.2793097 = idf(docFreq=1592, maxDocs=42306)
                0.078125 = fieldNorm(doc=1054)
          0.07758145 = weight(abstract_txt:discovery in 1054) [ClassicSimilarity], result of:
            0.07758145 = score(doc=1054,freq=1.0), product of:
              0.1739164 = queryWeight, product of:
                1.9150759 = boost
                5.7098846 = idf(docFreq=380, maxDocs=42306)
                0.015904764 = queryNorm
              0.44608474 = fieldWeight in 1054, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.7098846 = idf(docFreq=380, maxDocs=42306)
                0.078125 = fieldNorm(doc=1054)
          0.24375965 = weight(abstract_txt:litigation in 1054) [ClassicSimilarity], result of:
            0.24375965 = score(doc=1054,freq=1.0), product of:
              0.3259213 = queryWeight, product of:
                2.1405554 = boost
                9.573242 = idf(docFreq=7, maxDocs=42306)
                0.015904764 = queryNorm
              0.74790955 = fieldWeight in 1054, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.573242 = idf(docFreq=7, maxDocs=42306)
                0.078125 = fieldNorm(doc=1054)
        0.32 = coord(8/25)