Document (#31100)

Author
Thelwall, M.
Stuart, D.
Title
Web crawling ethics revisited : cost, privacy, and denial of service
Source
Journal of the American Society for Information Science and Technology. 57(2006) no.13, S.1771-1779
Year
2006
Abstract
Ethical aspects of the employment of Web crawlers for information science research and other contexts are reviewed. The difference between legal and ethical uses of communications technologies is emphasized as well as the changing boundary between ethical and unethical conduct. A review of the potential impacts on Web site owners is used to underpin a new framework for ethical crawling, and it is argued that delicate human judgment is required for each individual case, with verdicts likely to change over time. Decisions can be based upon an approximate cost-benefit analysis, but it is crucial that crawler owners find out about the technological issues affecting the owners of the sites being crawled in order to produce an informed assessment.
Theme
Suchmaschinen

Similar documents (author)

  1. Angus, E.; Thelwall, M.; Stuart, D.: General patterns of tag usage among university groups in Flickr (2008) 4.28
    4.2779436 = sum of:
      4.2779436 = sum of:
        1.3018568 = weight(author_txt:thelwall in 4555) [ClassicSimilarity], result of:
          1.3018568 = score(doc=4555,freq=1.0), product of:
            0.4992855 = queryWeight, product of:
              6.9531717 = idf(docFreq=108, maxDocs=41962)
              0.07180687 = queryNorm
            2.6074395 = fieldWeight in 4555, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              6.9531717 = idf(docFreq=108, maxDocs=41962)
              0.375 = fieldNorm(doc=4555)
        2.9760869 = weight(author_txt:stuart in 4555) [ClassicSimilarity], result of:
          2.9760869 = score(doc=4555,freq=1.0), product of:
            0.8664375 = queryWeight, product of:
              1.3173287 = boost
              9.159613 = idf(docFreq=11, maxDocs=41962)
              0.07180687 = queryNorm
            3.4348547 = fieldWeight in 4555, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.159613 = idf(docFreq=11, maxDocs=41962)
              0.375 = fieldNorm(doc=4555)
    
  2. Thelwall, M.; Klitkou, A.; Verbeek, A.; Stuart, D.; Vincent, C.: Policy-relevant Webometrics for individual scientific fields (2010) 3.56
    3.5649529 = sum of:
      3.5649529 = sum of:
        1.0848805 = weight(author_txt:thelwall in 575) [ClassicSimilarity], result of:
          1.0848805 = score(doc=575,freq=1.0), product of:
            0.4992855 = queryWeight, product of:
              6.9531717 = idf(docFreq=108, maxDocs=41962)
              0.07180687 = queryNorm
            2.172866 = fieldWeight in 575, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              6.9531717 = idf(docFreq=108, maxDocs=41962)
              0.3125 = fieldNorm(doc=575)
        2.4800725 = weight(author_txt:stuart in 575) [ClassicSimilarity], result of:
          2.4800725 = score(doc=575,freq=1.0), product of:
            0.8664375 = queryWeight, product of:
              1.3173287 = boost
              9.159613 = idf(docFreq=11, maxDocs=41962)
              0.07180687 = queryNorm
            2.862379 = fieldWeight in 575, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.159613 = idf(docFreq=11, maxDocs=41962)
              0.3125 = fieldNorm(doc=575)
    
  3. Thelwall, M.; Goriunova, O.; Vis, F.; Faulkner, S.; Burns, A.; Aulich, J.; Mas-Bleda, A.; Stuart, E.; D'Orazio, F.: Chatting through pictures : a classification of images tweeted in one week in the UK and USA (2016) 2.50
    2.495467 = sum of:
      2.495467 = sum of:
        0.7594164 = weight(author_txt:thelwall in 216) [ClassicSimilarity], result of:
          0.7594164 = score(doc=216,freq=1.0), product of:
            0.4992855 = queryWeight, product of:
              6.9531717 = idf(docFreq=108, maxDocs=41962)
              0.07180687 = queryNorm
            1.5210063 = fieldWeight in 216, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              6.9531717 = idf(docFreq=108, maxDocs=41962)
              0.21875 = fieldNorm(doc=216)
        1.7360506 = weight(author_txt:stuart in 216) [ClassicSimilarity], result of:
          1.7360506 = score(doc=216,freq=1.0), product of:
            0.8664375 = queryWeight, product of:
              1.3173287 = boost
              9.159613 = idf(docFreq=11, maxDocs=41962)
              0.07180687 = queryNorm
            2.0036652 = fieldWeight in 216, product of:
              1.0 = tf(freq=1.0), with freq of:
                1.0 = termFreq=1.0
              9.159613 = idf(docFreq=11, maxDocs=41962)
              0.21875 = fieldNorm(doc=216)
    
  4. Stuart, D.: Web metrics for library and information professionals (2014) 2.48
    2.4800725 = sum of:
      2.4800725 = product of:
        4.960145 = sum of:
          4.960145 = weight(author_txt:stuart in 4275) [ClassicSimilarity], result of:
            4.960145 = score(doc=4275,freq=1.0), product of:
              0.8664375 = queryWeight, product of:
                1.3173287 = boost
                9.159613 = idf(docFreq=11, maxDocs=41962)
                0.07180687 = queryNorm
              5.724758 = fieldWeight in 4275, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.159613 = idf(docFreq=11, maxDocs=41962)
                0.625 = fieldNorm(doc=4275)
        0.5 = coord(1/2)
    
  5. Stuart, L.M.; Harwell, K.R.: LEXIS-NEXIS instruction at Penn State (1997) 1.98
    1.9840579 = sum of:
      1.9840579 = product of:
        3.9681158 = sum of:
          3.9681158 = weight(author_txt:stuart in 6690) [ClassicSimilarity], result of:
            3.9681158 = score(doc=6690,freq=1.0), product of:
              0.8664375 = queryWeight, product of:
                1.3173287 = boost
                9.159613 = idf(docFreq=11, maxDocs=41962)
                0.07180687 = queryNorm
              4.5798063 = fieldWeight in 6690, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                9.159613 = idf(docFreq=11, maxDocs=41962)
                0.5 = fieldNorm(doc=6690)
        0.5 = coord(1/2)
    

Similar documents (content)

  1. Frohmann, B.: Subjectivity and information ethics (2008) 0.15
    0.14772013 = sum of:
      0.14772013 = product of:
        0.9232508 = sum of:
          0.01602702 = weight(abstract_txt:between in 3361) [ClassicSimilarity], result of:
            0.01602702 = score(doc=3361,freq=2.0), product of:
              0.059078053 = queryWeight, product of:
                1.1419696 = boost
                3.5077088 = idf(docFreq=3417, maxDocs=41962)
                0.014748508 = queryNorm
              0.27128553 = fieldWeight in 3361, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.5077088 = idf(docFreq=3417, maxDocs=41962)
                0.0546875 = fieldNorm(doc=3361)
          0.064221345 = weight(abstract_txt:privacy in 3361) [ClassicSimilarity], result of:
            0.064221345 = score(doc=3361,freq=2.0), product of:
              0.11829524 = queryWeight, product of:
                1.1426418 = boost
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.014748508 = queryNorm
              0.54289037 = fieldWeight in 3361, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.0546875 = fieldNorm(doc=3361)
          0.45285073 = weight(title_txt:ethics in 3361) [ClassicSimilarity], result of:
            0.45285073 = score(doc=3361,freq=1.0), product of:
              0.12534483 = queryWeight, product of:
                1.1761959 = boost
                7.225679 = idf(docFreq=82, maxDocs=41962)
                0.014748508 = queryNorm
              3.6128395 = fieldWeight in 3361, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.225679 = idf(docFreq=82, maxDocs=41962)
                0.5 = fieldNorm(doc=3361)
          0.3901517 = weight(abstract_txt:ethical in 3361) [ClassicSimilarity], result of:
            0.3901517 = score(doc=3361,freq=5.0), product of:
              0.46065593 = queryWeight, product of:
                4.5096703 = boost
                6.9260206 = idf(docFreq=111, maxDocs=41962)
                0.014748508 = queryNorm
              0.8469482 = fieldWeight in 3361, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                6.9260206 = idf(docFreq=111, maxDocs=41962)
                0.0546875 = fieldNorm(doc=3361)
        0.16 = coord(4/25)
    
  2. Danielson, E.S.: Ethics and reference services (1997) 0.13
    0.13447875 = sum of:
      0.13447875 = product of:
        1.1206563 = sum of:
          0.10379737 = weight(abstract_txt:privacy in 896) [ClassicSimilarity], result of:
            0.10379737 = score(doc=896,freq=1.0), product of:
              0.11829524 = queryWeight, product of:
                1.1426418 = boost
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.014748508 = queryNorm
              0.8774434 = fieldWeight in 896, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.125 = fieldNorm(doc=896)
          0.45285073 = weight(title_txt:ethics in 896) [ClassicSimilarity], result of:
            0.45285073 = score(doc=896,freq=1.0), product of:
              0.12534483 = queryWeight, product of:
                1.1761959 = boost
                7.225679 = idf(docFreq=82, maxDocs=41962)
                0.014748508 = queryNorm
              3.6128395 = fieldWeight in 896, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.225679 = idf(docFreq=82, maxDocs=41962)
                0.5 = fieldNorm(doc=896)
          0.56400824 = weight(abstract_txt:ethical in 896) [ClassicSimilarity], result of:
            0.56400824 = score(doc=896,freq=2.0), product of:
              0.46065593 = queryWeight, product of:
                4.5096703 = boost
                6.9260206 = idf(docFreq=111, maxDocs=41962)
                0.014748508 = queryNorm
              1.224359 = fieldWeight in 896, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.9260206 = idf(docFreq=111, maxDocs=41962)
                0.125 = fieldNorm(doc=896)
        0.12 = coord(3/25)
    
  3. Polat, H.; Du, W.: Privacy-preserving top-N recommendation on distributed data (2008) 0.12
    0.12454579 = sum of:
      0.12454579 = product of:
        0.62272894 = sum of:
          0.046735004 = weight(abstract_txt:legal in 3865) [ClassicSimilarity], result of:
            0.046735004 = score(doc=3865,freq=1.0), product of:
              0.095064394 = queryWeight, product of:
                1.0243194 = boost
                6.2926617 = idf(docFreq=210, maxDocs=41962)
                0.014748508 = queryNorm
              0.4916142 = fieldWeight in 3865, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.2926617 = idf(docFreq=210, maxDocs=41962)
                0.078125 = fieldNorm(doc=3865)
          0.05684821 = weight(abstract_txt:conduct in 3865) [ClassicSimilarity], result of:
            0.05684821 = score(doc=3865,freq=1.0), product of:
              0.108326375 = queryWeight, product of:
                1.0934365 = boost
                6.717266 = idf(docFreq=137, maxDocs=41962)
                0.014748508 = queryNorm
              0.5247864 = fieldWeight in 3865, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.717266 = idf(docFreq=137, maxDocs=41962)
                0.078125 = fieldNorm(doc=3865)
          0.016189735 = weight(abstract_txt:between in 3865) [ClassicSimilarity], result of:
            0.016189735 = score(doc=3865,freq=1.0), product of:
              0.059078053 = queryWeight, product of:
                1.1419696 = boost
                3.5077088 = idf(docFreq=3417, maxDocs=41962)
                0.014748508 = queryNorm
              0.27403975 = fieldWeight in 3865, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.5077088 = idf(docFreq=3417, maxDocs=41962)
                0.078125 = fieldNorm(doc=3865)
          0.14506124 = weight(abstract_txt:privacy in 3865) [ClassicSimilarity], result of:
            0.14506124 = score(doc=3865,freq=5.0), product of:
              0.11829524 = queryWeight, product of:
                1.1426418 = boost
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.014748508 = queryNorm
              1.2262644 = fieldWeight in 3865, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.078125 = fieldNorm(doc=3865)
          0.35789475 = weight(abstract_txt:owners in 3865) [ClassicSimilarity], result of:
            0.35789475 = score(doc=3865,freq=1.0), product of:
              0.5326807 = queryWeight, product of:
                4.1997223 = boost
                8.5999975 = idf(docFreq=20, maxDocs=41962)
                0.014748508 = queryNorm
              0.6718748 = fieldWeight in 3865, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.5999975 = idf(docFreq=20, maxDocs=41962)
                0.078125 = fieldNorm(doc=3865)
        0.2 = coord(5/25)
    
  4. Himma, K.E.: Foundational issues in information ethics (2007) 0.12
    0.12280563 = sum of:
      0.12280563 = product of:
        0.7675352 = sum of:
          0.037388004 = weight(abstract_txt:legal in 4592) [ClassicSimilarity], result of:
            0.037388004 = score(doc=4592,freq=1.0), product of:
              0.095064394 = queryWeight, product of:
                1.0243194 = boost
                6.2926617 = idf(docFreq=210, maxDocs=41962)
                0.014748508 = queryNorm
              0.39329135 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.2926617 = idf(docFreq=210, maxDocs=41962)
                0.0625 = fieldNorm(doc=4592)
          0.051898684 = weight(abstract_txt:privacy in 4592) [ClassicSimilarity], result of:
            0.051898684 = score(doc=4592,freq=1.0), product of:
              0.11829524 = queryWeight, product of:
                1.1426418 = boost
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.014748508 = queryNorm
              0.4387217 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.0625 = fieldNorm(doc=4592)
          0.3962444 = weight(title_txt:ethics in 4592) [ClassicSimilarity], result of:
            0.3962444 = score(doc=4592,freq=1.0), product of:
              0.12534483 = queryWeight, product of:
                1.1761959 = boost
                7.225679 = idf(docFreq=82, maxDocs=41962)
                0.014748508 = queryNorm
              3.1612346 = fieldWeight in 4592, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.225679 = idf(docFreq=82, maxDocs=41962)
                0.4375 = fieldNorm(doc=4592)
          0.28200412 = weight(abstract_txt:ethical in 4592) [ClassicSimilarity], result of:
            0.28200412 = score(doc=4592,freq=2.0), product of:
              0.46065593 = queryWeight, product of:
                4.5096703 = boost
                6.9260206 = idf(docFreq=111, maxDocs=41962)
                0.014748508 = queryNorm
              0.6121795 = fieldWeight in 4592, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                6.9260206 = idf(docFreq=111, maxDocs=41962)
                0.0625 = fieldNorm(doc=4592)
        0.16 = coord(4/25)
    
  5. Fernández-Molina, J.C.; Chaves Guimaraes, J.A.: Ethical aspects of knowledge organization and representation in the digital environment : their articulation in professional codes of ethics (2003) 0.12
    0.12025162 = sum of:
      0.12025162 = product of:
        0.7515726 = sum of:
          0.05684821 = weight(abstract_txt:conduct in 3766) [ClassicSimilarity], result of:
            0.05684821 = score(doc=3766,freq=1.0), product of:
              0.108326375 = queryWeight, product of:
                1.0934365 = boost
                6.717266 = idf(docFreq=137, maxDocs=41962)
                0.014748508 = queryNorm
              0.5247864 = fieldWeight in 3766, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.717266 = idf(docFreq=137, maxDocs=41962)
                0.078125 = fieldNorm(doc=3766)
          0.06487336 = weight(abstract_txt:privacy in 3766) [ClassicSimilarity], result of:
            0.06487336 = score(doc=3766,freq=1.0), product of:
              0.11829524 = queryWeight, product of:
                1.1426418 = boost
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.014748508 = queryNorm
              0.54840213 = fieldWeight in 3766, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.019547 = idf(docFreq=101, maxDocs=41962)
                0.078125 = fieldNorm(doc=3766)
          0.1981222 = weight(title_txt:ethics in 3766) [ClassicSimilarity], result of:
            0.1981222 = score(doc=3766,freq=1.0), product of:
              0.12534483 = queryWeight, product of:
                1.1761959 = boost
                7.225679 = idf(docFreq=82, maxDocs=41962)
                0.014748508 = queryNorm
              1.5806173 = fieldWeight in 3766, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.225679 = idf(docFreq=82, maxDocs=41962)
                0.21875 = fieldNorm(doc=3766)
          0.43172887 = weight(abstract_txt:ethical in 3766) [ClassicSimilarity], result of:
            0.43172887 = score(doc=3766,freq=3.0), product of:
              0.46065593 = queryWeight, product of:
                4.5096703 = boost
                6.9260206 = idf(docFreq=111, maxDocs=41962)
                0.014748508 = queryNorm
              0.93720466 = fieldWeight in 3766, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                6.9260206 = idf(docFreq=111, maxDocs=41962)
                0.078125 = fieldNorm(doc=3766)
        0.16 = coord(4/25)