Document (#31099)

Author

Thelwall, M.
Stuart, D.

Title

Web crawling ethics revisited : cost, privacy, and denial of service

Source

Journal of the American Society for Information Science and Technology. 57(2006) no.13, S.1771-1779

Year

2006

Abstract

Ethical aspects of the employment of Web crawlers for information science research and other contexts are reviewed. The difference between legal and ethical uses of communications technologies is emphasized as well as the changing boundary between ethical and unethical conduct. A review of the potential impacts on Web site owners is used to underpin a new framework for ethical crawling, and it is argued that delicate human judgment is required for each individual case, with verdicts likely to change over time. Decisions can be based upon an approximate cost-benefit analysis, but it is crucial that crawler owners find out about the technological issues affecting the owners of the sites being crawled in order to produce an informed assessment.

Theme

Suchmaschinen

Similar documents (author)

1. Angus, E.; Thelwall, M.; Stuart, D.: General patterns of tag usage among university groups in Flickr (2008) 4.19

   4.190403 = sum of:
     4.190403 = sum of:
       1.3535643 = weight(author_txt:thelwall in 2554) [ClassicSimilarity], result of:
         1.3535643 = score(doc=2554,freq=1.0), product of:
           0.5211376 = queryWeight, product of:
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.07524146 = queryNorm
           2.597326 = fieldWeight in 2554, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.375 = fieldNorm(doc=2554)
       2.8368387 = weight(author_txt:stuart in 2554) [ClassicSimilarity], result of:
         2.8368387 = score(doc=2554,freq=1.0), product of:
           0.8534726 = queryWeight, product of:
             1.2797307 = boost
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.07524146 = queryNorm
           3.3238778 = fieldWeight in 2554, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.375 = fieldNorm(doc=2554)
   

2. Thelwall, M.; Klitkou, A.; Verbeek, A.; Stuart, D.; Vincent, C.: Policy-relevant Webometrics for individual scientific fields (2010) 3.49

   3.4920025 = sum of:
     3.4920025 = sum of:
       1.1279701 = weight(author_txt:thelwall in 3574) [ClassicSimilarity], result of:
         1.1279701 = score(doc=3574,freq=1.0), product of:
           0.5211376 = queryWeight, product of:
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.07524146 = queryNorm
           2.1644382 = fieldWeight in 3574, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.3125 = fieldNorm(doc=3574)
       2.3640323 = weight(author_txt:stuart in 3574) [ClassicSimilarity], result of:
         2.3640323 = score(doc=3574,freq=1.0), product of:
           0.8534726 = queryWeight, product of:
             1.2797307 = boost
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.07524146 = queryNorm
           2.7698982 = fieldWeight in 3574, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.3125 = fieldNorm(doc=3574)
   

3. Thelwall, M.; Kousha, K.; Abdoli, M.; Stuart, E.; Makita, M.; Wilson, P.; Levitt, J.: Do altmetric scores reflect article quality? : evidence from the UK Research Excellence Framework 2021 (2023) 2.79

   2.793602 = sum of:
     2.793602 = sum of:
       0.9023762 = weight(author_txt:thelwall in 947) [ClassicSimilarity], result of:
         0.9023762 = score(doc=947,freq=1.0), product of:
           0.5211376 = queryWeight, product of:
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.07524146 = queryNorm
           1.7315507 = fieldWeight in 947, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.25 = fieldNorm(doc=947)
       1.8912257 = weight(author_txt:stuart in 947) [ClassicSimilarity], result of:
         1.8912257 = score(doc=947,freq=1.0), product of:
           0.8534726 = queryWeight, product of:
             1.2797307 = boost
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.07524146 = queryNorm
           2.2159185 = fieldWeight in 947, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.25 = fieldNorm(doc=947)
   

4. Thelwall, M.; Kousha, K.; Abdoli, M.; Stuart, E.; Makita, M.; Wilson, P.; Levitt, J.: Why are coauthored academic articles more cited : higher quality or larger audience? (2023) 2.79

   2.793602 = sum of:
     2.793602 = sum of:
       0.9023762 = weight(author_txt:thelwall in 995) [ClassicSimilarity], result of:
         0.9023762 = score(doc=995,freq=1.0), product of:
           0.5211376 = queryWeight, product of:
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.07524146 = queryNorm
           1.7315507 = fieldWeight in 995, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.25 = fieldNorm(doc=995)
       1.8912257 = weight(author_txt:stuart in 995) [ClassicSimilarity], result of:
         1.8912257 = score(doc=995,freq=1.0), product of:
           0.8534726 = queryWeight, product of:
             1.2797307 = boost
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.07524146 = queryNorm
           2.2159185 = fieldWeight in 995, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.25 = fieldNorm(doc=995)
   

5. Thelwall, M.; Kousha, K.; Stuart, E.; Makita, M.; Abdoli, M.; Wilson, P.; Levitt, J.: In which fields are citations indicators of research quality? (2023) 2.79

   2.793602 = sum of:
     2.793602 = sum of:
       0.9023762 = weight(author_txt:thelwall in 1033) [ClassicSimilarity], result of:
         0.9023762 = score(doc=1033,freq=1.0), product of:
           0.5211376 = queryWeight, product of:
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.07524146 = queryNorm
           1.7315507 = fieldWeight in 1033, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             6.926203 = idf(docFreq=117, maxDocs=44218)
             0.25 = fieldNorm(doc=1033)
       1.8912257 = weight(author_txt:stuart in 1033) [ClassicSimilarity], result of:
         1.8912257 = score(doc=1033,freq=1.0), product of:
           0.8534726 = queryWeight, product of:
             1.2797307 = boost
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.07524146 = queryNorm
           2.2159185 = fieldWeight in 1033, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.863674 = idf(docFreq=16, maxDocs=44218)
             0.25 = fieldNorm(doc=1033)
   

Similar documents (content)

1. Rubin, R.; Froehlich, T.J.: Ethical aspects of library and information science (2009) 0.12

   0.12444674 = sum of:
     0.12444674 = product of:
       0.77779216 = sum of:
         0.07537209 = weight(abstract_txt:conduct in 3778) [ClassicSimilarity], result of:
           0.07537209 = score(doc=3778,freq=2.0), product of:
             0.1028279 = queryWeight, product of:
               1.057774 = boost
               6.634292 = idf(docFreq=157, maxDocs=44218)
               0.014652897 = queryNorm
             0.73299265 = fieldWeight in 3778, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.634292 = idf(docFreq=157, maxDocs=44218)
               0.078125 = fieldNorm(doc=3778)
         0.056992 = weight(abstract_txt:privacy in 3778) [ClassicSimilarity], result of:
           0.056992 = score(doc=3778,freq=1.0), product of:
             0.107528396 = queryWeight, product of:
               1.0816804 = boost
               6.784232 = idf(docFreq=135, maxDocs=44218)
               0.014652897 = queryNorm
             0.53001815 = fieldWeight in 3778, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.784232 = idf(docFreq=135, maxDocs=44218)
               0.078125 = fieldNorm(doc=3778)
         0.06455393 = weight(abstract_txt:ethics in 3778) [ClassicSimilarity], result of:
           0.06455393 = score(doc=3778,freq=1.0), product of:
             0.11684112 = queryWeight, product of:
               1.1275486 = boost
               7.071914 = idf(docFreq=101, maxDocs=44218)
               0.014652897 = queryNorm
             0.5524933 = fieldWeight in 3778, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.071914 = idf(docFreq=101, maxDocs=44218)
               0.078125 = fieldNorm(doc=3778)
         0.58087415 = weight(abstract_txt:ethical in 3778) [ClassicSimilarity], result of:
           0.58087415 = score(doc=3778,freq=7.0), product of:
             0.41945875 = queryWeight, product of:
               4.2727947 = boost
               6.699675 = idf(docFreq=147, maxDocs=44218)
               0.014652897 = queryNorm
             1.3848183 = fieldWeight in 3778, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               6.699675 = idf(docFreq=147, maxDocs=44218)
               0.078125 = fieldNorm(doc=3778)
       0.16 = coord(4/25)
   

2. Polat, H.; Du, W.: Privacy-preserving top-N recommendation on distributed data (2008) 0.12

   0.11912227 = sum of:
     0.11912227 = product of:
       0.59561133 = sum of:
         0.045031555 = weight(abstract_txt:legal in 1864) [ClassicSimilarity], result of:
           0.045031555 = score(doc=1864,freq=1.0), product of:
             0.09190205 = queryWeight, product of:
               6.2719374 = idf(docFreq=226, maxDocs=44218)
               0.014652897 = queryNorm
             0.48999512 = fieldWeight in 1864, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.2719374 = idf(docFreq=226, maxDocs=44218)
               0.078125 = fieldNorm(doc=1864)
         0.053296115 = weight(abstract_txt:conduct in 1864) [ClassicSimilarity], result of:
           0.053296115 = score(doc=1864,freq=1.0), product of:
             0.1028279 = queryWeight, product of:
               1.057774 = boost
               6.634292 = idf(docFreq=157, maxDocs=44218)
               0.014652897 = queryNorm
             0.51830405 = fieldWeight in 1864, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.634292 = idf(docFreq=157, maxDocs=44218)
               0.078125 = fieldNorm(doc=1864)
         0.12743798 = weight(abstract_txt:privacy in 1864) [ClassicSimilarity], result of:
           0.12743798 = score(doc=1864,freq=5.0), product of:
             0.107528396 = queryWeight, product of:
               1.0816804 = boost
               6.784232 = idf(docFreq=135, maxDocs=44218)
               0.014652897 = queryNorm
             1.1851566 = fieldWeight in 1864, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.784232 = idf(docFreq=135, maxDocs=44218)
               0.078125 = fieldNorm(doc=1864)
         0.015165049 = weight(abstract_txt:between in 1864) [ClassicSimilarity], result of:
           0.015165049 = score(doc=1864,freq=1.0), product of:
             0.056047093 = queryWeight, product of:
               1.1044065 = boost
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.014652897 = queryNorm
             0.2705769 = fieldWeight in 1864, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.078125 = fieldNorm(doc=1864)
         0.35468066 = weight(abstract_txt:owners in 1864) [ClassicSimilarity], result of:
           0.35468066 = score(doc=1864,freq=1.0), product of:
             0.5247019 = queryWeight, product of:
               4.1386085 = boost
               8.652365 = idf(docFreq=20, maxDocs=44218)
               0.014652897 = queryNorm
             0.675966 = fieldWeight in 1864, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.652365 = idf(docFreq=20, maxDocs=44218)
               0.078125 = fieldNorm(doc=1864)
       0.2 = coord(5/25)
   

3. MacFarlane, A.; Missaoui, S.; Makri, S.; Gutierrez Lopez, M.: Sender vs. recipient-orientated information systems revisited (2022) 0.11

   0.108179316 = sum of:
     0.108179316 = product of:
       0.67612076 = sum of:
         0.03161751 = weight(abstract_txt:argued in 607) [ClassicSimilarity], result of:
           0.03161751 = score(doc=607,freq=1.0), product of:
             0.10205435 = queryWeight, product of:
               1.0537878 = boost
               6.609291 = idf(docFreq=161, maxDocs=44218)
               0.014652897 = queryNorm
             0.30981052 = fieldWeight in 607, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.609291 = idf(docFreq=161, maxDocs=44218)
               0.046875 = fieldNorm(doc=607)
         0.03873236 = weight(abstract_txt:ethics in 607) [ClassicSimilarity], result of:
           0.03873236 = score(doc=607,freq=1.0), product of:
             0.11684112 = queryWeight, product of:
               1.1275486 = boost
               7.071914 = idf(docFreq=101, maxDocs=44218)
               0.014652897 = queryNorm
             0.33149597 = fieldWeight in 607, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.071914 = idf(docFreq=101, maxDocs=44218)
               0.046875 = fieldNorm(doc=607)
         0.37760806 = weight(title_txt:revisited in 607) [ClassicSimilarity], result of:
           0.37760806 = score(doc=607,freq=1.0), product of:
             0.13330525 = queryWeight, product of:
               1.2043731 = boost
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.014652897 = queryNorm
             2.832657 = fieldWeight in 607, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.5537524 = idf(docFreq=62, maxDocs=44218)
               0.375 = fieldNorm(doc=607)
         0.22816283 = weight(abstract_txt:ethical in 607) [ClassicSimilarity], result of:
           0.22816283 = score(doc=607,freq=3.0), product of:
             0.41945875 = queryWeight, product of:
               4.2727947 = boost
               6.699675 = idf(docFreq=147, maxDocs=44218)
               0.014652897 = queryNorm
             0.5439458 = fieldWeight in 607, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.699675 = idf(docFreq=147, maxDocs=44218)
               0.046875 = fieldNorm(doc=607)
       0.16 = coord(4/25)
   

4. Frohmann, B.: Subjectivity and information ethics (2008) 0.09

   0.092481345 = sum of:
     0.092481345 = product of:
       0.5780084 = sum of:
         0.056419197 = weight(abstract_txt:privacy in 1360) [ClassicSimilarity], result of:
           0.056419197 = score(doc=1360,freq=2.0), product of:
             0.107528396 = queryWeight, product of:
               1.0816804 = boost
               6.784232 = idf(docFreq=135, maxDocs=44218)
               0.014652897 = queryNorm
             0.52469116 = fieldWeight in 1360, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.784232 = idf(docFreq=135, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1360)
         0.015012632 = weight(abstract_txt:between in 1360) [ClassicSimilarity], result of:
           0.015012632 = score(doc=1360,freq=2.0), product of:
             0.056047093 = queryWeight, product of:
               1.1044065 = boost
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.014652897 = queryNorm
             0.26785746 = fieldWeight in 1360, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1360)
         0.16292678 = weight(abstract_txt:ethics in 1360) [ClassicSimilarity], result of:
           0.16292678 = score(doc=1360,freq=13.0), product of:
             0.11684112 = queryWeight, product of:
               1.1275486 = boost
               7.071914 = idf(docFreq=101, maxDocs=44218)
               0.014652897 = queryNorm
             1.39443 = fieldWeight in 1360, product of:
               3.6055512 = tf(freq=13.0), with freq of:
                 13.0 = termFreq=13.0
               7.071914 = idf(docFreq=101, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1360)
         0.3436498 = weight(abstract_txt:ethical in 1360) [ClassicSimilarity], result of:
           0.3436498 = score(doc=1360,freq=5.0), product of:
             0.41945875 = queryWeight, product of:
               4.2727947 = boost
               6.699675 = idf(docFreq=147, maxDocs=44218)
               0.014652897 = queryNorm
             0.8192696 = fieldWeight in 1360, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.699675 = idf(docFreq=147, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1360)
       0.16 = coord(4/25)
   

5. Van der Walt, M.S.: Normative ethics in knowledge organisation (2008) 0.09

   0.0868553 = sum of:
     0.0868553 = product of:
       0.72379416 = sum of:
         0.08527379 = weight(abstract_txt:conduct in 1696) [ClassicSimilarity], result of:
           0.08527379 = score(doc=1696,freq=4.0), product of:
             0.1028279 = queryWeight, product of:
               1.057774 = boost
               6.634292 = idf(docFreq=157, maxDocs=44218)
               0.014652897 = queryNorm
             0.8292865 = fieldWeight in 1696, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.634292 = idf(docFreq=157, maxDocs=44218)
               0.0625 = fieldNorm(doc=1696)
         0.24577777 = weight(abstract_txt:unethical in 1696) [ClassicSimilarity], result of:
           0.24577777 = score(doc=1696,freq=3.0), product of:
             0.2292144 = queryWeight, product of:
               1.5792772 = boost
               9.905128 = idf(docFreq=5, maxDocs=44218)
               0.014652897 = queryNorm
             1.0722615 = fieldWeight in 1696, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               9.905128 = idf(docFreq=5, maxDocs=44218)
               0.0625 = fieldNorm(doc=1696)
         0.3927426 = weight(abstract_txt:ethical in 1696) [ClassicSimilarity], result of:
           0.3927426 = score(doc=1696,freq=5.0), product of:
             0.41945875 = queryWeight, product of:
               4.2727947 = boost
               6.699675 = idf(docFreq=147, maxDocs=44218)
               0.014652897 = queryNorm
             0.9363081 = fieldWeight in 1696, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.699675 = idf(docFreq=147, maxDocs=44218)
               0.0625 = fieldNorm(doc=1696)
       0.12 = coord(3/25)