000 04730cam a2200433Mi 4500
999 _c89239
_d89239
005 20241205124536.0
006 m o d
007 cr |||||||||||
008 241205b ||||| |||| 00| 0 eng d
020 _a9781848219045
020 _ahttps://onlinelibrary.wiley.com/doi/book/10.1002/9781119306696
_q(electronic bk.)
020 _a1119307651
020 _a9781119306696
_q(electronic bk.)
020 _a9781119307655
_qEPUB
035 _a(OCoLC)953865897
037 _a9781119307655
_bWiley
040 _aNLE
_beng
_erda
_epn
_cNLE
_dDG1
041 _aeng
050 4 _aQA76.9.N38
082 0 4 _a006.3/5
_223
100 1 _aFort, Karen,
_0http://id.loc.gov/authorities/names/no2016109001
_eauthor.
245 1 0 _aCollaborative annotation for reliable natural language processing :
_btechnical and sociological aspects /
_cKaren Fort.
250 _a1st.
264 1 _aLondon :
_bWiley-ISTE,
_c2016.
300 _a1 online resource.
336 _atext
_btxt
_2rdacontent.
337 _acomputer
_bc
_2rdamedia.
338 _aonline resource
_bcr
_2rdacarrier.
490 1 _aFocus series.
504 _aIncludes bibliographical references and index.
505 0 _aTable of Contents Preface ix List of Acronyms xi Introduction xiii Chapter 1. Annotating Collaboratively 1 1.1. The annotation process (re)visited 1 1.1.1. Building consensus 1 1.1.2. Existing methodologies 3 1.1.3. Preparatory work 7 1.1.4. Pre-campaign 13 1.1.5. Annotation 17 1.1.6. Finalization 21 1.2. Annotation complexity 24 1.2.1. Example overview 25 1.2.2. What to annotate? 28 1.2.3. How to annotate? 30 1.2.4. The weight of the context 36 1.2.5. Visualization 38 1.2.6. Elementary annotation tasks 40 1.3. Annotation tools 43 1.3.1. To be or not to be an annotation tool 43 1.3.2. Much more than prototypes 46 1.3.3. Addressing the new annotation challenges 49 1.3.4. The impossible dream tool 54 1.4. Evaluating the annotation quality 55 1.4.1. What is annotation quality? 55 1.4.2. Understanding the basics 56 1.4.3. Beyond kappas 63 1.4.4. Giving meaning to the metrics 67 1.5. Conclusion 75 Chapter 2. Crowdsourcing Annotation 77 2.1. What is crowdsourcing and why should we be interested in it? 77 2.1.1. A moving target 77 2.1.2. A massive success 80 2.2. Deconstructing the myths 81 2.2.1. Crowdsourcing is a recent phenomenon 81 2.2.2. Crowdsourcing involves a crowd (of non-experts) 83 2.2.3. “Crowdsourcing involves (a crowd of) non-experts” 87 2.3. Playing with a purpose 93 2.3.1. Using the players’ innate capabilities and world knowledge 94 2.3.2. Using the players’ school knowledge 96 2.3.3. Using the players’ learning capacities 97 2.4. Acknowledging crowdsourcing specifics 101 2.4.1. Motivating the participants 101 2.4.2. Producing quality data 107 2.5. Ethical issues 109 2.5.1. Game ethics 109 2.5.2. What’s wrong with Amazon Mechanical Turk? 111 2.5.3. A charter to rule them all 113 Conclusion 115 Appendix 117 Glossary 141 Bibliography 143 Index 163
520 _aThis book presents a unique opportunity for constructing a consistent image of collaborative manual annotation for Natural Language Processing (NLP). NLP has witnessed two major evolutions in the past 25 years: firstly, the extraordinary success of machine learning, which is now, for better or for worse, overwhelmingly dominant in the field, and secondly, the multiplication of evaluation campaigns or shared tasks. Both involve manually annotated corpora, for the training and evaluation of the systems. These corpora have progressively become the hidden pillars of our domain, providing food for our hungry machine learning algorithms and reference for evaluation. Annotation is now the place where linguistics hides in NLP. However, manual annotation has largely been ignored for some time, and it has taken a while even for annotation guidelines to be recognized as essential.
545 0 _aAbout the Author Karën Fort is Associate Professor at University Paris-Sorbonne (Paris 4) working on the STIH (meaning, text, computer science, history) team. Her current research interests include collaborative manual annotation, crowdsourcing and ethics.
650 0 _aNatural language processing (Computer science)
_0http://id.loc.gov/authorities/subjects/sh88002425.
655 4 _aElectronic books.
830 0 _aFocus series (London, England)
_0http://id.loc.gov/authorities/names/n2014186952.
856 4 0 _uhttps://onlinelibrary.wiley.com/doi/book/10.1002/9781119306696
_yFull text is available at Wiley Online Library Click here to view
942 _2ddc
_cER