Semantic Disambiguation exercise

Attached is the description from your textbook showing how to calculate precision and recall.

Group tasks

1. Calculate the precision and recall of the SERF tagger on these sentences.
2. Try rephrasing these sentences in a variety of ways and see if the tagger assigns the same argument structure.
3. Can you tell what each of the argument tags is used for (if not, google for 'propbank argument labels' (if not, see page 5 of the Propbank Tagging Instructions)?
4. Compare the role tags found for the verb 'move' in the third test sentence to the roles described in the Framenet project:
   1. Go to The Framenet web interface
   2. Choose motion-directional in the dropdown box, and hit 'view frame relations'
   3. On the next page, hit the 'view' button beside 'view frame reports'
   A framenet tagger in the Senseval-3 competition last year (see the Ahn et al paper in the Senseval proceedings) scored precision 86% and recall 73% on the task of assigning framenet roles to English text. Is this figure higher or lower than the performance you computed for SERF's labelling of the test sentencs?

Please email the answer to these two questions to Dr. Byron by Sunday night at 11:59.

1. What was the calculated precision and recall of the SERF tool on the test sentences.
2. What kind of measure would you use as a baseline number against which to compare this performance? (you don't have to calculate a baseline number, just describe the formula)