Enriched Afghanistan War Diary Topic Map - Revision history

Akivela at 13:15, 26 July 2012

2012-07-26T13:15:53Z

Akivela at 13:15, 26 July 2012

2012-07-26T13:15:25Z

Akivela at 13:03, 12 July 2011

2011-07-12T13:03:49Z

Akivela at 12:32, 12 July 2011

2011-07-12T12:32:00Z

Akivela at 12:30, 12 July 2011

2011-07-12T12:30:48Z

Akivela at 12:29, 12 July 2011

2011-07-12T12:29:50Z

Akivela at 12:28, 12 July 2011

2011-07-12T12:28:07Z

Akivela at 12:27, 12 July 2011

2011-07-12T12:27:45Z

Akivela at 12:27, 12 July 2011

2011-07-12T12:27:15Z

New page

This page is an extended version of blog writing at [http://www.wandora.net/wandora/forum/viewtopic.php?t=56 Wandora Forum].

Thomas Efer of [http://www.topicmapslab.de Topic Maps Labs] released a topic map named ''Afghanistan War Diary - 2004''. Topic map was based on Wikileaked documents and contains 2000 war reports from Afghanistan war. Reports are stored in topic map as unstructured text. The topic map is also available at Topic Maps Labs' topic map repository, [http://maiana.topicmapslab.de/u/efi/tm/wd2004 Maiana]. Lutz Maicher announced the release of the AWD topic map at TopicMapMail email list. The announcement evoked an interesting discussion related to the quality of the topic map. The discussion inspired me trying to enhance the AWD topic map using Wandora's extractors. Wandora has several extractors that distill entities and keywords out of unstructured text. I selected extractors

* [[AlchemyAPI extractors|Alchemy Entity extractor]]
* [[AlchemyAPI extractors|Alchemy Keywords extractor]]
* [[OpenCalais classifier|OpenCalais extractor]]
* Yahoo! term extractor

And applied each extractor, one by one, to all summary occurrences of report topics. Summary occurrence contains textual representation of the military event the report talks about.

Applying the extractor to occurrences generated more topics and associations. Extractors' outputs were stored to separate topic maps. As a result I got 4 different information packages, one for each extractor. All topic maps were packed into a zip. This zip package is available [http://www.wandora.org/wandora/download/other/AWD2004_experiment.zip here]. In practice anyone can merge any of the generated topic maps with the original ''Afghanistan War Diary - 2004'' topic map and see extracted topics interleaving with the original topics. As the original summary occurrence is available, anyone can evaluate if the extracted topics and associations really describe the report.

To summarize results:
* Yahoo term extractor found 3563 term topics (15092 associations).
* OpenCalais found 1749 tags (3455 associations) in 18 tag classes and 25 topic categories (1317 associations).
* Alchemy entity extractor found 424 entities (1285 associations).
* Alchemy keyword extractor found 9054 keywords (14684 associations).

Although the technical implementation of the experiment was easy, I am not really sure about the quality of automatic classifications provided by Calais, Alchemy and Yahoo. It is clear that the source material, reports of military actions, is very challenging due to military specific expressions, terms, and acronyms, and it looks like all classifiers have made false interpretations. It would be very interesting if someone would like to evaluate the overall quality of generated topic maps and point out typical error classes.

In any case, I hope this demonstration clearly shows a simple topic map storing merely occurrence data is not necessarily a dead end but a good start.

@@ Line 10: / Line 10: @@
 And applied each extractor, one by one, to all summary occurrences of report topics. Summary occurrence contains textual representation of the military event the report talks about.
-Applying the extractor to occurrences generated more topics and associations. Extractors' outputs were stored to separate topic maps. As a result I got 4 different information packages, one for each extractor. All resulting topic maps and the original topic map were zipped and this zip package is available [http://www.wandora.org/download/other/AWD2004_experiment.zip here]. In practice anyone can merge any of the generated topic maps with the original ''Afghanistan War Diary - 2004'' topic map and see extracted topics interleaving with the original topics. As the original summary occurrence is available, anyone can evaluate if the extracted topics and associations really describe the report. To summarize results:
+Applying the extractor to occurrences generated more topics and associations. Extractors' outputs were stored to separate topic maps. As a result I got 4 different information packages, one for each extractor. All resulting topic maps and the original topic map were zipped and this zip package is available '''[http://www.wandora.org/download/other/AWD2004_experiment.zip here]'''. In practice anyone can merge any of the generated topic maps with the original ''Afghanistan War Diary - 2004'' topic map and see extracted topics interleaving with the original topics. As the original summary occurrence is available, anyone can evaluate if the extracted topics and associations really describe the report. To summarize results:
 * Yahoo term extractor found 3563 term topics (15092 associations).

@@ Line 10: / Line 10: @@
 And applied each extractor, one by one, to all summary occurrences of report topics. Summary occurrence contains textual representation of the military event the report talks about.
-Applying the extractor to occurrences generated more topics and associations. Extractors' outputs were stored to separate topic maps. As a result I got 4 different information packages, one for each extractor. All resulting topic maps and the original topic map were zipped and this zip package is available [http://www.wandora.org/wandora/download/other/AWD2004_experiment.zip here]. In practice anyone can merge any of the generated topic maps with the original ''Afghanistan War Diary - 2004'' topic map and see extracted topics interleaving with the original topics. As the original summary occurrence is available, anyone can evaluate if the extracted topics and associations really describe the report. To summarize results:
+Applying the extractor to occurrences generated more topics and associations. Extractors' outputs were stored to separate topic maps. As a result I got 4 different information packages, one for each extractor. All resulting topic maps and the original topic map were zipped and this zip package is available [http://www.wandora.org/download/other/AWD2004_experiment.zip here]. In practice anyone can merge any of the generated topic maps with the original ''Afghanistan War Diary - 2004'' topic map and see extracted topics interleaving with the original topics. As the original summary occurrence is available, anyone can evaluate if the extracted topics and associations really describe the report. To summarize results:
 * Yahoo term extractor found 3563 term topics (15092 associations).

@@ Line 1: / Line 1: @@
 This page is a duplicate of blog writing at [http://www.wandora.net/wandora/forum/viewtopic.php?t=56 Wandora Forum].
-Thomas Efer of [http://www.topicmapslab.de Topic Maps Labs] has released a topic map named ''Afghanistan War Diary - 2004''. Topic map was based on Wikileaked documents and contains 2000 war reports from Afghanistan war. Reports are stored in topic map as unstructured text. The topic map is also available at Topic Maps Labs' topic map repository, [http://maiana.topicmapslab.de/u/efi/tm/wd2004 Maiana]. Lutz Maicher [http://www.infoloom.com/pipermail/topicmapmail/2010q4/008539.html announced the release of the AWD topic map] at TopicMapMail email list. The announcement evoked an interesting discussion related to the quality of the topic map. The discussion inspired me trying to enhance the AWD topic map using Wandora's extractors. Wandora has several extractors that distill entities and keywords out of unstructured text. I selected extractors
+Thomas Efer of [http://www.topicmapslab.de Topic Maps Labs] has released a topic map named ''Afghanistan War Diary - 2004''. Topic map is based on Wikileaked documents and contains 2000 war reports from Afghanistan war. Reports are stored in topic map as unstructured text. The topic map is available at Topic Maps Labs' topic map repository, [http://maiana.topicmapslab.de/u/efi/tm/wd2004 Maiana]. Lutz Maicher [http://www.infoloom.com/pipermail/topicmapmail/2010q4/008539.html announced the release of the AWD topic map] at TopicMapMail email list. The announcement evoked an interesting discussion related to the quality of the topic map. The discussion inspired me trying to enhance the AWD topic map using Wandora's extractors. Wandora has several extractors that distill entities and keywords out of unstructured text. I selected extractors
 * [[AlchemyAPI extractors|Alchemy Entity extractor]]

@@ Line 1: / Line 1: @@
 This page is a duplicate of blog writing at [http://www.wandora.net/wandora/forum/viewtopic.php?t=56 Wandora Forum].
-Thomas Efer of [http://www.topicmapslab.de Topic Maps Labs] released a topic map named ''Afghanistan War Diary - 2004''. Topic map was based on Wikileaked documents and contains 2000 war reports from Afghanistan war. Reports are stored in topic map as unstructured text. The topic map is also available at Topic Maps Labs' topic map repository, [http://maiana.topicmapslab.de/u/efi/tm/wd2004 Maiana]. Lutz Maicher announced the release of the AWD topic map at TopicMapMail email list. The announcement evoked an interesting discussion related to the quality of the topic map. The discussion inspired me trying to enhance the AWD topic map using Wandora's extractors. Wandora has several extractors that distill entities and keywords out of unstructured text. I selected extractors
+Thomas Efer of [http://www.topicmapslab.de Topic Maps Labs] has released a topic map named ''Afghanistan War Diary - 2004''. Topic map was based on Wikileaked documents and contains 2000 war reports from Afghanistan war. Reports are stored in topic map as unstructured text. The topic map is also available at Topic Maps Labs' topic map repository, [http://maiana.topicmapslab.de/u/efi/tm/wd2004 Maiana]. Lutz Maicher [http://www.infoloom.com/pipermail/topicmapmail/2010q4/008539.html announced the release of the AWD topic map] at TopicMapMail email list. The announcement evoked an interesting discussion related to the quality of the topic map. The discussion inspired me trying to enhance the AWD topic map using Wandora's extractors. Wandora has several extractors that distill entities and keywords out of unstructured text. I selected extractors
 * [[AlchemyAPI extractors|Alchemy Entity extractor]]

← Older revision		Revision as of 12:27, 12 July 2011
Line 1:		Line 1:
−	This page is ~~an extended version~~ of blog writing at [http://www.wandora.net/wandora/forum/viewtopic.php?t=56 Wandora Forum].	+	This page is a duplicate of blog writing at [http://www.wandora.net/wandora/forum/viewtopic.php?t=56 Wandora Forum].

	Thomas Efer of [http://www.topicmapslab.de Topic Maps Labs] released a topic map named ''Afghanistan War Diary - 2004''. Topic map was based on Wikileaked documents and contains 2000 war reports from Afghanistan war. Reports are stored in topic map as unstructured text. The topic map is also available at Topic Maps Labs' topic map repository, [http://maiana.topicmapslab.de/u/efi/tm/wd2004 Maiana]. Lutz Maicher announced the release of the AWD topic map at TopicMapMail email list. The announcement evoked an interesting discussion related to the quality of the topic map. The discussion inspired me trying to enhance the AWD topic map using Wandora's extractors. Wandora has several extractors that distill entities and keywords out of unstructured text. I selected extractors		Thomas Efer of [http://www.topicmapslab.de Topic Maps Labs] released a topic map named ''Afghanistan War Diary - 2004''. Topic map was based on Wikileaked documents and contains 2000 war reports from Afghanistan war. Reports are stored in topic map as unstructured text. The topic map is also available at Topic Maps Labs' topic map repository, [http://maiana.topicmapslab.de/u/efi/tm/wd2004 Maiana]. Lutz Maicher announced the release of the AWD topic map at TopicMapMail email list. The announcement evoked an interesting discussion related to the quality of the topic map. The discussion inspired me trying to enhance the AWD topic map using Wandora's extractors. Wandora has several extractors that distill entities and keywords out of unstructured text. I selected extractors

@@ Line 6: / Line 6: @@
 * [[AlchemyAPI extractors|Alchemy Keywords extractor]]
 * [[OpenCalais classifier|OpenCalais extractor]]
-* Yahoo! term extractor
+* [[Yahoo! YQL term extractor|Yahoo! term extractor]]
 And applied each extractor, one by one, to all summary occurrences of report topics. Summary occurrence contains textual representation of the military event the report talks about.