Churnalism - Revision history

199.166.207.144 at 14:43, 18 October 2010

2010-10-18T14:43:13Z

← Older revision		Revision as of 14:43, 18 October 2010
Line 1:		Line 1:
	Edited version of http://p2pu.org/comment/reply/10207/4405		Edited version of http://p2pu.org/comment/reply/10207/4405

	~~First off I should say I didn't really mean denormalization, more like~~ unnormalized. Store the data as convenient blob of fields, which means there's duplication but querying is easier. So instead of having to create a join on a key and doing complex queries because your design is so efficient each item is only represented once, you're querying on a list of items, maybe using 'distinct' to narrow down as required. Basically treating an RDBMS as a big spreadsheet, duplicating each combination, but for a purpose rather than generalized system it works. The table "visits" could be		= Processing unnormalized data =
			Store the data as convenient blob of fields, which means there's duplication but querying is easier. So instead of having to create a join on a key and doing complex queries because your design is so efficient each item is only represented once, you're querying on a list of items, maybe using 'distinct' to narrow down as required. Basically treating an RDBMS as a big spreadsheet, duplicating each combination, but for a purpose rather than generalized system it works. The table "visits" could be

	<table>		<table>
Line 25:		Line 26:

	Of course this is simplified but I hope it makes sense.		Of course this is simplified but I hope it makes sense.

			= Using a semantic wiki to interact with data =

	For the canbudget project, I used semantic mediawiki. It couldn't easily handle tens of millions of entries, but for hundreds of thousands of entries (a narrowed down list?) can be useful. I entered the items in a spreadsheet, but any data source can be used. Once you have a data source, it's imported and can be viewed, queried and updated.		For the canbudget project, I used semantic mediawiki. It couldn't easily handle tens of millions of entries, but for hundreds of thousands of entries (a narrowed down list?) can be useful. I entered the items in a spreadsheet, but any data source can be used. Once you have a data source, it's imported and can be viewed, queried and updated.
Line 61:		Line 64:
	Mediawiki essentially uses the page name as the key. It has a great system of redirects (AKAs); John Smith and John R. Smith can redirect to Jonathan Smith so they're equivalent (you'd probably want an additional qualifier, like John Smith (Tobacco lobbyist of Smallville, USA) but you're probably not going to out-condition Wikipedia's conventions).		Mediawiki essentially uses the page name as the key. It has a great system of redirects (AKAs); John Smith and John R. Smith can redirect to Jonathan Smith so they're equivalent (you'd probably want an additional qualifier, like John Smith (Tobacco lobbyist of Smallville, USA) but you're probably not going to out-condition Wikipedia's conventions).

	You can add inferences to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together."		You can add inferences on an ongoing basis to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together." These inferences will apply to existing data, since they're separate from the data values can be applied freely.

	Annotations (properties) can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows," as is "sibling of," "received funding from," etc. Then you can do broad to narrow queries on "probably knows" or any other annotations. Since the annotations are created, the queries can be quite simple.		Annotations (properties) can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows," as is "sibling of," "received funding from," etc. Then you can do broad to narrow queries on "probably knows" or any other annotations. Since the annotations are created, the queries can be quite simple.
Line 72:		Line 75:

	We're starting to see ways to use open linked data (LOD), so your data can include data from sites like Freebase, and "official" sources. Here's a sample list today.http://esw.w3.org/SparqlEndpoints http://www.data.gov/ might be of particular interest to [http://en.wikipedia.org/wiki/Usonia Usonians]. With LOD the "back end" system doesn't matter as long as it conforms to the data/query standard, so the data sets can be very large and distributed.		We're starting to see ways to use open linked data (LOD), so your data can include data from sites like Freebase, and "official" sources. Here's a sample list today.http://esw.w3.org/SparqlEndpoints http://www.data.gov/ might be of particular interest to [http://en.wikipedia.org/wiki/Usonia Usonians]. With LOD the "back end" system doesn't matter as long as it conforms to the data/query standard, so the data sets can be very large and distributed.

	~~Enjoy! (but stay away from the ciggies)~~

	[[Category:Open Journalism & the Open Web]]		[[Category:Open Journalism & the Open Web]]

David Mason at 03:55, 15 October 2010

2010-10-15T03:55:26Z

← Older revision		Revision as of 03:55, 15 October 2010
Line 15:		Line 15:
	# for each $org, $dept in "select organization, department visited" from visits		# for each $org, $dept in "select organization, department visited" from visits
	## for each $clientInterest in "select clientInterest from orgclients where org = $org"		## for each $clientInterest in "select clientInterest from orgclients where org = $org"
	## "select true from govDept where dept = $dept and deptInterest is not $clientInterest"		### "select true from govDept where dept = $dept and deptInterest is not $clientInterest"
	### if not true, print "What the heck is $org meeting with $dept about $clientInterest for?"		### if not true, print "What the heck is $org meeting with $dept about $clientInterest for?"

David Mason at 03:53, 15 October 2010

2010-10-15T03:53:14Z

← Older revision		Revision as of 03:53, 15 October 2010
Line 14:		Line 14:

	# for each $org, $dept in "select organization, department visited" from visits		# for each $org, $dept in "select organization, department visited" from visits
	## for each $clientInterest in "select clientInterest from orgclients where org = $org" do		## for each $clientInterest in "select clientInterest from orgclients where org = $org"
	## "select true from govDept where dept = $dept an deptInterest is not $clientInterest"		## "select true from govDept where dept = $dept and deptInterest is not $clientInterest"
	### if not true, print "What the heck is $org meeting with $dept about $clientInterest for?"		### if not true, print "What the heck is $org meeting with $dept about $clientInterest for?"

199.166.207.144 at 03:50, 15 October 2010

2010-10-15T03:50:59Z

← Older revision		Revision as of 03:50, 15 October 2010
Line 74:		Line 74:

	Enjoy! (but stay away from the ciggies)		Enjoy! (but stay away from the ciggies)

			[[Category:Open Journalism & the Open Web]]

199.166.207.144: seseme seeds in the keyboard

2010-10-15T03:49:34Z

seseme seeds in the keyboard

← Older revision		Revision as of 03:49, 15 October 2010
Line 71:		Line 71:
	Because it's a wiki with a front end, you can open it up to a group of people to query or edit with forms. It's common Free Software so you don't have to "invent" or maintain anything, though you can extend it as needed.		Because it's a wiki with a front end, you can open it up to a group of people to query or edit with forms. It's common Free Software so you don't have to "invent" or maintain anything, though you can extend it as needed.

	We're starting to see ways to use open linked data (LOD), so your data can include data from sites like Freebase, and "official" sources. Here's a sample list today.http://esw.w3.org/SparqlEndpoints http://www.data.gov/ might be of particular interest to [http://en.wikipedia.org/wiki/Usonia Usonians]. With LOD the "back end" system doesn't matter as long as it conforms to the data/query standard, so the data sets can be very large an distributed.		We're starting to see ways to use open linked data (LOD), so your data can include data from sites like Freebase, and "official" sources. Here's a sample list today.http://esw.w3.org/SparqlEndpoints http://www.data.gov/ might be of particular interest to [http://en.wikipedia.org/wiki/Usonia Usonians]. With LOD the "back end" system doesn't matter as long as it conforms to the data/query standard, so the data sets can be very large and distributed.

	Enjoy! (but stay away from the ciggies)		Enjoy! (but stay away from the ciggies)

199.166.207.144 at 03:48, 15 October 2010

2010-10-15T03:48:11Z

← Older revision		Revision as of 03:48, 15 October 2010
Line 59:		Line 59:
	[[GTAA]]		[[GTAA]]

	Mediawiki has a great system of redirects; John Smith an John R. Smith can redirect to Jonathan Smith so they're equivalent (you'd probably want an additional qualifier, but you're probably not going to out-condition Wikipedia's conventions).		Mediawiki essentially uses the page name as the key. It has a great system of redirects (AKAs); John Smith and John R. Smith can redirect to Jonathan Smith so they're equivalent (you'd probably want an additional qualifier, like John Smith (Tobacco lobbyist of Smallville, USA) but you're probably not going to out-condition Wikipedia's conventions).

	You can add inferences to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together."		You can add inferences to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together."

199.166.207.144 at 03:45, 15 October 2010

2010-10-15T03:45:46Z

← Older revision		Revision as of 03:45, 15 October 2010
Line 65:		Line 65:
	Annotations (properties) can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows," as is "sibling of," "received funding from," etc. Then you can do broad to narrow queries on "probably knows" or any other annotations. Since the annotations are created, the queries can be quite simple.		Annotations (properties) can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows," as is "sibling of," "received funding from," etc. Then you can do broad to narrow queries on "probably knows" or any other annotations. Since the annotations are created, the queries can be quite simple.

	{{ #ask: [[probably knows::John Smith]] }}		<nowiki>{{ #ask: [[probably knows::John Smith]] }}</nowiki>

	Categories and annotations can be queried and combined. Since it's a semantic system, marked text has meaning and can be reused and constantly built on.		Categories and annotations can be queried and combined. Since it's a semantic system, marked text has meaning and can be reused and constantly built on.

199.166.207.144 at 03:44, 15 October 2010

2010-10-15T03:44:55Z

← Older revision		Revision as of 03:44, 15 October 2010
Line 58:		Line 58:

	[[GTAA]]		[[GTAA]]

			Mediawiki has a great system of redirects; John Smith an John R. Smith can redirect to Jonathan Smith so they're equivalent (you'd probably want an additional qualifier, but you're probably not going to out-condition Wikipedia's conventions).

	You can add inferences to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together."		You can add inferences to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together."

	Annotations (properties) can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows," as is "sibling of," "received funding from," etc. Then you can do broad to narrow queries on "probably knows" or any other annotations.		Annotations (properties) can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows," as is "sibling of," "received funding from," etc. Then you can do broad to narrow queries on "probably knows" or any other annotations. Since the annotations are created, the queries can be quite simple.

			{{ #ask: [[probably knows::John Smith]] }}

	Categories and annotations can be queried and combined. Since it's a semantic system, marked text has meaning and can be reused and constantly built on.		Categories and annotations can be queried and combined. Since it's a semantic system, marked text has meaning and can be reused and constantly built on.

199.166.207.144 at 03:40, 15 October 2010

2010-10-15T03:40:18Z

← Older revision		Revision as of 03:40, 15 October 2010
Line 61:		Line 61:
	You can add inferences to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together."		You can add inferences to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together."

	Annotations (properties) can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows." Then you can do broad to narrow queries on "probably knows" or any other annotations.		Annotations (properties) can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows," as is "sibling of," "received funding from," etc. Then you can do broad to narrow queries on "probably knows" or any other annotations.

	Categories and annotations can be queried and combined. Since it's a semantic system, marked text has meaning and can be reused and constantly built on.		Categories and annotations can be queried and combined. Since it's a semantic system, marked text has meaning and can be reused and constantly built on.

199.166.207.144 at 03:37, 15 October 2010

2010-10-15T03:37:17Z

← Older revision		Revision as of 03:37, 15 October 2010
Line 61:		Line 61:
	You can add inferences to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together."		You can add inferences to a template, for example, if the address within 40 miles of a Ciggies Inc, add it to the category "Close to ciggies inc," or check for overlaps in "went to school at ..." to generate "went to school together."

	Annotations can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows." Then you can do broad to narrow queries on "probably knows" or any other annotations.		Annotations (properties) can be hierarchical (actually [http://en.wikipedia.org/wiki/Heterarchy hetrarchical]); "went to school together" could be a subproperty of "probably knows." Then you can do broad to narrow queries on "probably knows" or any other annotations.

	Categories an annotations can be queried and combined. Since it's a semantic system, marked text has meaning and can be reused and constantly built on.		Categories and annotations can be queried and combined. Since it's a semantic system, marked text has meaning and can be reused and constantly built on.

	Because it's a wiki with a front end, you can open it up to a group of people to query or edit with forms. It's common Free Software so you don't have to "invent" or maintain anything, though you can extend it as needed.		Because it's a wiki with a front end, you can open it up to a group of people to query or edit with forms. It's common Free Software so you don't have to "invent" or maintain anything, though you can extend it as needed.