I just completed a good scrub updating the SourceForge.net project websites for JWords and AutoSummary. I removed most of the unused features, uploaded & organized documentation, and configured the SF.net web hosting service. No new file releases are available (or pending), but the SourceForge.net page and documentation should make JWords & AutoSummary much more accessible and easier to use.
Javadoc source code documentation was uploaded onto SourceForge for both of the projects. Additionally, I created step-by-step instructions for getting each program up and running, which should be easy to follow for even the non-programmer. And finally, I posted more extensive project descriptions to the respective documentation pages.
The updated SourceForge.net project websites can be viewed at:
JWords Java Interface for WordNet
AutoSummary Semantic Analysis Engine
My SourceForge.net developer profile can be found at:
http://sourceforge.net/users/greenbacker/
Showing posts with label autosummary. Show all posts
Showing posts with label autosummary. Show all posts
Sunday, April 15, 2007
Friday, March 2, 2007
Commercially Available Automatic Summarization Software
In his weekly article, Robert X. Cringley profiles a company that has created software which is seemingly able to create relevant summaries of arbitrary size from bodies of text covering all possible subject domains. This is precisely the sort of thing I wanted to accomplish with my AutoSummary project. Automatic summarization is yet another application of NLP as a solution to the problem of too much information for humans to deal with. Learn more about iReader at Syntactica.
Monday, February 13, 2006
Frustrating Recursion
I thought I made a mistake once, but I was wrong. For some reason, this happens to me all the time when programming.
I was working on AutoSummary this weekend, adding a contextual framework using hypernym (superordinate) information for individual senses of a given word. In order to do this I needed to create a b-tree data structure. I thought I had set everything up properly, except the tree wouldn't populate past the second level. Weird crashes, etc. I tried everything, checked all of the functions and methods, testing everything I could think of. Nothing.
Tonight, I started checking everything over again, trying some different approaches. I did some digging and determined something was generating a null pointer exception. I checked everything all over again, and again... nothing. After a period of insufferable aggrivation, I discovered by trial and error that the exception was caused by the fact that I had forgotten to initialize the data container (an ArrayList). I was so worried about getting all the "hard stuff" figured out that I had overlooked a beginner's error.
The moral of the story? The simplest answers are often the hardest to find.
I was working on AutoSummary this weekend, adding a contextual framework using hypernym (superordinate) information for individual senses of a given word. In order to do this I needed to create a b-tree data structure. I thought I had set everything up properly, except the tree wouldn't populate past the second level. Weird crashes, etc. I tried everything, checked all of the functions and methods, testing everything I could think of. Nothing.
Tonight, I started checking everything over again, trying some different approaches. I did some digging and determined something was generating a null pointer exception. I checked everything all over again, and again... nothing. After a period of insufferable aggrivation, I discovered by trial and error that the exception was caused by the fact that I had forgotten to initialize the data container (an ArrayList). I was so worried about getting all the "hard stuff" figured out that I had overlooked a beginner's error.
The moral of the story? The simplest answers are often the hardest to find.
Sunday, January 29, 2006
Moving Forward
Last night I did some work on the Answer Machine, and today I'm working on adding topic detection to AutoSummary.
Check out the entry in the Answer Machine project log for details about its new functionality. As for AutoSummary, I came up with a good idea about how to implement topic detection within the current framework of the program. I was checking out this post at the Search Science blog that I read, and thought "I can do that."
What I plan on doing is after determining the likely sense of a given word, I'll build a list of all of the possible topics that word is connected to (using the WordNet domain function). From that I'll be able to find the topic of a sentence by taking the best intersection of all the anchor words (not "the" or "and" etc).
You can probably see how it will scale from there, building intersections of sentences into paragraphs, and paragraphs into entire texts. So very shortly I just might be able to take an fully body of text and determine just what the heck it is all about. Could be an interesting step forward in search relevance and data mining...
Check out the entry in the Answer Machine project log for details about its new functionality. As for AutoSummary, I came up with a good idea about how to implement topic detection within the current framework of the program. I was checking out this post at the Search Science blog that I read, and thought "I can do that."
What I plan on doing is after determining the likely sense of a given word, I'll build a list of all of the possible topics that word is connected to (using the WordNet domain function). From that I'll be able to find the topic of a sentence by taking the best intersection of all the anchor words (not "the" or "and" etc).
You can probably see how it will scale from there, building intersections of sentences into paragraphs, and paragraphs into entire texts. So very shortly I just might be able to take an fully body of text and determine just what the heck it is all about. Could be an interesting step forward in search relevance and data mining...
Wednesday, August 17, 2005
AutoSummary Alpha Release
It pleases me to announce that I have just released the first alpha testing version of the AutoSummary Semantic Analysis Engine under the BSD License.
When completed, AutoSummary will generate contextually-relevant summaries of plain text documents using various statistical and rule-based methods of Natural Language Processing. First, the part-of-speech and specific word-sense (meaning) is determined for each word. Next, each sentence is deconstructed and the subject/predicate/object is identified. A map of relationships between word is then created. From this, specific themes are identified and memes (general ideas) are generated. Finally, the memes are used to create summaries of the original document of varying length and detail.
Currently, only the part-of-speech tagger has been implemented. A JPhrase object is created which contains semantic information about a given phrase of words, including part-of-speech scores and possible word-sense combinations. The part-of-speech tagging method takes the JPhrase and returns a marked-up string with each word in the phrase associated with a tag corresponding to the part-of-speech (noun, verb, adjective, adverb) determined to be most likely used for this particular instance. To make this determination, usage statistics from the WordNet semantic concordance texts are used.
AutoSummary uses the WordNet lexical reference system via the JWords Java Interface for WordNet as its source of lexigraphical information. In order to run AutoSummary, you must first install WordNet 2.0 and edit the JWords configuration file.
This alpha test release is EXTREMELY limited. Support for articles (a, an, the, etc), plurals, tenses, pronouns, and the verb 'to be' has not yet been implemented. If these types of words are entered into the current version, the program will likely crash. The simple demo included in the release is merely intended to show how the part-of-speech tagging system will determine the part-of-speech for each word in a given phrase.
I've been pushing pretty hard over the last week or so to get this initial release out. Once I take a must deserved rest, I will get a publicly available prototype up and running for my wildcard-based general question answering script, go back and add WordNet 2.1 support to JWords, and also take a serious look at setting up a web version of the AutoSummary demo program.
AutoSummary is available for download at SourceForge.net http://sourceforge.net/projects/autosummary
Javadoc documentation for AutoSummary is available here.
When completed, AutoSummary will generate contextually-relevant summaries of plain text documents using various statistical and rule-based methods of Natural Language Processing. First, the part-of-speech and specific word-sense (meaning) is determined for each word. Next, each sentence is deconstructed and the subject/predicate/object is identified. A map of relationships between word is then created. From this, specific themes are identified and memes (general ideas) are generated. Finally, the memes are used to create summaries of the original document of varying length and detail.
Currently, only the part-of-speech tagger has been implemented. A JPhrase object is created which contains semantic information about a given phrase of words, including part-of-speech scores and possible word-sense combinations. The part-of-speech tagging method takes the JPhrase and returns a marked-up string with each word in the phrase associated with a tag corresponding to the part-of-speech (noun, verb, adjective, adverb) determined to be most likely used for this particular instance. To make this determination, usage statistics from the WordNet semantic concordance texts are used.
AutoSummary uses the WordNet lexical reference system via the JWords Java Interface for WordNet as its source of lexigraphical information. In order to run AutoSummary, you must first install WordNet 2.0 and edit the JWords configuration file.
This alpha test release is EXTREMELY limited. Support for articles (a, an, the, etc), plurals, tenses, pronouns, and the verb 'to be' has not yet been implemented. If these types of words are entered into the current version, the program will likely crash. The simple demo included in the release is merely intended to show how the part-of-speech tagging system will determine the part-of-speech for each word in a given phrase.
I've been pushing pretty hard over the last week or so to get this initial release out. Once I take a must deserved rest, I will get a publicly available prototype up and running for my wildcard-based general question answering script, go back and add WordNet 2.1 support to JWords, and also take a serious look at setting up a web version of the AutoSummary demo program.
AutoSummary is available for download at SourceForge.net http://sourceforge.net/projects/autosummary
Javadoc documentation for AutoSummary is available here.
Monday, August 8, 2005
Have you ever seen the sun rise ... twice in a row ... three times...?
Paul Graham recently had a very interesting article about what businesses can learn from open source.
The entire piece was very well written, and the author makes a lot of good points, but what captured my imagination the most was the part about the optimal working environment for a programmer. Paul says that people working in the comfortable surroundings of their own home are often exponentially more productive than corporate rats in a stuffy cubicle farm. Based on my own personal experience, I would certainly agree. I find I do my best work in marathon sessions, often involving massive amounts of caffeine and highly irregular sleep patterns. I understand many programmers share similar experiences. Once I get 'in the zone,' as they say, I will work solidly for hours on end paying little attention to time or the world around me. Were I restricted to a structured work environment constantly struggling against interruptions and distractions like co-workers, meetings and email, I fear my programming would suffer.
I am currently working hard to finish up the next release of JWords, as well as the initial release of another semantic analysis tool. I've been trying to complete this stuff for weeks now, but I find the only way I can get anything done is if I rush straight home from work and devote the entire evening to programming. If I decide to go workout, play basketball, see a movie or go out to dinner, and try to pick up where I left off and program a little bit at a time, I never get anything done. By the time I sit down and get going, its time to sign off for the night. My old friend the all-nighter is no longer an option with my responsibilities at work. Usually weekends offer a real chance to get down to business, but I'm afraid my willpower is just barely too weak to resist the temptation of living it up at the beach. But on those rare occasions when I can devote a significant amount of time to this endeavor, and everything is firing on all cylinders, nothing else compares.
The entire piece was very well written, and the author makes a lot of good points, but what captured my imagination the most was the part about the optimal working environment for a programmer. Paul says that people working in the comfortable surroundings of their own home are often exponentially more productive than corporate rats in a stuffy cubicle farm. Based on my own personal experience, I would certainly agree. I find I do my best work in marathon sessions, often involving massive amounts of caffeine and highly irregular sleep patterns. I understand many programmers share similar experiences. Once I get 'in the zone,' as they say, I will work solidly for hours on end paying little attention to time or the world around me. Were I restricted to a structured work environment constantly struggling against interruptions and distractions like co-workers, meetings and email, I fear my programming would suffer.
I am currently working hard to finish up the next release of JWords, as well as the initial release of another semantic analysis tool. I've been trying to complete this stuff for weeks now, but I find the only way I can get anything done is if I rush straight home from work and devote the entire evening to programming. If I decide to go workout, play basketball, see a movie or go out to dinner, and try to pick up where I left off and program a little bit at a time, I never get anything done. By the time I sit down and get going, its time to sign off for the night. My old friend the all-nighter is no longer an option with my responsibilities at work. Usually weekends offer a real chance to get down to business, but I'm afraid my willpower is just barely too weak to resist the temptation of living it up at the beach. But on those rare occasions when I can devote a significant amount of time to this endeavor, and everything is firing on all cylinders, nothing else compares.
Subscribe to:
Posts (Atom)