Sunday, 29 June 2014
Sunday, 8 June 2014
Thursday, 5 June 2014
Stemming defined inside class in python
in the book chapter 3, following code is given for implementing stemming in class and functions.
1: >>> import nltk.stem
2: >>> english_stemmer = nltk.stem.SnowballStemmer('english')
3: >>> class StemmedCountVectorizer(CountVectorizer):
4: ... def build_analyzer(self):
5: ... analyzer = super(StemmedCountVectorizer, self).build_analyzer()
7: ... return lambda doc: (english_stemmer.stem(w) for w in analyzer(doc))
9: >>> vectorizer = StemmedCountVectorizer(min_df=1, stop_words='english')
First of all I want to understand, how class can be defined in the program.
How to define class?
http://freepythontips.wordpress.com/2013/08/07/the-self-variable-in-python-explained/
What is super function?
http://learnpythonthehardway.org/book/ex44.html
How to use super function?
http://orangeml.blogspot.in/2014/06/use-of-super-in-built-python-function.html
What is Lambda?
Lambda is used to create anonymous function which is used just once in the application. In the above example an anonymous function is executed only once when that class is executed.
probably to run an expression in the return statement, lambda function is used.
http://pythonconquerstheuniverse.wordpress.com/2011/08/29/lambda_tutorial/
How return statement executes?
Return statement in the above example creates "lambda doc" an anonymous function. This anonymous function calls two functions
1.
english_stemmer()
2.
analyzer()
analyzer function vectorizing the contents in the doc variable, for iterates each values of the analyzer() and then passed to the english_stemmer to stem.
Example for for loop.http://orangeml.blogspot.in/2014/07/using-for-loop-in-python-for-lists-as.html
https://wiki.python.org/moin/ForLoop
Further explanation
This particular example confused me initially and I could not understand. So I wrote to the author's official website. He has given me another example, which is very clear and easy to understand
Code
from sklearn.feature_extraction.text import CountVectorizercv = CountVectorizer()
analyzer = cv.build_analyzer()
import nltk.stem
english_stemmer = nltk.stem.SnowballStemmer('english')
doc = "A cook cooks delicious meals"
print([english_stemmer.stem(w) for w in analyzer(doc)])
print (analyzer(doc))
Output
[u'cook', u'cook', u'delici', u'meal'][u'cook', u'cooks', u'delicious', u'meals']
here is the link to my discussion with the author http://www.twotoreal.com/q/156/return-lambda-function-in-ntlks-stemmer-chapter-3-page-59
Wednesday, 14 May 2014
Counting words - Clustering–Find Related post–part2
Key point
Euclidean distance comes under the unsupervised learning.
Why Euclidean distance?
Euclidean distance measures the distance between the two points in xy plane. formula Euclidean distance formula and its explanation can be seen in the below link. the formula will match exactly with our dist_raw(v1, v2) function.
http://www.cut-the-knot.org/pythagoras/DistanceFormula.shtml
Aptana studio 3 – Debugger tool
Stepping through code
set the break point by double clicking left side of the code. break point icon will appear next to the line where breakpoint is set.now click the debug button, it will halt break point set line. from there we can step through code line by line and see result of the execution.
https://www.youtube.com/watch?v=7ROg6Wwz7Z0
Loading Sub package and avoid attribution error
from scipy import linalg sub-package may be individually imported otherwise we will get error during sub-packages calls inside the code.http://stackoverflow.com/questions/9819733/scipy-special-import-issue
Tuesday, 13 May 2014
Counting words - Clustering–Find Related post–scikit
Code
1: import os 2: import sys 3: import scipy as sp4: from sklearn.feature_extraction.text import CountVectorizer
5: data_Dir = "E:\\Machine Learning\\Orange\\Finding Related Post\\toy\\"6: posts = [open(os.path.join(data_Dir,f)).read() for f in os.listdir(data_Dir)]
7: from sklearn.feature_extraction.text import CountVectorizer
8: vectorizer = CountVectorizer(min_df=1) 9: print vectorizer10: X_train = vectorizer.fit_transform(posts)
11: print posts 12: print X_train13: num_samples, num_features = X_train.shape
14: print("#samples: %d, #features: %d" % (num_samples, num_features))
15: print(vectorizer.get_feature_names())
Result:
the below result is printed the parameter settings of the count vectoriser. we can see min_df=1 in the result, which is assigned by us in the line no 8. all other settings are default setting.
1: CountVectorizer(analyzer=word, binary=False, charset=utf-8,2: charset_error=strict, dtype=<type 'long'>, input=content,
3: lowercase=True, max_df=1.0, max_features=None, max_n=None,
4: min_df=1, min_n=None, ngram_range=(1, 1), preprocessor=None,5: stop_words=None, strip_accents=None, token_pattern=(?u)\b\w\w+\b,
6: tokenizer=None, vocabulary=None)
from the result printed by the line no 11, 12 & 15, we can understand the what fit_transform function executed
the result of fit_transform is
1: (0, 0) 1 2: (0, 24) 1 3: (0, 23) 1 4: (0, 8) 1 5: (0, 9) 1 6: (0, 3) 1 7: (0, 10) 1 8: (0, 1) 1 9: (0, 12) 1 10: (0, 14) 1 11: (0, 11) 1 12: (0, 22) 1 13: (0, 15) 1 14: (0, 17) 1 15: (1, 18) 1 16: (1, 20) 1 17: (1, 7) 1 18: (1, 2) 1 19: (1, 5) 1 20: (2, 19) 1 21: (2, 13) 1 22: (2, 16) 1 23: (2, 7) 1 24: (2, 5) 1 25: (2, 6) 1 26: (3, 4) 1 27: (3, 7) 1 28: (3, 21) 1 29: (3, 5) 1 30: (4, 4) 3 31: (4, 7) 3 32: (4, 21) 3 33: (4, 5) 3
(0,0) 1 what it actually means?
(Sample numeber, feature name) number of occurrences
first 0 gives the information of sample number that is posts file no 01.txt. second 0 is u’about’ see the result of line no 15 feature_names.
1: [u'about', u'actually', u'capabilities', u'contains', u'data', u'databases', u'images', u'imaging', u'interesting', u'is', u'it', u'learning', u'machine', u'most', u'much', u'not', u'permanently', u'post', u'provide', u'safe', u'storage', u'store', u'stuff', u'this', u'toy']1 means the word “about” occurred only ones in that sample.
for example the second one (0,24) 1
0 means the sample number, which is 01.txt
“This is a toy post about machine learning. Actually, it contains not much interesting stuff.”
24 means the feature name “this”
1 means the number of occurrences of the word “this”
Monday, 12 May 2014
Regex pattern in token_pattern for CountVectorizer
Regex
The result of the book showed
#samples: 5, #features: 25
[u'about', u'actually', u'capabilities', u'contains', u'data', u'databases', u'images', u'imaging', u'interesting', u'is', u'it', u'learning', u'machine', u'most', u'much', u'not', u'permanently', u'post', u'provide', u'safe', u'storage', u'store', u'stuff', u'this', u'toy']
There is one missing word in the above result that is “a”, which is there in the 01.txt of the toy posts.
01.txt This is a toy post about machine learning. Actually, it contains
not much interesting stuff.
Why it is missed?
This is caused by the default token_pattern for CountVectorizer, which removes tokens of a single character:
Default token_pattern for CountVectorizer is (?u)\b\w\w+\b, which removed single character word ‘a’.
How to get the single character word?
by changing we can see the single character in the result. token_pattern for CountVectorizer to \\b\\w+\\b
Do we need the single character word?
Depend on the apllication, we may need to extract single character words.
Ref
http://stackoverflow.com/questions/20717641/countvectorizer-i-not-showing-up-in-vectorized-text