深度学习：使用 word2vec 和 gensim

通过gensim来实现深度学习算法。

Word2vec: the good, the bad (and the fast)

Google发布了一些无监督深度学习算法，PDF：“Our model can answer the query “give me a word like king, like woman, but unlike man” with “queen“.

Not only do these algorithms boast great performance, accuracy and a theoretically-not-so-well-founded-but-pragmatically-superior-model (all three solid plusses in my book), but they were also devised by my fellow country and county-man, Tomáš Mikolov from Brno! The googlers have also released an open source implementation of these algorithms.

Although, in words of word2vec’s authors, the toolkit is meant for “research purposes”, it’s actually optimized C, down to cache alignments, memory look-up tables, static memory allocations and a penchant for single letter variable names. Somebody obviously spent time profiling this, which is good news for people running it, and bad news for people wanting to understand it, extend it or integrate it (as researchers are wont to do).

In short, the spirit of word2vec fits gensim’s tagline of topic modelling for humans, but the actual code doesn’t, tight and beautiful as it is. I therefore decided to reimplement word2vec in gensim, starting with the hierarchical softmax skip-gram model, because that’s the one with the best reported accuracy. I reimplemented it from scratch, de-obfuscating word2vec into a less menial state. No need for a custom implementation of hashing, lists, dicts, random number generators… all of these come built-in with Python.

Free, fast, pretty — pick any two. As the ratio of clever code to comments shrank and shrank (down to ~100 Python lines, with 40% of them comments), so did the performance. About 1000x. Yuck. I rewrote the explicit Python loops in NumPy, speeding things up ~50x (yay), but that means it’s still ~20x slower than the original (ouch). I could optimize it further, using Cython and whatnot, but that would lead back to obfuscation, beating the purpose of this exercise.

So, what can it do?

Distributional semantics goodness; see here and the original article for more background. Basically, the algorithm takes some unstructured text and learns “features” about each word. The neat thing is (apart from it learning the features completely automatically, without any human input/supervision!) that these features capture different relationships — both semantic and syntactic. This allows some (very basic) algebraic operations, like the above mentioned “king-man+woman=queen“. More concretely:

>>> # import modules and set up logging  >>> from gensim.models import word2vec  >>> import logging  >>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)  >>> # load up unzipped corpus from http://mattmahoney.net/dc/text8.zip  >>> sentences = word2vec.Text8Corpus('/tmp/text8')  >>> # train the skip-gram model; default window=5  >>> model = word2vec.Word2Vec(sentences, size=200)  >>> # ... and some hours later... just as advertised...  >>> model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)  [('queen', 0.5359965)]     >>> # pickle the entire model to disk, so we can load&resume training later  >>> model.save('/tmp/text8.model')  >>> # store the learned weights, in a format the original C tool understands  >>> model.save_word2vec_format('/tmp/text8.model.bin', binary=True)  >>> # or, import word weights created by the (faster) C word2vec  >>> # this way, you can switch between the C/Python toolkits easily  >>> model = word2vec.Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True)     >>> # "boy" is to "father" as "girl" is to ...?  >>> model.most_similar(['girl', 'father'], ['boy'], topn=3)  [('mother', 0.61849487), ('wife', 0.57972813), ('daughter', 0.56296098)]  >>> more_examples = ["he his she", "big bigger bad", "going went being"]  >>> for example in more_examples:  ...     a, b, x = example.split()  ...     predicted = model.most_similar([x, b], [a])[0][0]  ...     print "'%s' is to '%s' as '%s' is to '%s'" % (a, b, x, predicted)  'he' is to 'his' as 'she' is to 'her'  'big' is to 'bigger' as 'bad' is to 'worse'  'going' is to 'went' as 'being' is to 'was'     >>> # which word doesn't go with the others?  >>> model.doesnt_match("breakfast cereal dinner lunch".split())  'cereal'