Why word embeddings learned from word2vec are linearly correlated

[zyxue]

Hi @WesleyyC , I conducted some experiments myself, and see some funny result as described here,

https://stats.stackexchange.com/questions/337083/why-word-embeddings-learned-from-word2vec-are-linearly-correlated

I wonder if you have experienced similar results or have any comment in general?

[WesleyyC]

I don't really have a good explanation, but have your tried different randomization method for your initialization？

Essentially the first position and the second position are encoding the same information (since points are linearly correlated). Therefore, the problem comes down to why the embedding just use one single position when it has two position. I suspect that's because in both cases they are genearting really bad result and the two position is initialized in a similar way (let's say all small values around 0) then the gradient might not help to differentiate the two positions.

In addition, to my knowledge, no one has tried to investigate how wod2vec works using highly compressed feature vector, so that might be something interesting to look at.