Please add your comments on the last week here: Hidden structure learning, emergent categoricity, language-specific constraints.
Categories
Please add your comments on the last week here: Hidden structure learning, emergent categoricity, language-specific constraints.
3 replies on “Week 4 Discussion”
I wonder if there might be some interesting intellectual kinship between the “summed probability” interpretation of hidden structure presented here and the “blend” model of linguistic production ongoing with Goldrick and Smolensky. In their view the computation of a categorical, symbolic output proceeds through an intermediate area of “mixture” between competing forms. In some ways the idea of “mixture” is similar to that of addition, but it’s not clear that the insights of either model are readily adaptable to the other…
The Kevin Knight tutorials I mentioned are:
Kevin Knight (http://www.isi.edu/~knight/) wrote the most useful tutorial I’ve read yet on applying statistical methods to linguistics. It’s called Bayesian Inference with Tears. Although I enjoy the humor a great deal, you can safely skip the first seven sections and go directly to the eighth without loss of information.
http://www.isi.edu/natural-language/people/bayes-with-tears.pdf
He wrote an earlier one on Expectation Maximization called A Statistical MT Tutorial Workbook. I’ve not read it closely, but it does focus on computational linguistics, especially machine translation.
http://www.isi.edu/natural-language/mt/wkbk.rtf
It does not cover Maximum Entropy, although I might have said that it did in class (confusing EM with ME).
TL;DR
The topic of the attractiveness of categorical reasoning in spite of it’s limitations is something about which I have thought a fair bit about. The hour is too late to go on at length, but there are couple comments I will make. One is that folks have been trying to bridge the gap between categorical systems and probabilistic, fuzzy, or connectionist systems for quite a while now but none of the solutions so far have gained broad acceptance. Fuzzy Logic for example gets no respect apart from it’s use in control systems and even then primarily in a few Asian countries and not the US. Perhaps something will come from work on graphical kernels or tensors, but I’m skeptical.
There’s a talk that Christopher Manning gave at the “Where Does Syntax Come From? Have We All Been Wrong?” MIT workshop (which may be the one that Joe was talking about) that I like. He cites Weinreich, Labov and Herzog (1968) who saw 20th century linguistics as having gone astray by mistakenly searching for homogeneity in language, on the misguided assumption that only homogeneous systems can be structured.
The video is here:
http://mitworld.mit.edu/video/506
and his slides are here:
http://nlp.stanford.edu/~manning/talks/
The most ironic thing here I think though, is that I suspect that our attraction for categorical reasoning arises from our facility for using syntax. It is exactly the processes that deal with sign and symbol analogies (which Douglas Hofstadter has elaborated on at length) that have that affinity for categorical distinctions. And it is the detachment of syntax from semantics that enables mathematics to be unconditionally “true” (apart from the need for some axiomatic assumptions, as Gödel pointed out). The irony then being that our language ability draws us into thinking about language in a way that draws us away from thinking about the mechanisms by which it actually operates. Manning also quotes Joos from 1950 who described linguistic phenomenon as being an extreme quantum mechanics in that it excludes continuous measures. But if that were case, then we would observe what the physicists have, namely the “unreasonable success” of mathematics in describing the world.
Ack. That was some length. I have no conclusion, but as I said, it was just some comments…
Telkom University