Abstract: Current state-of-the-art CNNs can localize and name objects in internet photos, yet, they miss the basic knowledge that a two-year-old toddler has possessed: objects persist over time despite changes in the camera view, they have 3D extent, they do not 3D intersect, and so on. In this talk, I will introduce neural architectures that learn to parse video streams of a static scene into world-centric 3D feature maps by disentangling camera motion from scene appearance. I will show the proposed architectures learn object permanence, can generate RGB views from novel viewpoints in truly novel scenes, can infer affordability in sentences by grounding language in 3D visual simulations, and can learn intuitive physics in a persistent 3D feature space. Our experiments suggest that the proposed architecture is essential to generalize across objects and locations, and it overcomes many limitations of 2D CNNs.

Bio: I received my M.S. in CMU MLD and my B.S. in Electrical Engineering from National Taiwan University in 2013. During my master, I worked with Professor Alex Smola on spectral method for Bayesian models and had succefully designed efficient and provable algorithms for topic modeling. In my undergraduate, I was one of the member in team “Algorithm @ National Taiwan University” led by Professor Chih-Jen Lin ‘s and we won the KDD CUP 2013 Championship on both Track1 and Track2.