The tri-partite architecture of language encompasses phonology, semantics and syntax. One of our general aims is to understand how the language system emerges from and influences non-verbal audio-visual processing. Semantics may form a natural interface between verbal and non-verbal processing. Semantic concepts are a prerequisite for thought and language. Object concepts can be characterized by various sensory features (e.g. auditory, visual) and referred to by an arbitrary linguistic label. In a series of studies, we have sought to identify the organizational principles of semantic memory. Together with other groups, our studies suggest that deep semantic processing may not only involve a fronto-temporal core semantic system, but also sensory-motor regions. For instance, processing of tool and action concepts has been shown to rely on re-activation of action representations in the visuo-motor system via top- down modulation.