This talk presents a novel approach to multimodal machine learning based on tolerance near sets (TNS) [1, 2]. We will discuss our approach in terms of representation, algorithm design and applications that focus on language (text) and visual (image and video) modalities [3]. We extend the definition of tolerance near sets [4] to admit multimodal feature embeddings derived from deep neural models for representational learning. Design of TNS algorithm(s) is based on our recent work on feature embeddings derived from deep neural models [5]. Specifically, we present a TNS based supervised classification algorithm that integrates both visual and textual components from memes [6]. We will present results from some applications that involve vision language tasks.