Multi-view clustering seeks an underlying partition of the data from multiple views. Organizing the data into a tensor and then learning a self-expressive latent one to exploit cross-view information has attracted much attention. Most of the recent works mainly focus on the tensor representation, but they fail to directly extract the task-driving affinity matrix for clustering. Such method is typically modeled by a separated two-stage optimization, making the decoupled representation perform unsatisfactorily. One of the core issues is how to explore the common subspace across all views while learning self-representation tensor. To tackle the problem, we propose to jointly learn the two parts within a united optimization framework for consistent clustering performance, thus obtaining Task-driving Affinity Matrix for accurate Multi-view Clustering (TAMMC). First, the proposed method preserves the local affinities of all views via a graph regularization on self-expressive tensor. Second, by penalizing a Laplacian rank on a learned common subspace, our algorithm can guarantee superior clustering. An effective optimization procedure is proposed for the proposed model. Extensive experiments on eight benchmark datasets well demonstrate that our approach, named by TAMMC, achieves superior performances over other popular methods.