We propose an optimization framework for interactive image segmentation (IIS) that operates in bilateral space to achieve robust object extraction and instant visual feedback. More specifically, we first resample an input image using a regular bilateral grid with a resolution that is typically coarser than the input image to reduce the complexity of subsequent IIS tasks. We then design a Markov random field energy on the vertices of the bilateral grid that can be solved efficiently using a standard graph cut label assignment. To achieve this, we introduce reliable color models to distinguish the foreground and background despite the presence of extremely difficult cases and a higher-order potential to encourage spatial consistency in segmentation. We conduct comprehensive experiments on three standard interactive segmentation datasets, MSRA 10K, IIS, and PASCAL VOC 2012 segmentation validation set. The results show that the proposed method achieves competitive performance compared with state-of-the-art methods while making the current system efficient in terms of speed.