We consider a hybrid atom-optomechanical system consisting of a mechanical membrane inside an optical cavity and an atomic Bose-Einstein condensate outside the cavity. The condensate is confined in an optical lattice potential formed by a traveling laser beam reflected off one cavity mirror. We derive the cavity-mediated effective atom-atom interaction potential and find that it is nonuniform, site-dependent, and does not decay as the interatomic distance increases. We show that the presence of this effective interaction breaks the $Z_2$ symmetry of the system and gives rise to new quantum phases and phase transitions. When the long-range interaction dominates, the condensate breaks the translation symmetry and turns into a novel self-organized latticelike state with increasing particle densities for sites farther away from the cavity. We present the phase diagram of the system and investigate the stabilities of different phases by calculating their respective excitation spectra. The system can serve as a platform to explore various self-organized phenomena induced by the long-range interactions.

• Received 29 May 2020
• Revised 2 December 2020
• Accepted 2 February 2021

DOI:https://doi.org/10.1103/PhysRevA.103.023328