A new efficient two-party semi-quantum key agreement protocol is proposed with high-dimensional single-particle states. Different from the previous semi-quantum key agreement protocols based on the two-level quantum system, the propounded protocol makes use of the advantage of the high-dimensional quantum system, which possesses higher efficiency and better robustness against eavesdropping. Besides, the protocol allows the classical participant to encode the secret key with qudit shifting operations without involving any quantum measurement abilities. The designed semi-quantum key agreement protocol could resist both participant attacks and outsider attacks. Meanwhile, the conjoint analysis of security and efficiency provides an appropriate choice for reference on the dimension of single-particle states and the number of decoy states.