Recently more attention has been given to adversarial attacks on neural networks for natural language processing (NLP). A central research topic has been the investigation of search algorithms and search constraints, accompanied by benchmark algorithms and tasks. We implement an algorithm inspired by zeroth order optimization-based attacks and compare with the benchmark results in the TextAttack framework. Surprisingly, we find that optimization-based methods do not yield any improvement in a constrained setup and slightly benefit from approximate gradient information only in unconstrained setups where search spaces are larger. In contrast, simple heuristics exploiting nearest neighbors without querying the target function yield substantial success rates in constrained setups, and nearly full success rate in unconstrained setups, at an order of magnitude fewer queries. We conclude from these results that current TextAttack benchmark tasks are too easy and constraints are too strict, preventing meaningful research on black-box adversarial text attacks.

中文翻译：

---

不要搜索搜索方法——简单的启发式方法足以应对对抗性文本攻击

最近，对用于自然语言处理 (NLP) 的神经网络的对抗性攻击给予了更多关注。一个中心研究课题是研究搜索算法和搜索约束，以及基准算法和任务。我们实现了一种受基于零阶优化的攻击启发的算法，并与 TextAttack 框架中的基准测试结果进行比较。令人惊讶的是，我们发现基于优化的方法在约束设置中没有产生任何改进，并且仅在搜索空间较大的无约束设置中才从近似梯度信息中略微受益。相比之下，在不查询目标函数的情况下利用最近邻的简单启发式方法在受约束的设置中产生了可观的成功率，在无约束的设置中几乎完全成功，查询数量减少一个数量级。我们从这些结果中得出结论，当前的 TextAttack 基准测试任务过于简单且约束过于严格，阻碍了对黑盒对抗性文本攻击的有意义的研究。