Learning to unknot

A language L is composed of words from a vocabulary V(L). In NLP it is useful to have an embedding of a word into a vector space that ideally encodes its meaning:

$$\begin{equation} E: V(L) \to \mathbb{R}^{d}, \end{equation} \tag{ 1 }$$

where d is the embedding dimension.

Since the vocabulary is a discrete set of words, one embedding, known as the one-hot encoding, maps the ith word w_i ∈V(L) as $w_i \mapsto e_i$, where e_i is a unit vector and $d = |V(L)|$. From the NLP perspective, this embedding has a number of issues. First, the dimension of the target vector space is $|V(L)|$, which for any non-trivial language will be quite large. Second, all but one of the entries is zero; the vector is sparse. Finally, the embedding only contains the information of the index in the set V(L), which is arbitrary and can be permuted; no useful information is encoded in the embedding.

One would like a better technique for associating a vector to a word. The problem of sparseness may be solved by choosing $d\lt|V(L)|$, typically $d\ll |V(L)|$. In some cases E is fixed by using pre-trained word vectors for the embedding, while in others E has randomly initialized parameters and a useful embedding is learned by training on some task. In the process, semantics may be learned that encoded meaning into the vector representatives of words. (e.g. [19]) A famous example is

$$\begin{equation} E({\textrm{king}}) - E({\textrm{man}}) + E({\textrm{woman}}) \simeq E({\textrm{queen}}), \end{equation} \tag{ 2 }$$

an approximate equivalence at the level of the vector relationships that encodes an actual semantic relationship in the language. Other semantic relations have also been learned, e.g. related to capitals:

$$\begin{equation} E({\textrm{Paris}}) - E({\textrm{France}}) + E({\textrm{Poland}}) \simeq E({\textrm{Warsaw}}), \end{equation} \tag{ 3 }$$

and pluralization:

$$\begin{equation} E({\textrm{cars}}) - E({\textrm{car}}) + E({\textrm{apple}}) \simeq E({\textrm{apples}}). \end{equation} \tag{ 4 }$$

Clearly, word embeddings that capture semantic features of a word or language could be useful in a variety of machine learning tasks with respect to that language.

In what follows we will be discussing queries and keys, and it will be assumed that each word in a sequence of length l has been mapped to d-vector via an embedding layer, so that each embedded sequence has shape [l, d].

Recent years have seen great progress in NLP with the evolution of the attention mechanism and its introduction into various architectures. It works as the name suggests: by training the neural network to pay attention to the most important parts of sentences.

To explain the mechanism we will utilize the notion of queries, keys, and values [20]. This notion is used because the mechanism mimics the retrieval of a value v_q for a query q based on a key k_i in a database, each of which has its own value v_i . In normal database retrieval, one finds the key k_i that is identical to the query and returns the value. In attention, we wish instead to have a similarity measure s(q, k_i ) between the query and key, which is used as the weight to determine the attention paid to the different elements in a weighted sum of values:

$$\begin{equation} {\textrm{Attention}}(q,k,v) = v_q = \sum_i s(q,k_i) v_i. \end{equation} \tag{ 5 }$$

In this formulation, the case of normal database retrieval is the case where s(q, k_i ) = 1 if q = k_i and 0 otherwise. The different types of attention that exist in the literature [20–23] correspond to different choices for similarity function s, which is chosen to be differentiable (unlike usual database retrieval) to allow for backpropagation in a neural network. The similarity is usually softmax applied to some score function, so that the weights sum to one.

The attention mechanism is a crucial component of the so-called transformer architecture [20], where the version of attention used is known as scaled dot-product attention,

$$\begin{equation} {\textrm{Attention}}(Q,K,V) = {\textrm{softmax}}\left(\frac{QK^T}{\sqrt{d_k}}\right) V, \end{equation} \tag{ 6 }$$

where Q is a set of queries and the keys and values are packed into matrices K and V, and d_k is the dimension of the keys. The softmax function of a vector x is defined as

$$\begin{align} \begin{split} {\textrm{softmax}}: \mathbb{R}^n & \to \mathbb{R}^n\\ x_i & \mapsto\frac{e^{x_i}}{\sum_{j = 1}^n e^{x_j}}, \end{split} \end{align} \tag{ 7 }$$

which is applied to the dot product of the queries with the keys. The scaling in the softmax in (6) by a factor of $1/\sqrt{d_k}$ improves stability of the gradients.

Multi-head attention [20] is a simple variant of attention that can lead to improved training. In multi-head attention, $h\in \mathbb{N}$ different linear projections of the d-dimensional queries, keys, and values are learned, to d_q , d_k , and d_v dimensions, respectively. Attention is then computed for each of the projected queries, keys, and values, which are then concatenated and projected again. The result is known as multi-head attention, with h heads.

The Transformer [20] is an encoder-decoder language translation architecture that uses stacked multi-head attention layers. Since we will be utilizing a memory-efficient modification of the Transformer, we refer the reader to the original literature for further details.

The Reformer is a new architecture, an efficient transformer, that makes a number of memory improvements with respect to the original Transformer and related follow-ups. In this section we review the essential elements of the Reformer, as presented in [24].

Perhaps the key improvement in the Reformer is the use of locality sensitive hashing (LSH) attention. The essential idea behind LSH attention is that, due to the exponential dependence in the softmax in (6), some keys contribute much stronger to attention (for fixed query) than others. This means that the matrix softmax(QK^T ) is sparse and dominated by a few entries, and we want to only compute these dominant ones. This will improve the complexity from $\mathcal{O}(l^2)$ to $\mathcal{O}(l\log l)$, which becomes especially important for long sequences. In more detail, the softmax of a key k_j contributes a factor ${\textrm{exp}}(q_i\cdot k_j)$ to the attention of a query q_i . One now wishes to find the keys k_j with maximal $q_i\cdot k_j = |q_i| \, |k_j| \, \cos(\theta_{ij})$, i.e. finding keys that are nearest neighbors to q_i in a high-dimensional vector space.

Formulated abstractly, a hashing function (or scheme) $h: \mathcal{V} \to \{1,\dots,b\}$ assigns a vector $x\in \mathcal{V}$ to one of b hash values. In cryptography, h is chosen such that the hash values h(x) of nearby values x are as uncorrelated as possible in order to avoid revealing whether a guessed secret x is close to the actual secret. Here, we want the inverse situation: nearby values x should be mapped to nearby hashes h(x). Such a hashing scheme is called locality-sensitive. An example for an LSH scheme uses

$$\begin{equation} h(x) = {\textrm{argmax}}([xR;-xR]), \end{equation} \tag{ 8 }$$

where [u; v] denotes the concatenation of two vectors u and v, R is a fixed random matrix of shape legnth(x)×b/2, and argmax returns the index of the largest vector component [25]. The idea is that under the random projection, nearby vectors will map to nearby vectors and thus receive the same hash with high probability.

Returning to computing the attention (6), we can now only evaluate those scalar products in QK^T that contribute the most. The attention a_i of a query q_i is given by

$$\begin{equation} a_i = \sum_{j \in {\mathcal{P}}_i} {\textrm{exp}}(q_i\cdot k_j - z(i,{\mathcal{P}}_i)) \, v_j. \end{equation} \tag{ 9 }$$

Here, ${\mathcal{P}}_i: = \{j:i\geq j\}$ is the set that the query at position i attends to, the exponential structure comes from the softmax, z is a normalizing term for the softmax, and we have omitted the factor $1/\sqrt{d_k}$ for clarity. Note that the structure of ${\mathcal{P}}_i$ ensures that the ith position in the query may only attend to itself and the prior positions [20].

We now change this attention scheme by only paying attention to elements within the same hash bucket, i.e. we set

$$\begin{equation} {\mathcal{P}}_i^{\textrm{LSH}} = \{\,j: h(q_i) = h(k_j)\}. \end{equation} \tag{ 10 }$$

As discussed above, the computational and memory gains arise because $|{\mathcal{P}}_i^{\textrm{LSH}}|\ll |{\mathcal{P}}_i|$. Sometimes (but rarely), similar vectors will fall in different hash buckets. The chance that this happens can be reduced by performing multi-round LSH attention, i.e. the Reformer uses $n_{\textrm{hashes}}$ distinct hashing functions, defined by distinct, random matrices R.

Additional details of LSH attention in the Reformer include causal masking that ensures positions may only attend to prior positions, and also a chunking scheme that allows for efficient batch processing. In practice, the input with batch-size N is a tensor of shape [N, l, d] which the Transformer then turns into Q, K, and V via three different linear layers. However, for LSH attention in the Reformer to make sense we need Q = K. Similarly, a shared-QK Transformer is a Transformer that has Q = K, and it turns out [24] that this has little effect on performance. Further improvements are achieved by using reversible layers.

In summary, the Reformer is a modern NLP architecture where improvements relative to the Transformer allow sophisticated sequence data to be trained effectively on a single GPU, bypassing the need for extensive computational resources and therefore allowing easy exploration of new domains with NLP techniques. The most important hyperparameters introduced by the Reformer are the number of hashes b in LSH attention, and also the choice of LSH attention or full attention, for the sake of comparison.

Knots have various data presentation as words in appropriate sets of letters, which makes it natural to think of them as language ⁷ . In this section we develop the idea in the context of natural language processing. We start by briefly summarizing some basics of knot theory, and then introduce the braid representation of a knot, which we use in most of our analysis and which can be interpreted as language.

A knot is an embedding of S¹ in 3-dimensional space, without self-intersections and up to ambient isotopy. The main goal of knot theory is to classify all knots, and to develop tools that enable to determine whether two different embeddings of S¹ are topologically equivalent, i.e. whether they represent the same knot—in other words, whether one can be transformed onto the other without cutting. An important specialization of this problem that we address in this paper is to determine whether a given knot is topologically equivalent to the unknot, i.e. an unknotted loop, also referred to as the trivial knot. A collection of several possibly entangled knots is called a link.

One useful approach to analyze knots is to consider their projections on a plane, see figure 1. Two knots are topologically equivalent if and only if their projections can be related to each other by a sequence of Reidemeister moves. These are three special moves that involve one, two, or three strands, see figure 2:

• A twist (figure 2(a)) takes a strand and twists it, changing the crossing number by 1,
• A poke (figure 2(b)) pulls one strand over another, changing the crossing number by 2,
• A slide (figure 2(c)) slides a strand over (or under) a crossing of two strands, not changing the crossing number.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Furthermore, the most basic characteristic of a knot is the minimal number of crossings that one gets upon its projection onto an (appropriately chosen) plane. The simplest knots are the unknot, trefoil and figure-eight knot, denoted respectively 0₁, 3₁ and 4₁, whose (minimal) numbers of crossings are given by the main number in this notation (i.e. 0, 3 and 4), while the subscript labels inequivalent knots with the same number of crossings. The unknot, trefoil and figure-eight are the only knots with fewer than five crossings. For a fixed, larger number of crossings there are many topologically inequivalent knots, e.g. there are two knots with five crossings (denoted 5₁ and 5₂). In addition to these unique prime knots, new 'composite' knots can be formed as the sum of two or more prime knots. This can be thought of as taking two or more prime knots, cutting them open at one position, and tieing the open ends of each knot together; see figure 3.

Zoom In Zoom Out Reset image size

The number of inequivalent knots (and indeed already the number of inequivalent prime knots) with a given number of crossings grows rapidly, so more elaborate characteristics must be employed to encode their structure and to distinguish them. For example, there are 165 prime knots with 10 crossings, 1 388 705 prime knots with 16 crossings, etc.

A given knot clearly has many representations; for example projections on various planes typically look different, and in particular may yield different numbers of crossings. Therefore, one issue we have to deal with is how to represent the structure of a given projection. The second issue one needs to deal with is how to determine whether different representations represent topologically the same type of knot. Let us briefly discuss these two points.

In order to determine a type of a knot, so-called knot invariants are constructed. Knot invariants are various mathematical objects (numbers, polynomials, groups, homologies, etc) which depend only on the topological type of a knot, and have the same form irrespective of the representative used to compute it. To prove that a given quantity is a knot invariant, it is sufficient to show that it is invariant under each of the Reidemeister moves. Note that if an invariant computed for two knots yields two different values, it means that these knots are inequivalent. On the other hand, if two knots yield the same invariant, they may be either equivalent or inequivalent. More powerful invariants distinguish more knots from each other, and a knot theorist's dream is to find a simple and practically computable invariant that would distinguish all knots.

For various purposes, in particular in order to compute various invariants, one needs to encode topological structure of a knot succinctly. The most common strategy to this end is to capture the pattern of crossings in a projection of a knot on a plane; it is clear that such a pattern determines a type of knot under consideration. Note that there are two types of crossings: once we traverse a knot, we may pass under or over each crossing that we come across. Keeping track of this information while we travel along the knot enables us to reconstruct its structure, and one way to capture this information is to use the Dowker–Thistlethwaite notation.

2.4.1. Dowker–Thistlethwaite

2.4.2. Braids

Let us therefore summarize what braids are and how to use them to encode the structure of knots. Recall that the (Artin) braid group Br_n is a non-Abelian, infinite, finitely generated group acting on n strands with generators $\sigma_1,\ldots, \sigma_{n-1}$ and their inverses $\sigma_1^{-1},\ldots, \sigma_{n-1}^{-1}$, which satisfy the relations

$$\begin{align} &{\textrm{Braid~relation~1:~}} \qquad\sigma_i \sigma_{i+1} \sigma_i = \sigma_{i+1} \sigma_i \sigma_{i+1}\,,\phantom{0000000} \end{align} \tag{ 11a }$$

$$\begin{align} &{\textrm{Braid~relation~2:~}} \quad\quad\,\,\,\,\,\,\, \sigma_i \sigma_j = \sigma_j \sigma_i \qquad {\textrm{for}}\ |i-j|\geq 2\,, \end{align} \tag{ 11b }$$

and similarly for the inverses. See figure 5 for depictions of the braid relations.

For a set of n parallel strands, the generator σ_i can be thought of as moving the ith strand over the $(i+1)^{\textrm{st}}$, and its inverse $\sigma^{-1}_i$ as moving the $(i+1)^{\textrm{st}}$ strand over the ith strand. A group element $\sigma_{i_1}^{\pm 1} \sigma_{i_2}^{\pm 1} \sigma_{i_3}^{\pm 1}\cdots$ can be represented as a pattern of interlacing strands and is referred to as a braid, see figure 4 (left). A braid can be turned into a knot diagram by connecting beginnings and endpoints of all strands by a set of n parallel arcs, as in figure 4 (right). This operation is also referred to as closure. Furthermore, a theorem by Alexander states that each knot can be represented as a braid, and there is an effective algorithm that turns a knot into a braid. Note that a braid that we obtain upon such an operation may be regarded as a different knot projection, which of course represents the same knot type.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

As mentioned above, two different projections of the same knot can be related by a series of Reidemeister moves. There is an analogous statement on the level of braids, which is formalized in Markov's theorem. This theorem states that two braids that represent the same knot can be transformed into each other by a series of Markov moves. There are two types of Markov moves, shown in figure 6:

• conjugation; and
• stabilization/destabilization.

Zoom In Zoom Out Reset image size

Conjugation sends a braid word $ww^{^{\prime}}$ to $w^{^{\prime}} w$. This can be achieved by repeating the following two-step procedure for every generator $\sigma_{i_j}$ in $w = \sigma_{i_1}\sigma_{i_2}\cdots \sigma_{i_l}$: First, we multiply $ww^{^{\prime}}$ by $\sigma_{i_1}^{-1}$ on the left and $\sigma_{i_1}$ on the right, giving

$$\begin{align} \sigma_{i_1}^{-1}ww^{^{\prime}}\sigma_{i_1} = \sigma_{i_1}^{-1}\sigma_{i_1}\sigma_{i_2}\cdots \sigma_{i_l} w^{^{\prime}}\sigma_{i_1} = \sigma_{i_2}\cdots \sigma_{i_l} w^{^{\prime}}\sigma_{i_1}\,. \end{align} \tag{ 12 }$$

In the last step, we removed the consecutive inverses $\sigma_{i_1}^{-1}\sigma_{i_1}$. Repeating this for $\sigma_{i_j}$, j = 2, 3, ..., l will result in sending $ww^{^{\prime}}$ to $w^{^{\prime}}w$. Note that the conjugation move corresponds, on the level of the knot or the braid closure, to inserting consecutive pairs of inverse generators, i.e. it acts trivially on the braid closure. Conjugation can be thought of as turning a knot into a braid by 'cutting it open at a different position'. Of course, one can in general conjugate a braid word $w = \sigma_{i_1}\sigma_{i_2}\cdots \sigma_{i_l}$ with any generator $\sigma_{i_k}$, not just with $\sigma_{i_1}$, then $\sigma_{i_2}$, etc, sending it to $\sigma_{i_k}^{-1} w \sigma_{i_k}$ (or $\sigma_{i_k} w \sigma_{i_k}^{-1}$). On the level of the braid closure or the knot, this inserts the identity $\sigma_{i_k}^{-1}\sigma_{i_k}$.

Stabilization and destabilization are given by

$$\begin{align} {\textrm{Stabilization:~}} w&\to w\sigma_{n}\,,\qquad {\textrm{Destabilization:~}} w\sigma_{n}\to w \end{align} \tag{ 13 }$$

for w ∈ Br_n . This changes the braid (in fact, it even changes the underlying braid group from Br_n to Br_n + 1, or the other way around), but not the knot. Note that if we did not change the braid group from Br_n + 1 to Br_n in a destabilization move, we would be left with one strand (the (n + 1)th) which would not be acted on by any of the braid generators. As a consequence, we would have a two-component link: the first component would be a braid that describes a knot equivalent to the one we started with, and the second component would be the unknot, corresponding to the closure of the (n + 1)th strand. Since this is not desired, we take destabilization to remove the generator σ_n and change the braid group.

Note that there is a close connection between the Reidemeister moves and the Markov moves together with the braid relations:

• Reidemeister move 1 (twist) corresponds to Markov move 2, i.e. (de-)stabilization.
• Reidemeister move 2 (poke) corresponds to adding a trivial element $\sigma_i\sigma_i^{-1}$ at some position in a braid word.
• Reidemeister move 3 (slide) corresponds to the action (11a ) of first braid relation on the closure of the braid.

At this stage, we can finally relate braid representations of knots to language. We simply interpret generators $\sigma_i^{\pm 1}$ of Br_n as letters, and braids of the form $\sigma_{i_1}^{\pm 1} \sigma_{i_2}^{\pm 1} \sigma_{i_3}^{\pm 1}\cdots$ (which represent knots after the closure) as words. In practice and in what follows, we represent a braid generator $\sigma_i^{\pm}$ simply by ±i, so that a word is represented by a string of integer numbers $i\in[-(n-1),n-1]$ that represents the braid.

There are several crucial points from the language perspective that should be stressed. First, we wish to identify, and treat as equivalent, different words (braids) that represent the same knot. This means that the AI needs to learn to identify such equivalent words. To conduct such a learning process we also need to generate equivalent words. To this end, we can take advantage of Markov moves. In particular, in the unknot problem, we can generate various representations of the unknot by applying a series of Markov moves to the empty braid. Furthermore, the topological character of knottedness makes the problem global rather than local: even a single change of a word in the sentence may change the type of knot under consideration; there is no notion of a 'small' error, which makes the learning process hard; but this is also true for applications to NLP.

In this section we introduce the main problem that we study: the UNKNOT problem.

2.5.1. Why unknotting?

While the problem of distinguishing knots is interesting in its own right, much of our motivation comes from the smooth 4-dimensional Poincaré conjecture (or, SPC4, as it is often called). Indeed, many problems in topology of 4-manifolds, including SPC4, can be described (and, sometimes, completely reduced) to the language of knots in S³.

At the most basic level, the reason is that every closed smooth 4-manifold M₄ can be represented by a Kirby diagram, which basically consists of knots drawn on a 3-sphere S³. More precisely, to build an M₄ one starts with a 0-handle, i.e. a 4-ball B⁴, and then can attach 1-handles, 2-handles, 3-handles, and/or finally a 4-handle, which is also a 4-ball. In fact, since this last step involves no ambiguity, we do not need to attach the 4-handle. Either way, the relation to knots in S³ comes after all k-handles with k ≤ 3 are attached ⁸ .

There are many candidate counterexamples to SPC4, i.e. 'exotic' spheres M₄ homeomorphic to S⁴ which are not known to be diffeomorphic to S⁴. One way to show that such an M₄ is the standard 4-sphere is to use equivalence relations (Kirby moves) to reduce its Kirby diagram to that of S⁴, which has no k-handles with k = 1, 2, 3. This problem is basically the unknotting problem, or a close variant of it.

Since, as mentioned earlier, adding a 4-handle is a fairly unambiguous operation, one often works with close relatives of SPC4 that involve B⁴ with S³ boundary in place of S⁴. For example, the corresponding version of SPC4 is known as the smooth relative 4-dimensional Poincaré conjecture. If true, it implies the original SPC4. As in the case of SPC4 itself, there are many candidate exotic ⁹ 4-balls, i.e. M₄ homeomorphic to B⁴ which are not known to be diffeomorphic to it. For example, every knot K ⊂ S³ which is fibered and ribbon gives such a candidate M₄ since, according to Casson and Gordon [26], it bounds a fibered disk D ⊂ M₄ in some M₄ which is homeomorphic to a 4-ball B⁴ but is not known to be diffeomorphic to it. Therefore, if a fibered ribbon knot K does not bound any fibered disk in B⁴, then the smooth relative 4-dimensional Poincaré conjecture is false.

Conceptually, this is the same reason why knots in S³ can tell us about smooth structures in one dimension higher that we already mentioned earlier. A knot $K = \partial \Sigma$ appears as a boundary of a surface Σ ⊂ M₄, and the question is whether Σ can be a disk in B⁴ or only in homotopy-B⁴. Whether $K \subset S^3 = \partial B^4$ bounds a disk in B⁴ is controlled by the 4-ball genus (a.k.a. slice genus), g₄(K), which is defined to be the minimal value of g(Σ), such that Σ⊂B⁴ is bounded by K. A knot K with g₄(K) = 0 is called slice.

Then, the strategy [27] to disprove (relative) SPC4 could be to take a knot K ⊂ S³ that is slice (i.e. bounds a disk) in a homotopy 4-ball M₄, with $M_4 \ne B^4$, and show that K is not slice in B⁴. For this, one needs obstructions to sliceness, i.e. lower bounds on g₄(K). One such bound comes from deformations and spectral sequences in Khovanov homology, namely the Rasmussen's s-invariant [28]. It bounds the 4-ball genus:

$$\begin{equation} \frac{|s(K)|}{2} \; \le \; g_4 (K). \end{equation} \tag{ 14 }$$

More generally, one may hope to find exotic 4-balls by looking for knots that exhibit different genus bounds in B⁴ and in $M_4 \simeq B^4$. In [27], this strategy was applied to co-cores of 2-handles, which are disks in $M_4 \simeq B^4$ bounding knots and links in S³. All those M₄ were soon shown to be standard [29].

One can also consider knots with the trivial Alexander polynomial, Δ_K (x) = 1. In the early 1980's Freedman showed that all such knots are topologically slice [30]. Therefore, demonstrating that any such knot has $g_4 (K) \gt 0$ would immediately imply the existence of an exotic 4-ball. A similar conclusion follows if any fibered ribbon knot, as discussed above, has $g_4 (K) \gt 0$.

2.5.2. Complexity

After describing some motivation for unknotting, let us see how hard it can be.

More than 20 years ago, Hass–Lagarias–Pippenger [31] proved that the unknotting problem, i.e. the decision problem whether a given knot K is actually an unknot, is in complexity class NP ('Nondeterministic Polynomial-time' Turing machine). This is the complexity class that, famously, contains P (class of problems ¹⁰ for which 'polynomial-time' algorithms are possible) but is not known to (and, in fact, widely not believed to) be equal to it. Problems in class NP are like Sudoku puzzles; they may not have a simple algorithm to solve, but a proposed solution can be verified in polynomial time. In other words, while problems in class P are the ones for which an answer can be found in polynomial time, problems in class NP are the ones for which checking the answer can be done in polynomial time, provided that the answer is yes. The result of [31] means that the unknotting problem joins the class of problems like protein folding, SAT (satisfying truth assignment), or the traveling salesman problem, which are also in class NP.

A close cousin of the class NP—which, though not too likely, may be equal to it—is the class coNP. It consists of decision problems whose negative answers can be checked in polynomial time, i.e. if the answer is no. If ${\textrm{NP}} \ne {\textrm{coNP}}$, then ${\textrm{NP}} \ne {\textrm{P}}$ (but the other direction is not known). The unknot recognition problem turns out to be not only in class NP but also in the complexity class coNP. This was first shown by Kuperberg [32] assuming the generalized Riemann hypothesis (GRH). This assumption was later relaxed in [33], where it was also pointed out that, in the unlikely event that either the unknotting problem or its negation (called knottedness) is NP-complete, then NP = coNP. NP-complete problems are problems that are in NP with the property that any other problem in NP can be mapped onto this problem in polynomial time. To summarize:

$$\begin{equation} {\textrm{unknot~recognition}} \; \in \; {\textrm{NP}} \, \cap.{\textrm{coNP}}. \end{equation} \tag{ 15 }$$

This result is particularly interesting because many decision problems that originally started in this intersection—e.g. deciding whether an integer number is prime or composite—were later found to be in class P [34]. Therefore, there is a chance that the unknotting problem we are trying to tackle here actually admits a polynomial time algorithm. Approaching this problem via AI/ML can hopefully help us find such an algorithm, if it exists.

In fact, it has been a long standing problem whether the unknot recognition is truly more difficult than a similar problem for braids, the braid word problem. The latter is known to be in class P according to the Garside-Thurston theorem, which says that one can identify the trivial braid in polynomial time, $O(|{\textrm{word~length}}|^2 \;n \log n)$ for the Artin braid group Br_n . This can be improved to $O(|{\textrm{word~length}}|^2 \;n)$ with the BKL algorithm [35] ¹¹ .

At the same time, perhaps one should not be overly optimistic. For example, it was shown recently that imposing an upper bound on the number of Reidemeister moves immediately makes the unknot recognition problem NP-hard [36] (NP-hard problems are problems that are at least as hard as any problem in NP). This paper also helps to understand how the unknotting problem compares to deciding whether two vertices of a given finite graph are connected or not, which is in class P. Indeed, if we think about knot diagrams as vertices of an abstract graph, with edges representing Reidemeister moves, then the unknotting problem is equivalent to deciding whether a vertex belongs to the same component of the graph as the 'origin' (the vertex associated with a trivial diagram of the unknot). If this abstract graph was finite and explicitly presented, then the unknotting problem would be in class P, but [36] can be viewed as an indication that these two problems are qualitatively different.

Finally, since earlier we talked about computation of delicate knot invariants, it should be noted that many closely related problems were recently shown to be parsimoniously #P-complete [37]. This is one of the more esoteric complexity classes, based on #P which is larger than NP but is contained in PSPACE ('Polynomial-space'). And, 'parsimoniously complete' refers to a more specific version of the completeness relation, such that for every solution of problem A there is a unique solution of problem B. Note, the class PSPACE also contains NP and coNP that we discussed earlier, as well as the probabilistic version of the polynomial time solver (BPP). Interestingly, both [32] and [37] use the representation variety $\pi_1 (S^3 \setminus K) \to G$ in a crucial way. When $G = SL(2,{\mathbb C})$, this is the familiar A-polynomial that plays an important role in Chern–Simons theory [38].

It would be interesting to study the distribution of knots introduced by the flat prior from which the Markov moves are drawn. In particular for small crossing numbers, all topologically inequivalent knots have been classified. This means one should in principle be able to check for our database of knots of this length, which of the inequivalent knots are produced (and how often). However, identifying the specific knot associated to our randomly generated braids is a hard problem that requires computing and comparing knot invariants (or ML). In general this problem is difficult and beyond the scope of our paper.

However, knots with nine or fewer crossings have a particularly nice property: they may be uniquely identified by their Jones polynomial [39], and therefore $n_{\textrm{letters}} = 9$ braids drawn from our prior have knots closures that may be identified. By computing the Jones polynomials of the prime knots of nine or fewer crossings (e.g. using knots in the Rolfsen table and their mirrors, which are obtained by inverting every generator of the corresponding braid word) one may compute the Jones polynomials of all knots of nine or fewer crossings by taking products. Drawing 6455 $n_{\textrm{letters}} = 9$ braids from our prior, we may identify the knot closure of each by its Jones polynomial. We see from figure 7 that the distribution on the number of crossings induced by our prior is much flatter than the one induced by a uniform distribution on knots with nine or fewer crossings, which grows exponentially. We provide detailed counts of which knots have been generated how often with our RandomKnot generator (algorithm 6) in the histograms in figure 8.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

We now study network confidence and its correlation with the Jones polynomial, focusing on the simplest case N = 12 because computing the Jones polynomial is #P-hard [41]. Specifically, we train a Reformer on non-trivial knots as well as unknots drawn from the braid priors of section 3, using the parameters summarized in table 2.

Table 2. Parameters for the Reformer used to study network confidence.

Parameter	Values
$n_{\textrm{hashes}}$	4
Causal	False
Bucket Size	6
Depth	10
Embedding dimension	250
Epochs	250
Optimizer	RMSProp
Learning rate	.0001

We test the trained network on 1000 non-trivial knots and unknots from a test set. Results are presented in figure 11. Since we label non-trivial knots and unknots as 0 and 1, respectively, these should be the locations of peaks in the output distributions of well-trained networks. The experimental distributions presented in figure 11 show this correlation. Indeed, the networks are performing well, with precision >95% (see blue curve in figure 9(c)). We would like to point out the following central observations about the NN predictions for both unknots and non-trivial knots:

• Very high confidence. Over 900 of the non-trivial knots (unknots) have outputs within 10⁻³ of their target value 0 (1). This shows that for over 90% of the non-trivial knots and unknots, the network is very confident in its prediction.
• Hard knots ¹³ revealed by small peaks on the wrong label. For both the non-trivial knot and unknot distributions, we see small peaks at the wrong end of the spectrum, at 1 for non-trivial knots and 0 for unknots. While this looks almost negligible on the plots because the peak at the correct values is very large, when one restricts the output distributions to the case of the network making wrong predictions, most of the wrong predictions for non-trivial knots (unknots) occur in the bin closest to 1 (0). For these non-trivial knots or unknots, it is not that the network is unsure of its prediction; rather, it is quite sure, but the prediction is wrong. We checked that this was not simply a function of initialization by running the same experiment with 10 different random initalizations. We found that 22 of the 1000 non-trivial test set knots have output $\gt.95$ for all 10 runs, and similarly 22 of the 1000 test set unknots have output $\lt.05$ for all 10 runs. It is therefore natural to conjecture that these examples are fundamentally hard for the NN, i.e. they possess some adversarial property that makes the NN predict the wrong answer with high confidence.Since knots with 9 or fewer crossings may be identified by their Jones polynomials, we ran experiments for N = 9 braids to identify the hard braids. Specifically, we ran five Reformers, each with 4 hashes, embedding dimension 250, and depth 10, on 5000 N = 9 braids from our prior with an 80/20 test-train split. We trained each for 50 epochs and kept the best model, and accuracy was ∼ 93% on the test set for all five runs. However, of the 1000 braids (with non-trivial knot closures) in the test set, 30 of them had outputs above 0.9 for all five experiments. These hard knots are
  $$\begin{align} (3_1,19,242),\,\,\, \, (4_1,3,37), \,\,\, \, (6_2,1,34), \,\,\, \, (6_3,1,30), \,\,\, \, (8_{13},1,5), \,\,\,\, (8_{20},4,19), \end{align} \tag{ 17 }$$
  where the first entry of the tuple is the knot number in the Rolfsen table, the second (third) is the number of 'hard' (total) instances of the knot in the test set. For knots that are not self-mirror, mirrors are included in the counts. We find that there are no hard knots with 9 crossings, and that ∼ 2/3 of the hard knots are trefoils, despite the fact that only ∼ 1/3 of the knots in the test set are trefoils. While more statistics are necessary to draw a firm conclusion, this preliminary analysis seems to suggest that knots with fewer crossings are more likely to be hard. A possible explanation for why of all things the simplest non-trivial knot, i.e. the trefoil, seems to confuse the networks could be that at fixed length N, knots with a smaller number of crossings in the minimal representation (three for the trefoil) contain more crossings that can be undone or disentangled. This is of course also the case for the unknot, where all crossings can be removed. This similarity might cause the NN to being tricked and 'overlooking' that there is still a non-trivial component left in the braid representation of the trefoil knot after removing all superfluous crossings.
• Network uncertainty. The network is uncertain when its output is around 0.5. To set a more precise threshold, let us say that the network is uncertain if $0.3 \lt {\textrm{output}} \lt 0.7$. As described in the first point, there are very few knots for which a given NN is uncertain to begin with. Moreover, the uncertain knots are not robust to initialization: across the 10 random initializations just mentioned, we find that there are no unknots or non-trivial knots for which all of the networks are uncertain.

Zoom In Zoom Out Reset image size

We also study correlations between network outputs and the Jones polynomial. Specifically, for all knots in the ensemble we compute the Jones polynomial using SageMath [42] and compute the maximum of the absolute value of the degrees of the monomials. Since the Jones polynomial of the unknot is just the constant polynomial 1 (in the normalization of SageMath), we think of this as serving as a measure of the complexity of the Jones polynomial and hence of the non-triviality of the knot. In the bottom plot of figure 11, we stratify the knots according to this measure of the degree and plot it against the mean of the network outputs. There is clearly a direct correspondence: on average, the higher the complexity of the Jones polynomial, the more confident the network is that the knot is, in fact, non-trivial. We emphasize that this correlation was learned by the network and not put in by hand: the Jones polynomial did not enter anywhere into the training process.

Interestingly, some unknots admit a precise notion of 'hardness': there is no sequence of Reidemeister moves that simplifies them without increasing the number of crossings at some point. We provide an example where this phenomenon is illustrated for a simple braid representation of a knot around equation (21) in section 5.1, but at this point we do not have a systematic way of constructing examples of such braids. However, 176 examples of such knots with 15 crossings have been constructed in the Dowker–Thistlethwaite representation in [43, 44]. It is an interesting question how this property is related to the representation of the knot, which is, however, beyond the scope of this paper. In particular, we do not know whether this property is preserved when moving from the Dowker–Thistlethwaite representation to the braid representation of a knot. Translating Dowker–Thistlethwaite to braids is not straight-forward and might require moving some of the strands, thereby destroying the hardness property.

In order to study these 'hard' knots nevertheless, we therefore have to use the hard Dowker–Thistlethwaite knots of [43, 44]. However, creating a training set in Dowker–Thistlethwaite notation would require new methods of generating random non-trivial knots as well as unknots in this notation. In order to use our existing methods, we generate knots and unknots in braid notation and translate them into Dowker–Thistlethwaite notation. This direction is, in contrast to the other direction, straight-forward.

With this data set, we trained a Reformer and simple feedforward network on the Dowker–Thistlethwaite representations of length 15 drawn from our prior on braids. The Reformer was acausal, with bucket size 8 and four hashes, depth 10, with five heads, and embedding dimension 250. Evaluated on a test set of 1000 non-trivial knots and unknots, the trained Reformer had accuracy 90% on knots and 87.7% on unknots, and the feedforward network performed slightly worse. Moreover, these results are ∼ 5% lower than Reformers or feedforward networks trained on the knots represented as braids, suggesting that braids may be a better representation of knots for machine learning than Dowker–Thistlethwaite notation.

The reason for doing the analysis on knots in Dowker–Thistlethwaite notation, however, was to test it on the 'hard' Dowker–Thistlethwaite knots, as just defined. Using the trained Reformer to make predictions for the 176 known hard Dowker–Thistlethwaite knots from [43, 44], none of which were explicitly used in the training set, the Reformer achieved only 2.2% accuracy—it was almost always wrong! Concretely, the mean and standard deviation of outputs for these unknots were .03 and .14, respectively, demonstrating that the network is quite sure of its wrong predictions. While this performance could be blamed on the out-of-sample prediction of hard unknots, we also observed that including a fraction of the hard unknots does not improve prediction accuracy for the other hard unknots. This gives a second reason, aside from the definition above, to think that these unknots are fundamentally hard.

5.1.1. State space

The states of the reinforcement agent are all braid words of a given length $2N_{\textrm{max}}$ for starting braids of length $N_{\textrm{max}}$ (the reason for the factor of 2 will become apparent once we discuss the actions below). Since the start state will be a braid whose closure is a knot, rather than a link, and this property is preserved under our actions, we are only considering braids with single-component closures. For such braids, a braid word of length N has at most 2 N generators and is thus an element of (at most) Br_N + 1. Therefore, there are (at most):

$$\begin{align} N_{\textrm{states}} = 1+\sum_{N = 1}^{2N_{\textrm{max}}} (2~\textrm{N})^N, \end{align} \tag{ 18 }$$

possible states. Out of these, the only terminal state is the state corresponding to the empty braid word in Br₁. Here, the subtlety with the destabilization move discussed around (13): Even if we start with a braid word $w\in{\textrm{Br}}_{N_{\textrm{max}}}$ corresponding to a one-component knot, after reducing its length to some $N\lt N_{\textrm{max}}$ the braid is taken to be an element of Br_N rather than ${\textrm{Br}}_{N_{\textrm{max}}}$ to preserve the single-component property ¹⁴ .

5.1.2. Reward function

5.1.3. Action space

Remember that performing the Markov moves depicted in figure 6 change the braid but not the braid closure, i.e. not the knot. However, just using Markov moves does not guarantee that the unknot which corresponds to the empty braid word can be reached. Indeed, any given braid configuration of the unknot will be reducible to the empty braid word by using Markov moves together with the braid relations (11). As a simple example, consider the braid word:

$$\begin{align} w = [1,2,1,-2]. \end{align} \tag{ 19 }$$

Just Markov moves alone will not simplify this to an empty braid word corresponding to the unknot. However, if we use the first braid relation, this braid can be seen to be equivalent to the braid:

$$\begin{align} w^{^{\prime}} = [2,1,2,-2]. \end{align} \tag{ 20 }$$

Now, removing the consecutive inverses [2,−2] will lead to the braid word [2, 1] ¹⁵ , which, after two destabilization moves, collapses to the empty braid word.

This means that we need in principle four different kinds of actions for the agent: the two types of braid relations (11), as well as the two Markov moves. Let us count the number of possible actions: In order to carry out the two braid relations, we need to specify the position in the braid at which the relations are to be used, adding $N_{\textrm{max}}$ actions each. Markov move 1, i.e. conjugation, consists of two consecutive actions: multiplying by a generator and its inverse on either side of the braid word, and subsequently simplifying the braid. This adds two (composite) actions. Markov move 2, if allowed, either adds or removes a strand from the braid, adding another two actions.

However, there is another subtlety here: If we perform conjugation as a composite action of adding consecutive inverses on the closure of the braid and removing a different set of consecutive inverse operators, we prescribe an order in which these operations are to be carried out. Fixing this order means that some braids of unknots cannot be simplified to the empty braid word anymore. As an example, consider $w = [-1,2,1,-2]$. This cannot be simplified by applying conjugation (i.e. cyclic shifts), braid relations, and destabilization moves. However, if we (i) insert consecutive inverses at the second to last position, (ii) use braid relation 1 on positions 2-4, (iii) remove consecutive inverses at positions 1 and 2, (iv) conjugate by −2 on the left and 2 on the right and remove consecutive inverses at the ends of the braid, and (v) performing two destabilization moves, the braid word w becomes the empty braid word:

$$\begin{align} \begin{split} w = [-1,2,1,-2] &\xrightarrow{{\textrm{(i)}}} [-1,2,1,2,-2,-2] \xrightarrow{{\textrm{(ii)}}} [-1,1,2,1,-2,-2] \xrightarrow{{\textrm{(iii)}}} [2,1,-2,-2]\\ & \xrightarrow{{\textrm{(iv)}}} [1,-2]\xrightarrow{{\textrm{(v)}}} \emptyset \end{split}. \end{align} \tag{ 21 }$$

This is an example of the previously mentioned fact that for some knots the length of the braid word (i.e. the number of crossings) has to be increased before it can be decreased and illustrates that we need to allow the insertion of consecutive inverse operators at some point in the braid, without specifying a priori the order in which they are removed, or whether one applies other operations, such as using the first braid relation and does not remove the generators at all. Naively, this requires adding the possibility of inserting $2N_{\textrm{max}}$ generators $\sigma_i^{\pm1}\sigma_i^{\mp1}$ at any of the $N_{\textrm{max}}$ positions of the braid, thus adding $2N_{\textrm{max}}^2$ actions.

In total, this set of actions would have

$$\begin{align} N_{\textrm{actions}} = 2N_{\textrm{max}}^2+2N_{\textrm{max}}+4, \end{align} \tag{ 22 }$$

actions, which can be several thousands. In our experience, RL works particularly well for small action spaces, while the state space can be very large ¹⁶ . In the case at hand, the action spaces can become quite sizable. Moreover, they contain many illegal actions, since most braid relations can only be applied at very few positions in the knot. This large number of illegal actions means that the braid is often not changed by an action, which in turn requires a very long exploration phase by the agent in order to realize which actions are valid based on which input states. To counter this, we do not use the set of actions described above but rather introduce a different set of high-level actions that does not grow as fast with $N_{\textrm{max}}$.

We have tried several agents with different types of composite actions, but we will only discuss the one that works best here. First, we add cyclic shifts to the left and right in the braid word, thus including Markov moves of type 1. This way, we have covered the Markov moves of type 1 with 2 actions. Next, we need to address equivalences of the braid. Here, we can potentially save a huge number of actions, if we find a more efficient way to describe the insertion of consecutive inverses (which introduced $2N_{\textrm{max}}^2$ actions) and for the braid relations (which introduced $2N_{\textrm{max}}$ actions).

For the consecutive inverses, we can get away with only $N_{\textrm{max}}$ actions in the following way: Since conjugation, i.e. cyclic shifts, are already included in the actions, it does not matter where the inverse operators are inserted, so we insert them at the beginning and the end of the braid word. This reduces the number of actions by a factor of $N_{\textrm{max}}$, since we need not specify the position anymore. Theoretically, this reduction comes at the cost of performing up to $\lceil N_{\textrm{max}}/2\rceil$ cyclic shifts after adding the inverse pair of generators in order to move them to any desired position in the braid word. In practice, we only care about the braid closure anyways, so this shift will never be necessary. Hence, it seems as if we need to add $2N_{\textrm{max}}$ actions, $N_{\textrm{max}}$ that send $w\to\sigma_i w \sigma_i^{-1}$ and another $N_{\textrm{max}}$ that send $w\to\sigma_i^{-1} w \sigma_i$. However, it is enough to include only the first $N_{\textrm{max}}$: The second $N_{\textrm{max}}$ can be obtained from the former by performing a cyclic right-shift, commuting the pair of inverse operators, and performing a cyclic left-shift:

$$\begin{align} \sigma_i w \sigma_i^{-1} \to \sigma_i^{-1}\sigma_i w \to \sigma_i\sigma_i^{-1} w \to \sigma_i^{-1} w\sigma_i. \end{align} \tag{ 23 }$$

We have thus reduced the number of actions coming from insertion of consecutive inverses from $2N_{\textrm{max}}^2$ to $N_{\textrm{max}}$. However, note that inserting consecutive inverses increases the length of the braid word by 2 (this is also true for the original $2N_{\textrm{max}}^2$ actions). Since we use NNs to approximate the state and action value functions, and since we need to fix the input dimension of these NNs, we allow a maximum intermediate length of $2N_{\textrm{max}}$ for a braid word with original length $N_{\textrm{max}}$.

In addition, we bundle several simplifications (i.e. removing trivial link components, relabeling braid generators, performing destabilization moves, and removing inverses of a pair of operators that are either consecutive or separated by other braid generators with which the generator and its inverse commute) into one action called SmartCollapse. These operations are repeated until the braid does not simplify further. Note that we did not add a stabilization move. While we do not know whether there are situations where one needs to perform a stabilization in order to eventually get to the empty braid word, we observe empirically that the performance of the agents went slightly down when adding this extra action. Of course, longer training or better hyperparameter tuning, plus maybe allowing for longer intermediate braid words, should overcome this drop in performance. However, from the drop in performance we do not see evidence that, for the knots in our database, stabilization contributes significantly to simplifying braid words of unknots to the empty braid word. If in doubt, one could of course add this extra action at the cost of a few percent accuracy or finding better hyperparameters.

Next, let us address reducing the $2N_{\textrm{max}}$ actions coming from the braid relations. This can be achieved as follows:

• (a)  
  Start at the beginning of the braid word.
• (b)  
  Use a braid relation at the first possibility.
• (c)  
  Use conjugations (i.e. cyclic shifts) to move the position where the braid relation has been applied to the end of the braid word.

This reduces the number of actions from $2N_{\textrm{max}}$ to 2. The price to pay is that if the network wants to perform a braid relation at a specific position, it might have to perform, in the worst case, $N_{\textrm{max}}$ additional cyclic shifts in order to move the position where the braid relation is to be applied far enough to the left such that it is the first occurrence.

Note that one could be tempted to not do the cyclic shift in the third step, but instead remember where the last action was performed and start from that position the next time. However, this would break the Markov property (since the next action on a state would then depend on how the state was reached) and this would make the problem not (necessarily) amenable to be solved with RL.

To summarize, this leaves us with

$$\begin{align} N_{\textrm{actions}} = 5+N_{\textrm{max}} \end{align} \tag{ 24 }$$

actions on a braid word $[i_1,i_2,\ldots,i_k] = w\in{\textrm{Br}}_{n+1}$, i_j ∈[−n, n]:

• SmartCollapse (see Algorithm 3): removes twists, performs destabilization, removes inverses.
• shift left (conjugation + remove inverses):
  $$\begin{equation*} w\xrightarrow{a_{n+1}}(-i_1)\circ w \circ (i_1) = [i_2,\ldots,i_k,i_1]. \end{equation*}$$
• shift right (conjugation + remove inverses):
  $$\begin{equation*} w\xrightarrow{a_{n+2}} (i_k) \circ w \circ (-i_k) = [i_k,i_1,i_2,\ldots,i_{k-1}]. \end{equation*}$$
• braid relation 1 and shift right: let m be the position where the braid relation can be applied and $s = [i_{m+1},i_{m+2},i_{m+3}]$ be the three-letter substring to which it is applied, yielding $s^{^{\prime}}$. Then
  $$\begin{equation*} [i_1,i_2,\ldots,i_m,s,i_{m+4},\ldots, i_k]\xrightarrow{a_{n+4}} [i_{m+4},\ldots, i_k,i_1,i_2,\ldots,i_m,s^{^{\prime}}]. \end{equation*}$$
• braid relation 2 and shift right: let m be the position where the braid relation can be applied and $s = [i_{m+1},i_{m+2}]$ be the two-letter substring to which it is applied, yielding $s^{^{\prime}}$. Then
  $$\begin{equation*} [i_1,i_2,\ldots,i_m,s,i_{m+3},\ldots, i_k]\xrightarrow{a_{n+5}} [i_{m+3},\ldots, i_k,i_1,i_2,\ldots,i_m,s^{^{\prime}}]. \end{equation*}$$
• $N_{\textrm{max}}$ Markov moves of type 1, i.e. conjugations by an arbitrary generator $i_m\in[-N_{\textrm{max}},-N_{\textrm{max}}+1,\ldots,-1,1,\ldots,N_{\textrm{max}}]-$, do not remove inverses:
  $$\begin{equation*} w\xrightarrow{a_m}(i_m)\circ w \circ (-i_m) = [i_m,i_1,i_2,\ldots,i_k,-i_m]. \end{equation*}$$

Note that the only action that simplifies (i.e. reduces the length of) a given braid word is SmartCollapse. All other actions serve the purpose of applying braid relations and Markov moves that will eventually allow SmartCollapse to simplify the braid word. Also note that now the only illegal actions are the ones that try to add a pair of inverse generators to a braid word that is already of length $2N_{\textrm{max}}$. In order to get to such a braid word, the agent has to perform $N_{\textrm{max}}$ of such actions before encountering an illegal move for the first time. We do not know whether there exist knots whose braids require increasing the length (i.e. the number of crossings) by more than a factor of two at some intermediate step, before the knot can be simplified and collapsed to the unknot. However, especially for larger $N_{\textrm{max}}$, this will incur quite a sizable punishment over a large number of steps, which makes it unlikely that the agent would follow this policy unless we choose a discount factor of 1 (or very close to 1). While the SmartCollapse action might not (be able to) change the braid, we do not consider this an illegal action.

Many RL algorithms have been developed to solve a Markov Decision Process. We have tried:

• A3C (asynchronous advantage actor-critic) [46] and the synchronous version A2C, with feedforward and reformer NNs.
• DQN (deep Q-networks) [47], with and without dueling, with feedforward and recursive (LSTM) NNs.
• PPO (proximal policy optimization) [48], with feedforward NNs.
• TRPO (trust-region policy optimization) [49] with GAE (generalized advantage estimation), with feedforward and reformer NNs.

We used different libraries based on Tensorflow/Keras [50], Pytorch [51], and ChainerRL [52]. We performed thorough (but not excessively extensive) box searches for the hyperparameters and tried varying the NN architecture. Based on these experiments we found that PPO and DQN performed worse. Moreover, DQN showed oscillatory behavior in the reward (and the actual performance). A2C and A3C performed approximately on the same level (with A3C slightly outperforming A2C), and both were much better than DQN and PPO. By far the best algorithm was TRPO. We tried using conventional FFNN as well as the Reformer NN with TRPO. While the reformer seemed to perform slightly better, it trains much slower. Since the results with a classic FFNN were already very good, we ended up using the latter in a ChainerRL implementation of the algorithm with the hyperparameters of [53] ¹⁷ . We train the agents for 5 × 10⁶ steps on a standard GPU, which takes around 24 h.

Since we ended up using TRPO, let us briefly explain the ideas behind this algorithm. As is the case for example in A3C, TRPO uses a NN to approximate the state value and action value functions. However, while A3C uses gradient descent to minimize the mean square error for the loss function and uses this as a baseline for the policy updates (which are also performed using gradients), TRPO follows a different approach. The problems TRPO tries to address are the following: First, since gradient-based algorithms are by definition insensitive to curvature, this can lead to problems for strongly curved loss landscapes. Second, the step size should be a function of the curvature, making smaller updates in strongly curved regions and larger updates in flatter regions. Third, TRPO can guarantee policy updates that improve the policy and will eventually lead to the optimal policy.

To address the first point, TRPO uses standard SGD for the value function updates, but a second-order update (based on conjugate gradients and a line search) for the policy function updates. The second point is addressed by introducing trust regions. The algorithm determines, for a given size of the trust region, the maximum step size that it wants to explore and then finds the optimal point within this trust region. Concerning the third point, instead of optimizing the policy function, TRPO optimizes a surrogate function (based on the KL divergence between the old and the new policy) that approximates the expected reward (computed from the current policy) locally. The authors of [49] show that if this surrogate function bounds the expected reward from below, this is guaranteed to lead to an improvement.

In order to evaluate the performance of the RL agents, we run them on braid words that represent unknots. The reason is that for these we know that there exists a series of moves that reduce the braid word to the empty braid word. For non-trivial knots, we do not know what the simplest braid word is (this is the whole point of using ML for this task), hence we cannot judge how well the algorithm performs on non-trivial knots.

The agents receive braids of length $N_{\textrm{max}}\in\{12,24,36,48,72,96\}$. Since, as discussed above, inserting a pair of inverse generators can increase the length of the braid, we introduce a maximum length of $2N_{\textrm{max}}$ as input size for the NN and apply zero-padding. Note that zero is not a valid generator in our convention; since we use ± i to encode the generator $\sigma_i^{\pm1}$, we start at i = 1. For that reason, 0 is a good pad value. We define accuracy as the fraction of unknots for which the RL agents found a series of moves that reduced the input braid word to the empty braid word. We also benchmark the trained agents against a random walker (RW), which does not follow any sophisticated policy but draws the next action from the set of all actions with a flat prior. We set an upper bound of 500 actions and check performance on the same 10 000 unknots for all algorithms.

We present the results of the RL runs in figure 12. To keep the plot manageable, we only show the two best-performing RL agents (TRPO and A3C), both with FFNNs, as well as the performance of the random walker. Let us start by discussing the plot on the left in figure 12. For TRPO, we find that the fraction of braids encoding the unknot that can be fully reduced to the empty braid word is around 85% for all unknots, more or less irrespective of their length and number of generators (within the interval we tested). While results for A3C are comparable for smaller braids (up to $N_{\textrm{max}} = 48$), the accuracy drops significantly for $N_{\textrm{max}} = 72$ and $N_{\textrm{max}} = 96$ from around 70% to 34%. Since there seems to be no fundamental obstruction and A3C is working well for smaller $N_{\textrm{max}}$, we expect that more complex NNs, longer exploration phases, a discount factor closer to 1, and possibly tweaks to other hyperparameters could lead to better performance also at larger $N_{\textrm{max}}$. However, we did not attempt this since TRPO already performed at a constant, higher rate. Both RL runs are to be contrasted with the performance of the random walker. This shows a sharp drop in accuracies, from around 64% accuracy for $N_{\textrm{max}} = 12$ to around 10% for $N_{\textrm{max}} = 96$. This illustrates that the strategy (policy) learned by the agents significantly outperforms brute-force searches, especially for larger $N_{\textrm{max}}$.

Zoom In Zoom Out Reset image size

We also plot the average number of actions the agent takes until obtaining the empty braid word on the right in figure 12. For $N_{\textrm{max}} = 12$, the trained TRPO agent takes an average of around 50 steps to fully reduce a braid word of the unknot to the empty word. In contrast, the random walker needs twice as many steps until it stumbles upon a solution. For this low $N_{\textrm{max}}$, the A3C agent still performs very similarly to the TRPO agent. For larger $N_{\textrm{max}}$, the average number of actions needed by the A3C agent and the RW increases. However, the average number of steps needed by the RW is by a factor of 1.3–1.8 larger as compared to the A3C agent. Interestingly, the number of steps is almost constant for the TRPO agent and a factor of 2 or more below the random walker. Note that this is only averaging the number of actions over those knots that could actually be reduced to the unknot, which is around 8800 for TRPO, 3400 for A3C, and 1000 for RW at N = 96. The results suggests that if the number of steps would be bound to be around 60 instead of 500, the accuracy of the random walker would already be very low even for $N_{\textrm{max}} = 12$. Since the computational complexity of the actions (especially of SmartCollapse) grows with the length of the braid word, the smaller the number of actions, the faster the algorithm performs. Conversely, this means that a brute-force approach to the unknotting problem becomes unfeasible for larger $N_{\textrm{max}}$.

In reinforcement learning, the learning process of the AI agent is represented by the flow of the state-dependent distribution on action space, i.e. a change in policy. Therefore, as a simple attempt in trying to understand what the agent has learned, we look at the distribution of actions that the agent performs before and after training, as summarized in figure 13. We look at the largest, most complicated braids at N = 96. The actions are those summarized at the end of section 5.1. In the plot, we have counted all n actions that perform Markov move 1 in one category labeled 'Markov 1' and divided by N.

Zoom In Zoom Out Reset image size

The first thing to observe is that the untrained agent performs every actions equally often. This is to be expected, since it is just choosing actions randomly from a flat prior. The fact that around 10% of the unknots represented by braid words of length 96 can be reduced to the empty braid word by just performing actions drawn from our base set of actions with a flat prior (see figure 12) sets a baseline against which we should benchmark the performance.

For the trained agent, we see that shift left and SmartCollapse are the most frequent actions. This makes sense. First, SmartCollapse is the only action that can actually reduce the length of the braid word (and thus decrease the punishment); all other actions either leave the length unchanged or increases it by two. Second, the large asymmetry between shifting left and right (the former is the most frequent action, the latter the least frequent action and essentially negligible) is owed to the fact that applying braid relations 1 and 2 are followed by many right shifts, which move the part that the braid relation has acted upon to the end of the braid word. Hence, the agent gets many right shifts for free, which it might have to counter with shifting left in order to rearrange the braid such that the part where the agent wants to apply a braid relation to next is the first possible occurrence.

Next, we observe that braid relation 1 is used around 10% points more often than braid relation 2. This is explained by the fact that SmartCollapse will remove non-consecutive inverses if the generators between a generator and its inverse commute with the generator, in other words SmartCollapse $(w \sigma_i w^{^{\prime}} \sigma_i^{-1}) = ww^{^{\prime}}$ if σ_i commutes with all generators in $w^{^{\prime}}$. Hence, the agent gets again many commutation moves for free in smart collapse and hence needs to perform these less frequently.

Finally, we find that Markov move 1 is performed rather infrequently. This is interesting and tells us that the braid relations seem far more important than Markov move 1 when it comes to simplifying our braids. Note that these increase the braid length and with it the punishment, and hence the agent is also reluctant to perform these. The fact that the agent performs them nevertheless (and even more frequently than shift rights, which do not change the punishment) illustrates that the agent is indeed maximizing its long-term return and the fact that some braids need to become longer before they can be simplified as illustrated in an example in section 5.1.