Reactive navigation under a fuzzy rules-based scheme and reinforcement learning for mobile robots

The artificial potential field method

The APF method provides a simple and effective motion planning method for a practical purpose. Under this method, the attractive force is computed according to Eq. (1): (1)${\displaystyle F_{attr}\left(q,g\right)=-\xi p\left(q,g\right)}$

where p(q, g) is the euclidean distance between the robot’s position, denoted by q, and the destination point’s position g. The term ξ is defined as the attractive factor. The repulsive force, on the other hand, is calculated with the aid of Eq. (2). (2)${\displaystyle F_{rep}\left(q\right)=\left\{\begin{array}{cc}\eta \left(\frac{1}{p\left(q,q_{o_j}\right)}-\frac{1}{p_{o_j}}\right)\frac{p^2\left(q,q_{o_j}\right)}{p\left(q,q_{o_j}\right)},\hfill & \text{if}p\left(q,q_{o_j}\right)<p_{o_j}\hfill \\ \eta \left(-\frac{1}{p_{o_j}}\right),\hfill & \text{if}p\left(q,q_{o_j}\right)=p_{o_j}\hfill \\ 0,\hfill & \text{if}\text{other}\text{case}\hfill \end{array}\right.}$

where p(q, q_oj) is the distance between the robot’s position and the jth obstacle’s position q_oj. The obstacle radius threshold is denoted by p_oj. The term η is the repulsive factor. Furthermore, the resultant force Eq. (3) is the sum of attractive and repulsive forces. (3)${\displaystyle F_{res}=F_{attr}\left(q,g\right)+F_{rep}\left(q\right)}$

Classical reinforcement learning methods

Reinforcement learning (RL) is a machine learning paradigm, where the main identified elements are the agents, the states, the actions, the rewards, and punishments. In general, RL problems involve learning what to do and how to map situations to actions to maximize a numerical reward signal when the learning agent does not know what steps to take. The agent involved in the learning process must discover which actions give the best reward based on a trial and error process (Sutton & Barto, 2018). Essentially, RL involves closed-loop problems because the learning system’s actions influence its later inputs. Different methods use this paradigm; among the classic methods are QL and SARSA. QL provides learning agents with the ability to act optimally in a Markovian domain experiencing the consequences of their actions, i. e., the agents learn through rewards and punishments (Sutton & Barto, 2018). QL is a method based on estimating the value of the Q function of the states and actions, and it can be defined as shown in Eq. (4), where Q(S_t, A_t) is the expected sum of the discounted reward for the performance of action a in a state s, α is the learning rate, γ is the discounting factor, S_t is the current state, S_t+1 is the new state, R_t+1 is the reward in the new state, and A_t is the action in S_t, and a is the action with the best q-value in S_t. (4)${\displaystyle Q\left(S_t,A_t\right)\leftarrow Q\left(S_t,A_t\right)+\alpha \left[R_{t+1}+\gamma \underset{a}{max}Q\left(S_{t+1},a\right)-Q\left(S_t,A_t\right)\right]}$

The method called SARSA results from a modification made to QL. The main difference between these algorithms is that SARSA does not use the max operator from the Q function during the rule update, as shown in Eq. (5) (Sutton & Barto, 2018). (5)${\displaystyle Q\left(S_t,A_t\right)\leftarrow Q\left(S_t,A_t\right)+\alpha \left[R_{t+1}+\gamma Q\left(S_{t+1},A_{t+1}\right)-Q\left(S_t,A_t\right)\right]}$

Fuzzy Q-learning method

FQL (Anam et al., 2009) is an extension of fuzzy inference systems (FIS) (Ross, 2010), where the fuzzy rules define the learning agent’s states. At the first step of the process, crisp input variables are converted into fuzzy inputs through input membership functions. Next, rule evaluation is conducted, where any convenient fuzzy t-norm can be used at this stage, a common choice is the MIN operator. Only one fuzzy rule, i.e., the ith fuzzy rule, r_i, determines the learning agent’s state, and i ∈ [1, N], N denotes the total number of fuzzy rules. Each rule has associated a numerical value, α_i, called the rule’s strength, defining the degree to which the agent is in a certain state. This value α_i allows the agent to choose an action from the set of all possible actions A, the jth possible action in the ith rule is called a(i, j), and its corresponding q-value is q(i, j). Therefore, fuzzy rules can be formulated as:

If x is S_i then a(i, 1) with q(i, 1) or ... or a(i, j) with q(i, j)

FQL’s primary goal implies the learning agent finds the best solution for each rule, i.e., the action with the higher q-value. This value is rendered from a table of i × j dimensions, containing q number of values. The table dimension corresponds to the number of fuzzy rules times the total number of actions. A learning policy selects the right action based on the quality of a state-action pair, based on the Eq. (6), where V(x, a) is the quality value of the states, i^∗ corresponds to the optimal action index, i.e., the action index with the highest q-value, and x is the current state. On the other hand, the exploration-exploitation probability is given by $\epsilon =\frac{10}{10+T}$, where T corresponds to the step number, is assumed in this work. (6)${\displaystyle V\left(x,a\right)=\frac{\sum _{i=1}^N{\alpha }_i\left(x\right)\times q\left(i,i^{\ast }\right)}{\sum _{i=1}^N{\alpha }_i\left(x\right)}}$

The functions in Eq. (7) allow to get the inferred action and its corresponding q-value, where x is the input value in state i, a is the inferred action, i^o is the inferred action index, α_i is the strength of the rule and N is a positive number, N ∈ ℕ⁺, which corresponds to the total number of rules. (7)${\displaystyle a\left(x\right)=\frac{\sum _{i=1}^N{\alpha }_i\times a\left(i,i^o\right)}{\sum _{i=1}^N{\alpha }_i\left(x\right)};Q\left(x,a\right)=\frac{\sum _{i=1}^N{\alpha }_i\times q\left(i,i^o\right)}{\sum _{i=1}^N{\alpha }_i\left(x\right)}}$

Also, it is necessary to calculate an eligibility value, e(i, j), during the q-value updating phase by means of Eq. (8), (8)${\displaystyle e\left(i,j\right)=\left\{\begin{array}{cc}\lambda \gamma e\left(i,j\right)+\frac{{\alpha }_i\left(x\right)}{\sum _{i=1}^N{\alpha }_i\left(x\right)}\hfill & \text{if}j=i^o\hfill \\ \lambda \gamma e\left(i,j\right)\hfill & \text{other case}\hfill \end{array}\right.}$

where j is the selected action, γ is the discount factor 0 ≤ γ ≤ 1 and λ is the decay parameter in the range of [0,1). Next, the Eq. (9) is used for the ΔQ calculation, where r corresponds to the reward. (9)${\displaystyle \Delta Q=r+\gamma V\left(x,a\right)-Q\left(x,a\right).}$

And finally, Eq. (10) updates q-value, where ϵ is a small number, ϵ ∈ (0, 1), which affect the learning rate, (10)${\displaystyle \Delta q\left(i,j\right)=ϵ\times \Delta Q\times e\left(i,j\right).}$