On the amount of nonconstructivity in learning formal languages from text

https://doi.org/10.1016/j.ic.2020.104668Get rights and content

Abstract

Nonconstructive computations by various types of machines and automata have been considered by, for example, Karp and Lipton as well as Freivalds. They allow to regard more complicated algorithms from the viewpoint of much more primitive computational devices. The amount of nonconstructivity is a quantitative characterization of the distance between types of computational devices with respect to solving a specific problem.

This paper studies the amount of nonconstructivity needed to learn classes of formal languages. Different learning types are compared with respect to the amount of nonconstructivity needed to learn indexable classes and recursively enumerable classes, respectively, of formal languages from positive data. Matching upper and lower bounds for the amount of nonconstructivity needed are shown.

Introduction

For millenia mathematicians have been aware of the method to show the existence of a mathematical object by constructing it, and the nonconstructive method of proving that it cannot fail to exist. On the other hand, before the end of the 19th century constructive proofs have been the dominant ones. And in the forties of the last century nonconstructive methods found their way to discrete mathematics (cf., for example, Erdős [8]).

If it is known that a mathematical object exists, one can also ask what does it take to construct such an object. In this context, the notion of complexity as well as the distinction between uniform and non-uniform models of computations comes into play. Allowing a uniform model of computation to use an arbitrary string as additional input converts it into a non-uniform model. Such strings are often called advice. An influential paper in this regard was Bārzdiņš [3], who introduced the notion of advice in the setting of Kolmogorov complexity of recursively enumerable sets.

Further examples comprise Karp and Lipton [21]. They studied the problem under what circumstances non-uniform upper bounds can be used to obtain uniform upper bounds. To achieve this goal the notion of a Turing machine that can take advice has been coined. Furthermore, Damm and Holzer [7] adapted the notion of advice to finite automata, and Cook and Krajiček [6] initiated the study of proof systems that use advice for the verification of proofs. Even more recently, Beyersdorff et al. [4] continued along this line of research.

Quite often, we experience that finding a proof for a new deep theorem is triggered by a certain amount of inspiration. Being inspired does not mean that we do not have to work hard in order to complete the proof and to elaborate all the technical details. However, this work is quite different from enumerating all possible proofs until we have found the one sought for. Also, as experience shows, the more complicated the proof, the higher is the amount of inspiration needed. These observations motivated Freivalds [9], [10] to introduce a qualitative approach to measure the amount of nonconstructivity (or advice) in a proof. Analyzing three examples of nonconstructive proofs led him to a notion of nonconstructive computation which can be used for many types of automata and machines and which essentially coincides with Karp and Lipton's [21] notion when applied to Turing machines.

As outlined by Freivalds [9], [10], there are several results in the theory of inductive inference of recursive functions which suggest that the notion of nonconstructivity may be worth a deeper study in this setting, too. Subsequently, Freivalds and Zeugmann [11] introduced a model to study the amount of nonconstructivity needed to learn recursive functions.

In the present paper we generalize the model of Freivalds and Zeugmann [11] to the inductive inference of formal languages. That is, we aim to characterize the difficulty to learn classes of formal languages from positive data by using the amount of nonconstructivity needed to learn these classes. We shortly describe this model. The learner receives, as usual, growing initial segments of a text for the target language L, where a text is any infinite sequence of strings and a special pause symbol # such that the range of the text minus the pause symbol contains all elements of L and nothing else. In addition, the learner receives as a second input a bitstring of finite length which we call help-word. Given a correct / appropriate help-word, the learner can learn in the desired sense. Since there are infinitely many languages to learn, a parameterization is necessary, that is, we allow for every n a possibly different help-word and we require the learner to learn every language contained in {L0,,Ln} with respect to the hypothesis space (Li)iN chosen (cf. Definition 2.7). The difficulty of the learning problem is then measured by the length of the help-words needed, that is, in terms of the growth rate of the function d bounding this length. As in previous approaches, the help-word does not just provide an answer to the learning problem. There is still much work to be done by the learner.

First, we consider the learnability of indexable classes in the limit from positive data and ask for the amount of nonconstructivity needed to learn them. This is quite a natural choice, since even simple indexable subclasses of the class of all regular languages are known not to be inferable in the limit from positive data [13], [15], [28]. Second we investigate the amount of nonconstructivity needed to infer recursively enumerable classes of recursively enumerable languages. Moreover, several variations of Gold's [13] model of learning in the limit have been considered, see [15], [25] and the references therein. Thus, it is only natural to consider some of these variations, too. In particular, we shall study conservative as well as strong-monotonic learning.

We prove upper and lower bounds for the amount of nonconstructivity in learning classes of formal languages from positive data. The usefulness of this approach is nicely reflected by our results which show that the function d may considerably vary. In particular, the function d may be arbitrarily slow-growing for learning indexable classes in the limit from positive data (cf. Theorem 3.1), while we have an upper bound of logn and a lower bound of logn2 for conservative learning of indexable classes from positive data (cf. Theorem 3.2, Theorem 3.3). Furthermore, we have a 2logn upper bound and a 2logn4 lower bound for strong-monotonic inference of indexable classes from positive data (cf. Theorem 3.4, Theorem 3.5).

Moreover, the situation changes considerably when looking at recursively enumerable classes of recursively enumerable languages. For learning in the limit from positive data we have an upper bound of logn and a lower bound of logn2, while for conservative learning even any limiting recursive bound on the growth of the function d is not sufficient to learn all recursively enumerable classes of recursively enumerable languages from positive data (cf. Theorem 3.7, Theorem 3.9, Theorem 3.11).

Section snippets

Preliminaries

Any unspecified notation follows Rogers [26]. In addition to or in contrast with [26] we use the following. By N={0,1,2,} we denote the set of all natural numbers, and we set N+=N{0}.

The cardinality of a set S is denoted by |S|. We write (S) for the power set of the set S. Let ,,,, ⊃, and ⊇ denote the empty set, element of, proper subset, subset, proper superset, and superset, respectively. Let S1,S2 be any sets. By S1S2 we denote the difference of sets, and we write S1S2 to denote the

Results

Already Gold [13] showed that REGLimTxt and as mentioned in (1), even quite simple subclasses of REG are not in LimTxt. So, we start our investigations by asking for the amount of nonconstructivity needed to identify any indexable class in the limit from text with respect to any indexed hypothesis space H.

Conclusions

We have presented a model for the inductive inference of formal languages from text that incorporates a certain amount of nonconstructivity. In our model, the amount of nonconstructivity needed to solve the learning problems considered has been used as a quantitative characterization of their difficulty.

We studied the problem of learning indexable classes under three postulates, that is, learning in the limit, conservative identification, and strong-monotonic inference. As far as learning in

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

We thank the anonymous referee for several helpful comments which improved the presentation of the paper. This research was performed partially while the third author was visiting the Institute of Mathematical Sciences at the National University of Singapore in September 2011. His visit was supported by the Institute. Preliminary version of this paper appeared in TAMC 2012 [19].

References (28)

  • Stephen Cook et al.

    Consequences of the provability of NPP/poly

    J. Symb. Log.

    (2007)
  • Carsten Damm et al.

    Automata that take advice

  • Paul Erdős

    Some remarks on the theory of graphs

    Bull. Am. Math. Soc.

    (1947)
  • Rūsiņš Freivalds

    Amount of nonconstructivity in finite automata

  • Cited by (0)

    1

    This research has been supported by Ministry of Education - Singapore Academic Research Fund grants R252-000-420-112, R146-000-234-112 (MOE2016-T2-1-019) and R146-000-304-112 (MOE2019-T2-2-121) to Frank Stephan (PI) and Sanjay Jain (Co-PI); furthermore, Sanjay Jain is supported in part by NUS grant C252-000-087-001.

    View full text