A new approach to creating language generation algorithms

"Currently, most language generation algorithms are based on deterministic neural networks, resulting in word sequence decoding mechanisms that produce either degenerate or simplified and overly generic language." This is the problem that guides the work of Alice Martin, a young researcher in the 'Next-Gen RetAIl' Chair, supported by Éric Moulines, a researcher at the Centre de mathématiques appliquées (CMAP*).

Indeed, classical text generation algorithms (such as chatbots, visual caption generation algorithms or text synthesis) are trained by supervised learning. This method involves providing the algorithm with a dataset where both the input and output values are specified. However, supervised datasets have to be constructed by hand, which is a costly and time-consuming process. In addition, the use of supervised learning often leads to algorithms that reproduce the biases present in the data.

From uncertainty comes flexibility
In order to solve these problems, the young researcher first created a stochastic neural network (i.e. incorporating a degree of randomness) for sequential data, called the "Monte Carlo Transformer". It incorporates randomness into its operation, thus making it possible to model a degree of uncertainty in its predictions. The objective behind this addition is that the output data is not a single value but a distribution of possible values, from which a margin of error is calculated. To train the neural network and estimate this distribution of predictions, Alice Martin used sequential Monte Carlo methods that are integrated into the neural network training algorithm.

Thus constructed, the transformer is capable of restoring a sequence of data accompanied by a distribution of the values that they can take. The integration of uncertainty allows, for example, in the context of natural language processing, a translation algorithm to assign a confidence value to the words it translates, and to propose a range of translations rather than a single word. It is similarly applicable to time series forecasting problems where it allows the modelling of randomness in the input data (measurement errors, etc).

One algorithm to guide another
Once linguistic flexibility was taken into account, Alice Martin sought to train text generation algorithms by reinforcement learning in order to dispense with supervised datasets, and to solve some of the language degeneration problems. This method allows the algorithm to explore a space of actions (in this case the words of a vocabulary) in an environment where rules are imposed on it. Each action of the algorithm, called an agent in this process, is associated with a "reward" value that tells it whether its action has brought it closer to its goal or not. A very similar method has been used, for example, to train algorithms for chess games.

However, a language model has at least 10,000 vocabulary words that the agent can randomly put together when learning. Efficient exploration in such a space of actions becomes very difficult, and the chances of an intelligible sentence being created in this way are very low. For this reason, the young researcher has built a reinforcement learning algorithm to 'guide' the agent's learning. This is based on the past word sequence already spoken and uses the structure of the language to reduce the number of possible words that can follow in the action space. With the possibilities drastically reduced, reinforcement learning can take place.

At present, Alice Martin is working on integrating the previously created transformer on pre-trained language models on multiple massive text databases, such as the famous "GPT-2" and "GPT-3" of OpenAI. This latest work aims to propose a universal text generation method (which can be used in supervised and reinforcement learning) that builds on the strengths of existing language systems and improves the diversity and richness of the language produced.

*CMAP: a joint research unit of CNRS, École Polytechnique - Institut Polytechnique de Paris

>> about the Chair:
Led since 2019 by Eric Moulines, the "Next-Gen RetAIl" Chair aims to encourage teaching and research in Artificial Intelligence and Data Science, and to develop algorithms that can be applied to mass retail. Supported by Carrefour and in interaction with its Digital HUB, the Chair is in line with these themes through its research and teaching, particularly on conversational AI and product recommendation systems.