Ballistic Action Planning in Robotics by means of Artificial Imagery

This abstract aims to present our recent work on exploring the concept of mental imagery and mental simulation as a fundamental cognitive capability applied to robot controllers, with the aim of improving the motor performance of the robot in terms of motor control and multi-degrees of freedom coordination. Indeed we believe that mental imagery models can give the opportunity to apply such behaviour toward the development of artificial cognitive systems, in order to improve robots’ motor performance in general and in complex motor planning. This objective can be achieved using bio-inspired computational modelling technologies, such as artificial recurrent neural networks, able to emulate processes of mental training by mental simulation. In particular, as proof-of-concept, we designed a dual neural network architecture, that allows the iCub to improve autonomously its sensorimotor skills, with techniques inspired by the ones that are employed with human subjects in sports training. This is achieved by endowing a feedforward controller of a secondary recurrent neural system that, by exploiting the sensorimotor skills already acquired by the robot, is able to generate additional imaginary examples that can be used by the controller itself to improve the performance through a additional learning process. Moreover we show that data obtained with artificial imagination could be used to simulate mental training to learn new tasks and enhance their performance. Results of experimental tests in controlling a ballistic movement with the simulator of the iCub humanoid robot platform are presented as evidence of the opportunities presented by the use of artificial mental imagery in cognitive robotics.


Introduction
This abstract aims to present our recent work on exploring the concept of mental imagery and mental simulation as a fundamental cognitive capability applied to robot controllers, with the aim of improving the motor performance of the robot in terms of motor control and multi-degrees of freedom coordination.Indeed we believe that mental imagery models can give the opportunity to apply such behaviour toward the development of artificial cognitive systems, in order to improve robots' motor performance in general and in complex motor planning.This objective can be achieved using bio-inspired computational modelling technologies, such as artificial recurrent neural networks, able to emulate processes of mental training by mental simulation.
In particular, as proof-of-concept, we designed a dual neural network architecture, that allows the iCub to improve autonomously its sensorimotor skills, with techniques inspired by the ones that are employed with human subjects in sports training.This is achieved by endowing a feedforward controller of a secondary recurrent neural system that, by exploiting the sensorimotor skills already acquired by the robot, is able to generate additional imaginary examples that can be used by the controller itself to improve the performance through a additional learning process.Moreover we show that data obtained with artificial imagination could be used to simulate mental training to learn new tasks and enhance their performance.Results of experimental tests in controlling a ballistic movement with the simulator of the iCub humanoid robot platform are presented as evidence of the opportunities presented by the use of artificial mental imagery in cognitive robotics.

Material and Methods
The neural system that controls the robot is represented in Figure 1(a), that consists of a three layer feedforward network (FFNN) that implements the actual motor controller, and of a Recurrent Neural Network (RNN).The RNN models the motor imagery and it is represented in detail in yaw, and hand wrist pitch are the proprioceptive information for input and output neurons.Another neuron is the grab/release command, respectively with value 1 and 0. All values are normalized in the range [0,1].We used a classic back-propagation algorithm as the learning process.The learning phase lasted 10000 epochs with a learning rate of 0.2 without momentum.The experimental task is shown in Figure 2(a) and it is the realisation of a ballistic action, involving the simultaneous movement of the right arm and of the torso.It should be noted here that, since ballistic movements are by definition not affected by external interferences, the training can be performed without considering the surrounding environment, as well as vision and auditory information.The task of the robot is to throw a small cube of side size 2 cm and weight 40 grams as far as possible according to an externally given velocity for the movement.
The robotic model used for the experiments presented here is a simulation of the iCub humanoid robot, that was developed with the aim to accurately reproduce the physics and the dynamics of the physical iCub using a software library that provides an accurate simulation of rigid body dynamics and collisions.In the first phase the RNN was trained by a simple heuristic to predict its own subsequent sensorimotor state.To this end joint angle information over time was sampled in order to build 20 input-output sequences corresponding to different directions of the movement.In addition, in order to model the autonomous throw of an object, the primitive action to grab/release was also considered in the motor information fed to the network.In the second phase the RNN operates in offline mode and, thus, its prediction are made according only to the internal model built during the training phase.

Experimental results and Discussion
In our experiments we tested the impact of mental training in action performance in a different speed range that was not experienced before.To this end we split both the learning and testing dataset into two subsets according to the duration of the movement: fast range subset comprises examples that last less than 0.3 seconds; slow range subset comprises all the others.Figure 2(b) shows the performance of the two nets as controller of the ballistic task.As expected the FFNN is the best controller for the task if the full range is given as training, thus, it is the ideal controller for the task.
To test the mental training, we compared results on three different case studies: (a) full range: For benchmarking purposes, it is the performance obtained by the FFNN when it is trained using the full range of examples (slow + f ast); (b) slow range only training: The performance obtained by the FFNN only when it is trained using only the slow range subset.This case stressed the generalization capability of the controller when it is tested with the f ast range subset; (c) slow range plus mental training: In this case the two architectures operate together as a single hierarchical architecture, in which first both nets are trained with the slow range subset, then the RNN runs in mental imagery mode to build a new dataset of f ast examples for the FFNN, that is incrementally trained this way.
Results show that generalization capability of the RNN helps to feed the FFNN with new data to cover the f ast range, simulating mental training.In fact, the FFNN, trained only with the slow subset is not able to foresee the trend of duration in the f ast range, this implies that f ast movements last longer than needed and, because the inclination angle is over 90 degrees, the object falls backward (see Figure 3).The FFNN failure in predicting temporal dynamics is explainable by the simplistic information used to train the FFNN, which seems to be not enough to reliably predict the duration time in a faster range, never experienced before.On the contrary, the greater amount of information that comes from the proprioception and the fact that the RNN has to integrate over time those information in order to perform the movement, makes the RNN able to create a sort of internal model of the robot's body behavior.This allows the RNN to better generalize and, therefore guide the FFNN in enhancing its performance.

Conclusion
The results presented in this work, in conclusion, allow to imagine the creation of novel algorithms and cognitive systems that implement even better and with more efficacy the concept of artificial mental training.Such a concept appears very useful in robotics, for at least two reasons: it helps to speed-up the learning process in terms of time resources by reducing the number of real examples and real movements performed by the robot.An interesting direction for future work is the integration of the artificial imagery with reinforcement learning techniques, with the aim to improve the learning phase replacing real actions with mental simulations.

Figure 1 :
Figure 1: Artificial Neural Networks: (a) The Dual Network Architecture (FFNN + RNN).(b) Detail of RNN: Red connections are active in imagery mode only, while green connections are deactivated in imagery.

Figure 2 ECALFigure 2 :
Figure 2: Action phases and performance comparison (by means of duration and release point errors) of FFNN and RNN as controllers with full range training

Figure 3 :
Figure 3: Results: Distance reached by the object after throwing movements of varying velocities.Negative values represents the objects falling backward;