Author: [Bruce Zhou]
Affiliation: [Jota Machinery Composites Material Prepreg Solution]
Corresponding Author: [jotamachinery@gmail.com]
Published : November 28 , 2025

Abstract

Automated tape laying (ATL) of thermoplastic unidirectional (UD) tapes is highly sensitive to temperature fluctuations at the nip point, where tape, substrate and consolidation roller meet. Classical open-loop heater settings and model-based controllers often struggle to compensate for environmental drift, tool heating and changing boundary conditions without extensive physical modelling or additional cooling systems. This work reviews and extends the findings of Römer et al. (2022), who implemented a Deep Deterministic Policy Gradient (DDPG) reinforcement learning (RL) controller on a real infrared-heated ATL head for glass-fiber/PA6 tapes. Using only 597 layup episodes (239 m of tape), the agent learned to adjust heater power in real time and keep the nip-point temperature within a ±10 °C window around 250 °C, significantly improving stability over open-loop control and eliminating overheating events after mid-training. We summarise the methodology, training strategy and results, then discuss implications for variant-flexible electromobility production and future AFP/ATL systems. Finally, we position RL-based control as a pragmatic, model-free path toward adaptive, AI-supported composites manufacturing, while highlighting current limitations in operating envelope, benchmarking and scalability.

Keywords

Automated tape laying; thermoplastic composites; infrared heating; reinforcement learning; DDPG; nip-point temperature control; electromobility; process automation

1. Introduction

Lightweight continuous-fiber thermoplastic composites are becoming a core technology for electromobility and aerospace, where weight directly translates into energy consumption, range and emissions. Automated tape laying (ATL) with thermoplastic UD tapes enables fibre orientations aligned with load paths and variant-flexible preforms on reconfigurable tools. However, process temperature at the nip point remains a critical bottleneck: even small deviations from the optimal window can cause incomplete consolidation, voids, poor bonding or thermal damage.

Infrared (IR) heating is attractive for ATL because it is contactless and relatively simple to integrate, but it introduces strong thermal inertia and environmental sensitivity. The IR field heats not only the tape but also the substrate, consolidation roller, tool and surrounding hardware. Over a production run, the entire thermal environment drifts, making fixed open-loop heater settings unreliable. Conventional closed-loop approaches, such as model predictive control (MPC) or finely tuned PID regulators, require either detailed thermal models or frequent re-tuning whenever materials, geometries or conditions change.

Reinforcement learning (RL) offers a model-free alternative: instead of deriving the control law from a physics model, the controller learns directly from interaction with the real process. The paper “Temperature Control for Automated Tape Laying with Infrared Heaters Based on Reinforcement Learning” by Römer et al. (Machines, 2022) is a focused demonstration of this concept on a working ATL head. This new paper builds on their work, synthesising the key ideas, evaluating the methodology in an engineering context, and discussing how similar strategies can be embedded in practical AFP/ATL and prepreg processing lines.

2. Literature Review

2.1 Temperature control in ATL and AFP

In both ATL and automated fibre placement (AFP), nip-point temperature and pressure dominate consolidation quality for thermoplastic composites. Prior work has shown that:

• The tape must be heated into a defined process window above melt or glass-transition temperature to achieve intimate contact and squeeze out voids.
• Overheating risks resin degradation, fibre imprinting and surface burning; underheating leads to weak bonding and interlaminar defects.

Traditional control strategies fall into three categories:

1. Open-loop maps: heater power or laser output is pre-tabulated as a function of tape type, layup speed, angle and geometry. These maps are usually built from extensive trials or simplified thermal models but cannot cope well with long-term drift or environment changes.
2. Model-based control (e.g. MPC): detailed thermal models plus state observers (Kalman filters) are used to predict interface temperatures and adjust actuator inputs. This can be effective but demands significant modelling effort and re-validation when the hardware or process changes.
3. PID and multi-loop feedback: temperature sensors (pyrometers, thermocouples) are used to regulate heater power with simple proportional-integral-derivative loops. Tuning is non-trivial because of strong delays, nonlinearities and coupling between heater, tool and environment.

Recent research has begun to explore data-driven approaches, including supervised machine learning to predict process parameters and RL for building temperature and heater control. The work by Römer et al. is one of the first to apply deep RL directly to ATL process temperature on real hardware, not just in simulation.

2.2 Reinforcement learning for thermal processes

RL has been used in building HVAC control, water heating and process furnace optimisation. The Deep Deterministic Policy Gradient (DDPG) algorithm is particularly suited to continuous-state, continuous-action problems, where the controller must output a real-valued actuator signal (e.g. heater power) from continuous sensor readings (e.g. temperatures).

Key advantages:

• Model-free: no explicit thermal model required.
• Continuous adaptation: in principle, the policy can keep learning as the environment changes.
• Multi-input, multi-output potential: RL can, in theory, handle several coupled temperatures and actuators.

However, RL is often criticised for data hunger and instability. Demonstrating useful learning with limited data on a production-scale process is therefore important for industrial acceptance.

3. Methodology

3.1 ATL system and process conditions

The study uses an ATL end-effector mounted on a KUKA KR 60 robot. The head processes glass-fiber/PA6 UD tape, 200 mm wide and 0.22 mm thick. The heating system consists of:

• An inner IR module (1.8 kW) used for pre-heating and tape bending, operated at fixed 35 % power.
• An outer IR module (3 kW) that directly influences the nip-point region; this module is the single controlled actuator.

Three pyrometers measure:

• Nip-point temperature, TPT_PTP​ – primary process variable, target 250 °C with ±10 °C tolerance.
• Roll temperature, TRT_RTR​ – reflects roller heating and possible heat sink effects.
• Substrate/tool temperature, TST_STS​ – tracks long-term heating of the mould and fixture.

To reduce noise, raw temperature readings are passed through a moving average filter (25 samples). Layup speed is fixed at 20 mm/s on a flat steel substrate.

3.2 RL formulation: states, actions and reward

The control problem is formulated as a Markov Decision Process:

• State vector

• Action

the normalised power setpoint for the outer IR module.

Because the IR heater and the thermal field exhibit significant delay, each action is followed by a dead time of 3 s before the next state is sampled, giving the system time to respond.

The reward function encodes the temperature control objective:

• A maximum reward of 1 is given at the target temperature TP=250°CT_P = 250 °CTP​=250°C.
• A tolerance band of ±10 °C defines an acceptable window; reward decreases linearly with distance from 250 °C inside this window.
• Temperatures below 240 °C or above 260 °C incur linearly increasing penalties.
• Exceeding 300 °C triggers a severe penalty (−10) and early episode termination, reflecting material-damage risk.

This piecewise linear reward creates a smooth gradient near the target while strongly discouraging overheating.

3.3 DDPG controller architecture and training

The RL agent uses the DDPG algorithm with separate actor and critic neural networks:

• Actor network:
  • Input: 3-element state vector.
  • Hidden layers: 400 and 300 neurons with ReLU activations.
  • Output: single action in [0,1] (heater power fraction) via Sigmoid activation.
• Critic network:
  • Input: concatenated state and action vector (4 values).
  • Hidden layers: 400 and 300 neurons with ReLU activations.
  • Output: scalar Q-value.

Target networks and an experience replay buffer are used for stability; mini-batches of past transitions are sampled for each update.

To balance exploration and exploitation, the authors combine:

• Ornstein–Uhlenbeck noise added to actions to encourage local exploration in continuous space.
• An “expert exploration” policy based on an existing open-loop heater setting (60 % power). Early in training, actions are chosen with a high probability from this expert policy or from random exploration; over time, the probability shifts toward the DDPG policy as its performance improves.

Training is conducted directly on the real ATL cell:

• Each episode consists of a 400 mm layup at 20 mm/s.
• About 5–6 control decisions are made per episode due to the 3 s dead time.
• In total, 597 episodes (239 m of tape) are used for training.

4. Results

4.1 Learning progress

The primary learning metrics are:

• Normalised episode reward (average reward per time step, scaled by theoretical maximum).
• Fraction of time steps where the next-state nip-point temperature lies within 240–260 °C.

Over the course of training:

• The moving average reward rises from low, volatile values to approximately 29.5 % of the maximum possible reward in later episodes.
• The frequency of overheating events (TP≥300°CT_P ≥ 300 °CTP​≥300°C) drops to zero after about mid-training, indicating that the agent has learned to avoid dangerous heater commands.
• The proportion of temperature transitions remaining within the ±10 °C tolerance window steadily increases; several late episodes show nip-point temperatures staying almost entirely within 240–260 °C.

A visual comparison of nip-point temperature profiles before and after training shows that:

• Untrained control (open-loop baseline) exhibits significant drift and overshoot, with some layups crossing from under- to over-temperature conditions within a single pass.
• Trained RL control produces smoother, more consistent trajectories clustered around the target window, even as the substrate and roller slowly heat up.

4.2 Data efficiency

Although deep RL is often associated with large data requirements, this study demonstrates useful performance gains with roughly 3,000–3,500 distinct state–action–reward transitions. This is modest by RL standards and suggests that, for relatively low-dimensional control problems with clear reward structures, real-process learning can be data-efficient if prior expert settings are embedded into the exploration policy.

5. Discussion

5.1 Technical strengths

The work demonstrates several important points for industrial ATL/AFP control:

1. Feasibility on real hardware
   The RL agent is trained entirely on a working ATL head rather than in a simulator. This avoids the “sim-to-real” transfer issues that often limit RL applications in manufacturing.
2. Model-free adaptation
   No detailed thermal model is required. The controller implicitly learns the delayed, nonlinear relationship between heater power and nip temperature in the presence of tool and environment heating.
3. Built-in safety behaviour
   Through the reward function and penalties, the agent learns a strong aversion to overheating. After the early training phase, damaging temperature peaks no longer occur.
4. Use of prior domain knowledge
   Incorporating an existing open-loop heater setting as an “expert policy” reduces waste and accelerates learning—an approach that can be replicated easily on other lines.

5.2 Limitations

At the same time, several constraints limit immediate industrial adoption:

• The operating envelope is narrow: single tape type, single layup speed, flat steel substrate and one fixed target temperature. Real production involves changing materials, thicknesses, geometries and speeds.
• Only a single actuator (outer IR module) is controlled. In practice, multi-zone heaters, laser heads, or speed adjustments may be needed for full control.
• There is no quantitative comparison against a well-tuned PID or MPC controller in terms of tracking error, energy consumption or robustness. Without such benchmarks, RL’s relative benefit remains partly qualitative.
• Training data are skewed toward “hot” scenarios; the authors note that performance in certain cold regimes is less well explored.

5.3 Industrial implications

For high-rate, variant-flexible production of thermoplastic composite preforms, especially for electromobility and aerospace, this approach offers:

• Reduced need for painstaking model building and controller re-tuning when tools, materials or environments change.
• A pathway toward self-adapting heater control that can compensate for substrate heating and seasonal ambient variations.
• The possibility of extending RL to multi-input control, including multiple heaters and even speed or pressure modulation, once the single-actuator case is well-established.

Subsequent research has begun to cite this work in multi-agent RL for thermoforming and heater design optimisation, suggesting growing interest in similar strategies for other thermal processes.

6. Conclusion

This paper synthesises and evaluates a reinforcement learning–based temperature control strategy for infrared-heated automated tape laying of thermoplastic UD tapes. Building on the work of Römer et al., we highlight how a DDPG agent, trained on a few hundred metres of tape, can learn to stabilise nip-point temperature within a practical tolerance band, avoid overheating and adapt to environmental drift—without an explicit thermal model or added cooling systems.

From an engineering perspective, RL is not a universal replacement for classical control but a pragmatic complement in scenarios where:

• Accurate models are difficult or expensive to obtain,
• Operating conditions vary widely over time, and
• Safety limits can be encoded clearly into a reward structure.

Future work should extend the approach to multi-zone heating, variable layup speeds and broader material sets, and provide rigorous comparisons against tuned PID or MPC schemes. Nonetheless, the demonstrated feasibility on real ATL hardware makes RL-based control a credible tool for next-generation AFP/ATL and prepreg processing lines.

References

1. Römer, M.; Bergers, J.; Gabriel, F.; Dröder, K. Temperature Control for Automated Tape Laying with Infrared Heaters Based on Reinforcement Learning. Machines 2022, 10, 164.
2. Standard ATL/AFP and thermoplastic consolidation literature (e.g., process window and nip-point temperature studies) as cited in Römer et al.
3. Foundational works on DDPG and RL-based temperature control in related thermal applications (HVAC, water heaters, thermoforming), which corroborate the generalisability of the approach.