SHARP: Generating Synthesizable Molecules via Fragment-Based Hierarchical Action-Space Reinforcement Learning for Pareto Optimization

Abstract

Designing drug-like molecules that satisfy multiple objectives-such as high binding affinity, synthesizability, and drug-likeness-poses a complex global optimization problem over an astronomically large chemical space. Existing deep learning-based molecular generative models often treat this task as distribution modeling, relying on atom-level autoregressive actions with less consideration of explicit optimization feedback. Consequently, they frequently generate invalid structures, converge to local optima, or produce synthetically infeasible candidates. Here, we introduce Synthesizable Hierarchical Action-space Reinforcement learning for Pareto optimization (SHARP), a molecular generator that addresses these limitations via a fragment-based hierarchical action space and reinforcement learning. SHARP ensures synthetic accessibility by applying action masks guided by a pretrained Synthesizability Estimation Model (SEM). The reinforcement learning (RL) policy is trained using a composite reward function integrating docking scores, pharmacophore matching, and solvent accessibility to generate functionally relevant and experimentally tractable molecules. Furthermore, across four lead optimization tasks-fragment growing, linker design, scaffold hopping, and side chain decoration-on a diverse receptor set, SHARP consistently outperforms prior methods in producing molecules at high affinity with reasonable synthesizability. These results demonstrate that reinforcement learning with a chemically intuitive action space design can be an effective solution to the optimization challenges in AI-driven drug discovery, offering a robust framework for rational molecular design in structure-based applications.