Optimization research using deep reinforcement learning has been actively conducted in the ship design field recently, but existing studies have been limited in accurately reflecting complex flow characteristics as they mainly use simple resistance estimation models based on empirical equations. To overcome this, this study proposes a novel hull form optimization architecture that simulates the expert design process by combining a Proximal Policy Optimization algorithm and a U-Net-based high-precision surrogate model. This framework effectively controls the degree of freedom of hull forms by defining the latent space of the generative model as an action space of reinforcement learning. In the environmental configuration stage, the pressure and shear stress distribution information on the hull surface are trained and then predicted by U-Net as well as simple resistance values. This can drastically reduce the computational cost of numerical analysis while providing a high-fidelity evaluation mechanism close to the conventional process in which the designer directly judges the physical distribution of the computational fluid dynamics analysis results and modifies the hull forms.