The prediction of the dynamical response of vessels in large amplitude random waves is an important consideration in the design and operation of both manned and autonomous ships. Performance evaluation of each new design typically employs tools with varying levels of fidelity. Low-fidelity linear seakeeping models typically operate in the frequency domain or with simplified physics in the time-domain and run fast, and medium-fidelity potential flow methods are used selectively to assess larger sea-states. High-fidelity computational fluid dynamics (CFD) methods would be used more often if they were not so expensive, and non-simulation methodologies like model testing and full-scale data are also important sources of information that drive ship designs. Due to costs, there is always a limit on the quantity of available experimental model tests and high-fidelity numerical hydrodynamic simulations. Recent research in ship hydrodynamics has investigated the use of neural networks for building fast-running surrogate models that represent CFD predictions of the dynamical response of a ship. D’Agostino et al. (2022) developed models with CFD for short-term forecasting, while the work of Xu et al. (2021) presented a system identification framework for different marine dynamics cases. Most recently, Silva and Maki (2022) developed a methodology for the system identification of the dynamical response of free-running vessels. Though all these methodologies demonstrated implementations of machine learning for representing the dynamics of ship motions, they all only considered a single fidelity. The objective of the current work is to combine different levels of fidelity with multiple long short-term memory (LSTM) neural networks to leverage the accuracy of high-fidelity methodologies (e.g. CFD, model testing, or full-scale) and the computational efficiency of lower-fidelity approaches (e.g. nonlinear and linear potential flow). The new framework and architecture of the networks will be presented, and the trade-off between computational cost and accuracy will be explored between single-fidelity and multi-fidelity approaches. If successful, the methodology will be able to produce predictions of the dynamical ship response that are representative of high-fidelity methodologies with a cost that is comparable to lower fidelity approaches.