Making the most of observation and modeling capacities to forecast and reanalyse upper ocean dynamics is a key challenge. State-of-the-art data assimilation (DA) schemes generally blend prior knowledge on the underlying governing laws with available observation data. They however often show a limited ability to fully exploit the richness of observation datasets, especially when considering fine-scale processes. To address this challenge, we turn data assimilation into a physics-informed machine learning problem. The proposed method, referred to as 4DVarNet, relies on an end-to-end differentiable framework backed on a variational DA formulation. 4DVarNet can learn from data not only a DA solver but also jointly some representation of the inverse problem. Importantly, this end-to-end learning framework applies to irregularly-sampled and multimodal observation data. We apply 4DVarNet to the assimilation of sea surface dynamics from multimodal satellite-derived observations, especially satellite altimetry (including SWOT data) and sea surface temperature (SST). Within Observing System Simulation Experiments (OSSE) using realistic high-resolution ocean simulations, we first show that 4DVarNet may outperform state-of-the-art optimal interpolation and model-driven schemes for the space-time interpolation of the sea surface height. We also point out that it may reveal total sea surface currents from the synergistic analysis of altimetry and SST data. We will further demonstrate the scalability of 4DVarNet using native AI technologies and distributed multi-GPU systems to scale up to an ocean basin or the global scale. Besides, we will discuss how 4DVarNet could provide new means to revisit the design of ocean forecasting and assimilation models as well as sampling strategies of ocean observation systems. Especially, our experiments open new research avenues towards the design of objective-specific schemes for ocean modeling, forecasting and reanalyses.