The characterization of suspended sediment dynamics in the coastal ocean provides key information for both scientific studies and operational challenges regarding, among others, turbidity, water transparency and the development of micro-organisms using photosynthesis, which is critical to primary production. Due to the complex interplay between natural and anthropogenic forcings, the understanding and monitoring of the dynamics of suspended sediments remain highly challenging. Numerical models still lack the capabilities to account for the variability depicted by in situ and satellite-derived datasets. Through the ever increasing availability of both in situ and satellite-derived observation data, data-driven schemes have naturally become relevant approaches to complement model-driven ones. Our previous work has stressed this potential within an observing system simulation experiment. Here, we further explore their application to the interpolation of sea surface sediment concentration fields from real gappy satellite-derived observation datasets. We demonstrate that end-to-end deep learning schemes—namely 4DVarNet, which relies on variational data assimilation formulation—apply to the considered real dataset where the training phase cannot rely on gap-free references but only on the available gappy data. 4DVarNet significantly outperforms other data-driven schemes such as optimal interpolation and DINEOF with a relative gain greater than 20% in terms of RMSLE and improves the high spatial resolution of patterns in the reconstruction process. Interestingly, 4DVarNet also shows a better agreement between the interpolation performance assessed for an OSSE and for real data. This result emphasizes the relevance of OSSE settings for future development calibration phases before the applications to real datasets.