The detection of the electromagnetic counterpart to the binary neutron star (BNS) merger remnant GW170817 established for the first time the connection between short γ-ray bursts and BNS mergers. The observation of a simple power-law spectrum from radio to X-ray, as well as the brightening of this emission for more than 100 days, were somewhat unexpected. Current theoretical models explain this behavior as originating either from a relativistic off-axis jet or a kilonova-like outflow. In either scenario, there is some ambiguity between the power in non-thermal electrons and the magnetic field strength in the emission region. In this work we include observations in the GeV-TeV domain to help break this ambiguity in the models. We report for the first time on deep H.E.S.S. observations of this source between 124 and 272 days after the merger with the full H.E.S.S. array of telescopes, as well as an updated analysis of the prompt (<5 days) observations with the upgraded H.E.S.S. phase-I telescopes. We discuss implications of the H.E.S.S. measurement for the magnetic field within different source scenarios.