You may be experiencing a well known copter blade phenomenon known as
"ground effect". The flight behavior of a quadcopter becomes unstable at extremely low altitudes because it's own propwash bounces off the ground below and creates turbulent air.
With sufficient altitude this effect completely disappears because there is room for the turbulent air to disperse. It's very noticeable at 2 feet but at 2 metres not so much.
An imperceptible breeze could also push the drone off course and yet be unnoticeable to a operator just a few metres away. You can figure out if it's wind by the direction it's pushed when you fly the straight line course in the opposite direction. Those are usually side to side course deviations.
Unbalanced or damaged props are also a common reason for erratic flight paths. Check your blades with eyes and fingers for cracks and nicks. Swap the props to different motors to balance any subtle warping.
My drones don't all fly
perfectly straight line -- I've had to accept that. They're just not all that precise out of the box and become less so over time.
But the up/down drift you see when really close above a hard surface is
ground effect -- darn physics again!