Backfill material with high water table

[LazySOB]

Advantages of sand:
Compacts quickly
Doesn’t hold water
Drains well (another way of saying the same thing…)

Advantages of dirt:
better for growing things, since it retains water
looks better

Not knowing more about your particular situation, most likely sand was the more appropriate backfill.