Computing research is increasingly addressing underwater environments and examining how computing can support diving and other activities. Unlike on land, where well-established positioning methods are widely available, underwater environments lack a common positioning mechanism, which is a prerequisite for many applications. Dead reckoning, the use of angle and distance estimates to track position changes from a known point of origin, is a promising candidate for underwater positioning as it does not rely on wireless signals (which decay rapidly in underwater environments) and as there is a wide range of literature and algorithms freely available. Yet, currently it is unclear whether the existing techniques can be adopted in underwater environments or whether the differences in medium and environment affect the performance of the dead reckoning techniques. We contribute by evaluating and systematically analyzing the performance and trade-offs associated with dead reckoning techniques in underwater environments. We present AEOLUS, a prototype unit comprising of a low-cost microcontroller and inertial measurement unit, to perform experiments on the ground and in underwater environments to assess how well the performance of different techniques translates from ground-based use cases to underwater environments. We benchmark 15 different algorithms and compare their performance in such environments to identify common patterns and dissimilarities, and identify root causes for these differences. The results show that displacement and turn errors can be estimated to within 5% error but that the best performing methods vary between land and underwater environments. We also show that the performance depends on the shape of the motion patterns with some algorithms performing better for hard turns whereas others perform better for gradual, more continuous turns.