The practice of saying "o'clock" is simply a remnant of simpler times when clocks weren't very prevalent and people told time by a variety of means, depending on where they were and what references were available.
. . .
Thus, to distinguish the fact that one was referencing a clock's time, rather than something like a sundial, as early as the fourteenth century one would say something like, "It is six of the clock," which later got slurred down to "six o'clock" sometime around the sixteenth or seventeenth centuries. In those centuries, it was also somewhat common to just drop the "o'" altogether and just say something like "six clock."
. . .