One of the most important features of every Elder Scrolls title is the absolute freedom when it comes to leisure activities. If one had a slight kleptomaniac tendency and had to constantly hide something in their pockets, at the mere sight of a dresser, this poses no problem.
One just needed to be able to sneak, as one certainly didn’t want to be caught in the act, especially not by a city guard. They not only demand the stolen goods back, but they also want to collect a bounty. I would rather call that a bribe, but better not to say that to the guards’ faces.