Ball hits fielder's hat

We were playing a friendly match, with the players taking turns to umpire for a while. The batsman hit a powerful shot and, while chasing it to the boundary, the fielder's hat fell off. He gathered the ball before it hit the boundary rope, and threw it back to the wicket-keeper. On the way the ball bounced on his hat, which was lying on the ground. Should we have awarded five Penalty runs in addition to the three runs the batsman made?

No, because the discarding of his cap was not a wilful action by the fielder. But if, for instance, he had deliberately caught the ball in his cap, or it hit a helmet on the ground behind the wicket-keeper, then the ball is instantly dead and five Penalty runs are awarded to the batting side but, note, not to the batsman himself.