Guard is confused by combination of shadow + swashy hat

Usually when you have a swashy hat on, the guards are supposed to chase you right? 

Was being chased by a shadow. Ran past guard, and stole some flowers. Guard chased me as expected. 

I put on the swashy hat. Guard stopped chasing me, but continued shouting insults at me/the shadow while running around in the same area. 

Continued stealing flowers, guard still running in same area until I got close enough, or got sane enough that shadow despawned (unsure which), but seemed to reaggress him when I stole the last few flowers. 

Also, guard was walking around with a torch during the day (other guards I ran past earlier had their regular axes out).

https://clips.twitch.tv/InspiringZealousWoodcockAMPEnergy