You can deselect the priority number. Doesn't do anything but is a bit confusing.
Priority number is highlighted.
Now I click 6 again.
Not highlighted anymore.
Now I select another building that also has priority 6.
The bug will fix itself when selecting a building with any other priority.