John Bercow's office says "strong views" were expressed amid claims he was rude about a female minister.