Smiley actually has very little patience for stupid, hearty aristocrats, but he's generally too polite/socially anxious to tell them to go away. So obviously Bertie must be inflicted on him often.