The Centre has sought consensus from all parties on the removal of Judge Yashwant Varma, who is facing allegations of corruption after the discovery of a large amount of cash from his official residence in Delhi.