Scotland's Brexit minister says a fear of Sinn Féin means the government do not want to devolve EU powers.