The recently sacked defence secretary says negotiations with Labour over a Brexit deal will "fail".