Pontypridd MP Owen Smith says he and a "lot of people" are considering leaving the party.