Natalie Portman
Natalie Portman played Jane Foster in the first two Thor movies. The first was directed by Patty Jenkins and the second, Dark World, was directed by Alan Taylor. Portman and Taylor butted heads so much that Portman never returned to set for re-shoots after filming wrapped. Naturally, she declined to be a part of a third film.
Richard Dreyfuss
Richard Dreyfuss played Matt Hooper in Jaws. Dreyfuss and Spielberg nurtured a good relationship thanks to this film and even teamed up for Close Encounters of the Third Kind a couple of years later. The Jaws sequel was set in motion but Spielberg was not interested in taking part. Dreyfuss backed out as well, stating, "I knew what made Jaws good, and we weren't talking about Steven directing."