The Giants went 5-11 under Shurmur in 2018 and then 4-12 in 2019. New York will now begin its third head coaching search in the last five years.
New York also announced that general manager Dave Gettleman, who was hired alongside Shurmur, will be retained.
"Dave Gettleman is our general manager in 2020 and hopefully for many years after that," co-owner John Mara said in a statement.
Shurmur is the second straight Giants head coach to be fired after two seasons in charge. Ben McAdoo, who took over for longtime bench boss Tom Coughlin, was let go after posting a 13-15 record across 2016 and '17.
"The last two seasons have been a continuation of what has been a very difficult and disappointing period for our franchise," Giants co-owner Steve Tisch said. "It is never easy to part with someone the caliber of Pat. But (co-owner) John (Mara) and I came to the conclusion that we need a new voice in the coach’s office and made the decision to bring in new leadership.
"We understand how frustrated our fans are. They expect more from us and we expect more from ourselves. Our focus now is on developing and improving our football team so that our fans can enjoy the winning team they expect and deserve."
Under Shurmur, New York's offense experienced modest improvement. After ranking 31st in 2017, the unit finished 16th in points last year and then 18th in 2019.
But the team was held back by one of the league's worst defenses. The Giants ranked 30th in scoring defense this year and were routinely exploited in the passing game.
Baylor head coach Matt Rhule and Dallas Cowboys head coach Jason Garrett are among those who've been connected with New York's vacancy.