No, it wasn't yesterday when he lost the veto. It also wasn't the day before when he was nominated, or the previous day when Russell was evicted and Kevin won HOH.
The day that sealed Jeff's fate (barring some weirdness over the Pandora's Box key) was ten days ago, as two terrible decisions have come back to haunt him:
1. Keeping Natalie over Lydia may have been THE worst move of the season. Lydia was clearly a hard to person to share a house with, but she was no player. She was the first or second player out of every competition, and unlike Natalie it wasn't because she was purposely losing. Kevin was also a lesser player with her around, as he had to constantly clean up her messes.
2. Not letting Kevin win that night's HOH competition. Jeff was not on Kevin's radar at that time; Kevin would have certainly been targeting Michelle and Russell. Even if Jordan or himself had been put up, one of those two would have likely gone home.
Of course, another terrible decision was set into play that same night, as Natalie and Kevin pulled out a miracle and convinced Jeff and Jordan that they could be trusted. Backdooring Russell was the icing on the cake in a week of bad decisions.