Mark Esper was not my favorite secretary of defense. I worked at the U.S. Naval War College while he was in office, and taken on his own terms as a Department of Defense leader rather than a Trump appointee—I’ll get to that—he was a competent bureaucrat. That kind of SECDEF, as we government guys call them, was fine by me. I’d heard rumors of some cockamamie plans he had for the war colleges, including mine, but like most such plans, nothing came of it. He was, to take a line from the HBO series Chernobyl, “not great, not terrible.”

I am perhaps being too hard on him. He didn’t have much time for imaginative guidance of the defense community because, as he now reveals in a new book, he was busy trying to prevent President Donald Trump from doing all kinds of dangerously stupid and unconstitutional things. If we measure his service to the nation by the things he prevented, then maybe he was a pretty good secretary of defense after all.

The problem is that he waited until now to tell us what happened.

This is not an argument that Esper or anyone else should have resigned at the first sign of constitutional trouble. Esper’s predecessor, James Mattis, stayed until he could not in good conscience carry out the president’s orders, but first he spent a lot of time basically saying “yes, sir” when he meant “no, sir" and then hanging up the phone. I was glad he did this, but I was also deeply uncomfortable that Mattis was setting a precedent that the Pentagon could ignore a president, and so I was somewhat relieved when he finally resigned. (As a rule, I do not think former generals, including Lloyd Austin, should be secretaries of defense, but that’s a subject for another day.)

Such resignations, of course, are never easy. Sometimes, people are held in place by the egotistical belief that they are indispensable, but neither military nor civilian leaders want to walk out the door while yelling “good luck” to all the people they’re leaving behind, heedless of who might replace them and carry out the orders they had refused to obey. The Trump administration was full of people who are now telling us they stayed in their jobs to stave off disasters. The person after me, they say, would have been worse. A goon. A hack. A minion.

In some cases, I think this was a fair defense, especially for people who were in critical national-security positions. There was actually a debate about this among the original group of Never Trump Republicans (most of whom came from the national-security community) back in 2016. Some of it was conducted in the open, some of it in private, but it centered on this question about whether Republicans should accept jobs with Trump. Many of us, including me, signed letters saying we would never work in a Trump administration. (No one was going to be beating down my door to bring me to Washington, but I thought it was good to join many other then-Republicans on the record.)

The risks of working for Trump were elaborated upon well in 2017 by my Atlantic colleague David Frum; our colleague Eliot Cohen also went back and forth on it and even changed his mind. The danger was obvious: You will end up selling your soul and you will likely fail to do much good. The counterargument was also obvious: The interests of the United States of America require that this train wreck of an administration—staffed with the likes of Stephen Miller, Steve Bannon, and His Faux-Britannic Excellency Sebastian Gorka—should have at least some non-stupid, non-craven, non-nutball types in the executive branch.

I argued at the time that there was no way to put child-safety bumpers on all the sharp edges of the White House, and that if Trump was going to drive the country into a ditch, the sooner we got on with it, the better. I am not sure now if I was wrong, but the best evidence against my position is that Esper may well have prevented a war with North Korea by averting Trump’s idiotic evacuation order for Americans in South Korea. If that’s the case, I’d have to say it was worth it to have someone in the right place.

But the price for this quiet custodianship (a form of opposition to Trump described in detail by Miles Taylor, now known as the author of the famous “Anonymous” op-ed in The New York Times) is that the American people never really knew how much danger they were facing, at home and abroad, at any given moment. The people who mitigated Trump's idiocy were like pain-killing medication whose soothing effects end up masking symptoms of a serious illness. These efforts allowed both Trump’s supporters and his critics to comfort themselves with the knowledge that someone, somewhere, was trying to limit the damage to the country. His fans could say, “He’s just inexperienced but he has good people around him,” while the opponents could say, “He’s an execrable moron but reasonable people are in charge, and they’ll save us from the worst.”

Which brings me back to Esper.

Even if we cut some slack for Esper and all the others who served as honorably and conscientiously as they could until they were faced with either the dead end of resignation or being fired, the fact is that these men and women remained silent for far too long once they were out of government service. They held back important things that the American people and their elected representatives needed to know. They kept them as their own personal secrets, either out of some misplaced sense of bureaucratic propriety, or because they had a book deal and didn’t want to steal their own thunder from release day.

They had a duty to speak up sooner. And they failed in that duty.

Esper, Mattis, Rex Tillerson, and many, many other people who crawled through the Shawshank sewer pipe that was the four years of the Trump administration needed to speak up the minute they were out. Instead, they teased their book bombshells or played coy games of slap and tickle on cable outlets.

Sitting on crucial information about whether the president himself is a menace to the security of the United States—hello, John Bolton—is bad enough. But there is an assumption that undergirds this reticence that is shared by both former officials and the public, one that is inimical to our democracy: The officials who have not spoken up (even without a book deal) have been silent because they somehow seem to believe that things will just work out.

This is more than just trusting in some “responsible people” in charge. These appointees think the entire edifice of American democracy is more stable and durable than it really is. They might say they think Trump is unfit, unstable, a danger. But in the end, they have faith in the system. They see Trump as only one man, and the system as a bulwark of laws and regulations, people and committees, institutions and practices that will somehow kick in and prevent a catastrophe.

Now, to be fair, most of these executive leaders came from years of working in big, complex organizations, and in that context, “the system will survive” is a reasonable view. Anyone who has tried to implement change or reform in a large organization knows that it is, to use a Navy metaphor, like trying to turn a battleship around. Even successful presidents have been handed defeats when trying to change how the American government operates: Ronald Reagan’s big ideas about eliminating Cabinet departments and cutting taxes were halted by 1986; Bill Clinton declared “the era of big government” to be over, but big government turned out only to be in hibernation.

But these are bureaucratic and political issues. Governments are more than just large organizations. They are a far more delicate web of norms and habits, and liberal democracies especially are built on informal agreements rather than black-letter law. Yes, we have tons of laws and administrative bumf that complicate our lives, but when it comes to the nature of our democracy, the Constitution manages to do it all in fewer than  5,000 words. Our basic rights as citizens take less than a page. The rest relies on us.

And so when you know that the president is unhinged, when you know the country is in danger, when you know that plots are being hatched to subvert the Constitution, you have a duty to speak. This duty supersedes confidentiality, partisanship, or personal loyalty. It sure as hell is more important than making bank.

Think of all the people from whom we don’t have a full account of this mess, who did not speak up even as Trump was running for reelection or inciting an insurrection: Mattis, Tillerson, John Kelly, Robert O’Brien, H. R. McMaster, and many others. Yes, some of them turned on Trump by saying later that he was unfit, lazy, selfish. That’s not enough. These are experienced political figures who know that the public needs to be grabbed by the lapels and made to listen to a compelling story. The too-late book excerpts, along with all the throat clearing, the circumlocutions, the carefully phrased “but I’d still support the nominee” escape hatches don’t cut it.

Some of these people, we must assume, are biting their tongue because as much as they might hate Trump, they hate Democrats and Joe Biden more. They personally think Trump is an idiot but they would have endured a Trump reelection, and even now they hope for a return to power by the Republican Party. Some of them, on the other hand, are likely staying silent because they know what the Republicans have become, but they will not risk the opprobrium—and the loss of income—that will come from turning on the GOP. (I’m pretty sure, for example, there will be no more group letters based on principle in 2024 if Trump runs again.)

None of these are reason enough to endanger the country and the Constitution.

I realize some of you might think it’s easy for me to say this, since I was never a political appointee. But I was in a vulnerable position as a government employee, and from the first time I spoke up, people tried to get me fired from the Naval War College. Even with tenure, I could have been dismissed if I was found to violate the Hatch Act, the law prohibiting on-the-job politicking by federal employees.

I called my family together nearly six years ago and said that I could lose my job if I kept writing about Trump. All of them told me to keep writing, and we’d deal with whatever comes. So I did. I stayed on the right side of the Hatch Act—unlike Trump’s subordinates who violated it with relish and disdain. I am now retired (and hold the title of professor of national security affairs emeritus, which is nice), but for more than five years, the demands to fire me came so often, as one administrator later told me, that after a while they didn’t even bother to inform me about them anymore.

Still, I was just a small fish in a big pond, an academic writing some opinion pieces. I cannot imagine what it would be like to be burdened with knowing the president was mentally unstable, that he wanted to fire missiles at Mexico, that he was planning to exit NATO, that he wanted to shoot unarmed protesters, that he wanted to invalidate a national election. That is a level of responsibility beyond anything I have ever experienced. This was Night of Camp David stuff, and I’m not sure what I’d have done.

But I’m reasonably certain I wouldn’t have kept it to myself until my agent told me I had a deal.