Trim handgun brass? No way. Rifle? Yup.
Unless you're shooting MOA with your handgun, there is no point. There may be one or two people who can do it, like that fellow that popped the balloon with his revolver from 600 yards. He clearly is an expert shot and did this to demonstrate that Elmer Keith's famed 600-yard deer shot was actually possible. Elmer himself was another rare individual who might've been able to do MOA with a handgun.
But most people are not Elmer. Even Jerry Miculek, as great as he is, would have trouble matching that particular feat. Not saying he couldn't. But even he'd have to severely work at it. So, I view trimming handgun brass as a total waste of time.
By contrast, doing 1/2 MOA with a rifle is plausible after a year or two of regular, and frequent, practice. Say you're practicing 3 times a week with your rifle at various distances. I would expect you'd be able to do 1/2 MOA pretty easily with a typical Remington 700, Savage, Mauser, or Mosin (all in good shape, of course) and a good handload.
Also bear in mind that typical rifle cases stretch considerably more than handgun cases. Even .44M cases with repeated full-house loads will typically split before they stretch too far. Maybe .460 S&W Magnum, due to its length, is a special-case scenario (no pun intended). But none of my .357M or .44M cases have ever required it.