I've done this before! No mill needed. Very much a pain to do, though.
I've milled an 80% aluminum lower on my drill press before, and I've milled an entire single shot 80% pistol from a block of plastic, too. Good stuff!
In addition to the drill press and the jigs to make your gun work, you should really have a caliper, a scribe, the drill bits, and a mill vise.
If you get a metal caliper, it may also work as a scribe. I love my Harbor Freight one.
The mill vise that I bought for my 80% pistol build worked flawlessly. There's one available here for under $40:
You need to make sure that every single axis is tight, and while it may be slow going, it's worth it. Basically, you don't want to use the mill vise as an actual mill would work - you want to use the mill vise to be able to move your material around without having to unclamp/reclamp it. Bring the bits you use straight down every time. If you need to do a finishing pass to clean it up a bit, that will work (if every axis on the vise is really, really tight). I definitely did a pass to remove material from the hole in the bottom that the trigger goes through.
The bits you need? Drill bits work nicely to remove most of the material, but a 3/8ths bit is necessary for the safety hole.
Also, I went to Lowe's and bought an upspiral 2-flute 1/4" router bit for my build. One bit for one build is a good bet - aluminum dulls the bits really quickly. Delrin (the plastic I was using) doesn't dull them at all, though.
When you're doing the milling itself, the trick is simply to make sure that everything is clamped up as tight as it goes, then to set your depth gauge on your drill press so you can keep stopping the down drill at the same point.
Drill out as much material from the aluminum portion as you can. DO NOT drill the holes too far towards the edges of the fire control group - you can remove more material later, but you cannot put it back later!
Make sure you have your depth set correctly. Don't go down quite all the way with any of the bits - the bottom of the fire control group is 1.25", so only go about 1 1/8th inches instead. The top of the fire control group, where the upper fits in, is at .50", but you can go down to about .65" without problems - so you can set that one for half an inch exactly.
After drilling out as much material as you possibly can, take the router bit, clamp it in, and remove all the little wall parts. Make sure the clamp is tight for this. The mill vise is cheap junk (even though it works), and won't hold tightly at all. Make sure you have it clamped down so far that the wheels don't move with the little knobs on them - you should have to use two fingers for leverage to move those knobs. The router bit should take out all the extra material without a problem. You can go straight down with it and only engage 1/8th of the surface area at the bottom of the bit (take out those extra walls)- should work fine.
All final material should be removed using a rotary tool. Try to fit the hammer in the fire control group and see if it goes in without binding all the way down to the bottom.
Also - make sure you tighten that clamp for the table on the back of this thing REALLY tight. I had problems with the upspiral bits literally pulling the whole table up towards the bits I was working with, so I started using pliers to tighten the heck out of that clamp. Coincidentally, after too much working with it, I stripped the clamp, and now my drill press is roughly useless.
PM me if you want any more tips or pictures.