So I got my handy, dandy long brush tool and started brushing it out. That's when I realized there was really a lot of lint in there.
I pulled the dryer out from the wall and checked inside the vent hose at the back. It was almost completely full of lint!! I brushed and vacuumed and cleaned it out, and was totally disgusted with the whole mess.