This gets rid of some more big memsets. It may replace them with a lot of tiny mallocs, though. If this turns out to be a bad idea then at least we can easily revert the change.