BeagleBone Blue default pull up direction

I’ve discovered on the Blue the GPIO input pins on the GP0 and GP1 connectors are inconsistent. GP0_3, GP0_5, GP1_3 and GP1_4 are all pull ups and GP0_4 and GP0_6 are pull down.

How do I set the direction of the pull up/down?

I’m running Debian Image 2017-09-03