After some testing, seems reset-deassert-us needs a REALLY large value to make it work correctly. 5000 is not enough but 10000 seems ok on my board; better to test on more devices to see if 10000 is ok.
Edit: 5500 is not enough but 6000 is ok. I think maybe 6500 is enough for all devices? I’m working on a patch to upstream mainline kernel now.