So I figured it out in the end. The problem was the network configuration on my server/host.
Basically TFTP would work and it would use the ifname created with nmcli.
When it would try to mount the NFS rootfs though it would reset and change the ifname and as a result could not find it.
The solution was for me to assign a dedicated ifname matched to the MAC Address.
I did this through systemd by adding a file /etc/systemd/network/10-bootlin.link
[Match]
PermanentMACAddress=f8:dc:7a:00:00:01[Link]
Name=bootlin0
Now it used the same ifname and it just connects.
Bootlin links