Also, if I have two ssh connections, they both "unblock" at the same time. I wonder if it could be something powersaving - either the wifi or cpu? Since there's no console/gui login.
FWIW I've disabled the bluetooth kernel module, in case it was due to interference, with no joy.
I have some troubleshooting you could do with this situation related to WiFi speeds.
1. Please make sure the unit is being powered on by the power adapter it came with and not with USB port from a TV or monitor.
2. Try uninstalling/Installing the driver for the wireless adapter
3. You could also perform a flush DNS to see the behavior of your system.
4. If the connection is of for web browsing or other applications, I would say is more like configuration issue for the SSH.
I would also like to recommend you to visit the Ubuntu forums to check this issue.
I look forward to your reply.