I have some troubleshooting you could do with this situation related to WiFi speeds.
1. Please make sure the unit is being powered on by the power adapter it came with and not with USB port from a TV or monitor.
2. Try uninstalling/Installing the driver for the wireless adapter
3. You could also perform a flush DNS to see the behavior of your system.
4. If the connection is of for web browsing or other applications, I would say is more like configuration issue for the SSH.
I would also like to recommend you to visit the Ubuntu forums to check this issue.
I look forward to your reply.