When you plug-in your USB 3-gigabit Ethernet Adapter, it has to allocate system resources for the network device.
The resources are not freed when you unplug the adapter, because the Gigabit Ethernet driver does not stops.
There is nothing unusual in this case. Network drivers & protocols need more resources than USB 3.0 Mass storage drivers. This is done only to ensure Fast Ethernet operation.
Nothing is wrong there ...you are good ...!