Hello Schrand, thanks a lot for posting your inquire at Intel communities.
There is a similar question on this thread, just in case you want to read it. High idle CPU temperature??
We release BIOS version 0024, which included an adjustment to fan control settings, so if you do not have that one installed, please do so. It is available at Intel® Download Center
In addition to that, when you are in BIOS, the fan might decrease the speed since it is not being monitored by any software. Instead when you boot to the OS, the fan is now regulated by the BIOS and the operating system.
This could be the reason why the temperature rises a bit when the NUC is on BIOS.