Most likely, you can delete them.
Microsoft Update, Windows Update and the background “intelligent” file transfer (BITS) will download updates to your system drive, and then extract the compressed archives to the drive with the most space free in preparation for the install, regardless of whether that is a removable drive or not. Yes, that is very stupid.
Normally these files will be deleted after the actual install of the update.
However, if using a removable drive, and removing it before the install, Microsoft Update / BITS will switch to a different drive and extract the downloaded files there, forgetting all about the files that were on the removable drive. The files will then be left forever.
So what can you do to prevent this from happening? Turn off the automated download of updates. If using Vista/W7 or newer server versions, change the settings in the Windows Update control panel to “Check for updates, but let me choose whether to download and install them”. (Do not confuse this with “Check for updates, but let me choose whether to install them”, which will download without asking.) In XP and older server versions, the control panel and wording are different, but you can achieve the same – no automated downloads.