I feel your pain. I have 2 seperate shares connected. 1 as a linux share(6TB MBLD) and a 4TB Seagate attached to it’s USB port as a windows share. Named Videos1 and Videos2. I read on the forum that the same shares with the same first letters have problems refreshing. So I changed them but still have the refreshing problem you could give that a try.
It’s the worst feeling when using this device is refreshing, waiting and then having to go back into settings to try again.
The surefire way and what I do everytime is turn off media library, clear the library, then turn the media library back on, that has worked everytime for me.