i believe that you are trying to change the cover art thumbnail for the video files.... there is no easy/straight forward way to do this.
Windows takes a frame from the first shot of the video and use it as thumbnails. However, if you could edit the video and add a frame at the beginning of the video, then you can change the thumbnails as you like. You might need a third party software for that. Try to google it and look for a Video Cover art editor/changer.
I would recommend VirtualDub :tup