ISE has lots of installation issues when installing on Vista x64. In order to try and work around some of them I decided to force an x32 only installation. The required steps are to run the xsetup.exe directly from the bin/nt folder instead of the default xsetup.exe in the root of the installation. Of course to update the installation you will have to use the 11.3 32-bit update. You will have to run the bin/nt/xilinxupdate.exe to install the service pack.
From preliminary tests I see this fixes ISIM and Chipscope issues. For ISIM it now installs the required files under the hdp folder.