Fix slow Vite rebuilds in projects with large gitignored directories (#20408)