Keep xFormers attention masks on the GPU running each layer (#8516)