I trust Vision with it. And his keeping the stone was far better than allowing it to be used by Ultron, which is what was going to happen until I intervened.

[Thor doubts the stone could be removed from Vision without killing the man. It was the stone that effectively brought him to life, after all.]