There are different viewpoints toward these problems. Of course, YouTube thinks that it's illegal to download video from itself. Here is YouTube's answer to the question in terms of their Terms of Service -
You shall not download any Content unless you see a "download" or similar link displayed by YouTube on the Service for that Content. You shall not copy, reproduce, distribute, transmit, broadcast, display, sell, license, or otherwise exploit any Content for any other purposes without the prior written consent of YouTube or the respective licensors of the Content. YouTube and its licencors reserve all rights not expressly granted in and to the Service and the Content.
However, this answer is very impracticable and ambiguous. For example, They do not imply it's illegal to download using third-party software like Leawo Free YouTube Downloader, only that the site provides no method to download videos. Second example, if I download YouTube videos just for watching them off line on my own computer or iPod, and not caught with illegal videos by the copyright owner or YouTube, then I can still download video from YouTube.
If copyrighted video was uploaded illegally, then it would be certainly illegal to download it. If you are caught with illegal videos, the penalties will cost much more than if you had just bought the item in the first place.
If you mean to distribute, broadcast, or even sell the copyrighted YouTube video to others without permission from video owner and YouTube, then it's surely illegal. For example, you can't share it with others or post it on your blog or Facebook page without permission, or you'll be banned by YouTube from their services for breaking the EULA terms. Or worst of all you could end up facing criminal copyright infringement charges that have substantial fines and possible jail time (at least in the United States). Depending on who your ISP is there could be issues there as well with their terms, you could get expelled from school or college if using their networks as a student, or get fired from work if using an employer's internet for copyright infringement.
The last thing I want to say is that a lot of people have downloaded videos from YouTube to watch them off line on their compuers or protable electronic devices, and they never get caught. Just remember that you won't reuse, distribute or sell the downloaded YouTube videos, you'll never get caught.