The movie was quite a bit different from the book, so read the book first so that at least you will know what is going on. The film has a much darker tone than the last two, this tone helps the movie a great deal - it's much more easy to take it seriously.
It is by far, much funnier that the first two movies. There are more jokes and 'must see' bits like Hermione punching Malfoy. It's quite a bit scarier that the first 2 as well. The dementors are well done, as were the flying scenes with the firebolt at the end, and flying with Buckbeak.
Gary Oldman is simply a wonderful actor and his inclusion in the cast does much for the film. Prisoner of Azkaban does an infinitely better job than its predecessors at portraying the emotion between characters and is fast paced and thrilling. Of course the special effects are amazing - as one would expect.
If you love fantasy, here are some more really good ones:
Godstone - The Kairos BoxesThe Magician's Nephew (The Chronicles of Narnia)