Until I read Sherril's comment on my last post, I didn't see it. I guess I'm used to clothes not fitting perfectly... (tee hee). She said: "A size 20 is too large for you in the shoulders and neck. If you use this pattern again, cut the neck, shoulders, and armscye at a 16 or 18 and then cut the side seams at a 20...."
At first I wondered how she could just look at the picture and see that. But I've been looking at this picture and finally (a day later, yes it takes me a while) it clicked. I see that it's too large at the shoulders and neck. Wow. Double wow.
Well, I'm going to be making another fleece jacket, though with Simplicity 3628 this time - in a slightly thinner black fleece, and I think I'll take her advice on that one. One step at a time. Thanks again, Sherril!