Here's my personal opinion, not necessarily speaking for anyone else.
I believe that votes and comments serve separate purposes, and either can be useful with or without the other.
To me, a vote serves primarily as a signal to the community as to whether I think this post is worth reading. Most people don't have time to read every post on the site, or even all the posts on topics that may interest them, and so a post's score can be useful to them as an indicator of whether it is worth their time.
A comment, on the other hand, is feedback to the post's author about issues their post might have, or how it could be improved.
My view is that either of the two can legitimately be used on its own. As an analogy, when I read a book I don't like, I think most people would agree it is legitimate for me to tweet "I disliked Fred Smith's latest book and I don't recommend reading it", or to write a letter to Fred Smith explaining how I think he should improve his characterization, or both, or neither. But doing the former does not, and should not, lay on me any obligation to also do the latter.
That said, when I think a post has serious enough issues to warrant a downvote, I will usually leave a comment to inform the author of those issues, if I think it will be productive to do so.
But here are some situations where I may decide that comments would be unproductive, and would instead "silently downvote" without commenting:
The post is completely unsalvageable, bearing no resemblance to quality on-topic content.
The post does not appear to have been posted as a good-faith question or answer. The poster doesn't seem to really be interested in receiving or sharing on-topic information, but rather is just trolling, complaining, bragging, or seeking attention.
The post is actually seeking information, but shows such a lack of effort that I doubt the poster is interested in spending time to improve it. For instance, a copy-pasted homework assignment.
Another user has already left a comment about the same issues that I would have pointed out. In that case I would usually upvote that comment instead of leaving my own.
The poster has a history of making posts that are problematic in the same way as this one, has previously been informed about the issue (by me or someone else), and is evidently choosing not to change their practices. In that case I see no point in providing advice that the user has already heard and rejected.
The poster shows signs that they would not receive my feedback constructively, but would instead become argumentative, hostile, or abusive. This could be indicated by their language in the present posts, comments they have left, edit history, or their previous activity on the network.
To the last point, a few years ago, I left a comment on a post on another SE site, pointing out some technical errors. The post contained some warning signs but I decided to go ahead and assume good faith. The user apparently did not appreciate the correction, and responded over the following days with a stream of several hundred obscene messages targeted at me personally, posted to multiple SE sites (using Tor or some similar countermeasure to circumvent IP bans), as well as to my personal and work email addresses and my personal blog. I found it more tedious to clean up than traumatic, but even so, since then I tend to be more judicious in choosing who to interact with, and I wouldn't fault anyone else who felt similarly.