I regard sight-reading as being similar to reading a speech aloud that you've never seen before, right off the bat, and making it sound as if you're speaking more-or-less naturally.
Being poor at sight-reading, but still able to "read music", would be like being able to read the speech haltingly, having to stop a lot to sound out many of the unfamiliar words, make out some of the letters, etc., like perhaps you did when very young (or when learning a foreign language). You can eventually get to the "natural speech" level, but not until after going over it several times.
As with the written word, the more a poor sight-reader reads (and challenges themselves with material at the edge of their current ability), the better and better they get at sight-reading, almost without noticing it!
And, also like the written word, actually writing helps with your reading. Imagine trying to learn to read without learning to spell! Learning the basics of notation, and taking the effort to transcribe or copy out music (by hand or by software), can really help.