Reply to post:

How do they know it's the film or the book that's being referenced?

They're talking about references to films, in other films. Since film is a visual medium, such references generally have a visual component; since that's not present in the book,¹ it identifies the film as the primary referent.

Even when there isn't a visual component - for example when someone quotes a line that appears in both the book and the film adaptation - there are often other clues, such as prosodic features of delivery (intonation, pacing, etc). As a synchronous, recorded² medium, film has both information channels and restrictions that prose doesn't, and those can often be used to demonstrate that the referent is probably another film.

There are entire scholarly disciplines (influence studies, some aspects of textual studies, etc) which have been dealing with this sort of problem for a couple of centuries (in the domain of books; for film obviously it's been somewhat less, but still a good long time).

And in some cases it's simply because the new film refers to some detail that's original to the older film, and not to the book. If I mention "ruby slippers", I'm talking about the film of The Wizard of Oz, and not the novel The Wonderful Wizard of Oz, where the WWotE and then Dorothy are shod in silver shoes. ("There's no place like home" is also the film's invention, &c.)

¹Assuming we're not talking about illustrated books, and generally in this context we aren't.

²Assuming we're not talking about animation, and ditto.