Years ago the talented and generous writer James D. McDonald had the screen name Yog Sysop, and happened in this guise to develop the most important law of publishing:
Yog's Law - Money Flows Toward the Writer.
That's it, that's all. My version is a bit more awkward: If at any time you have sent your publisher more money than your publisher has sent you, you are with a vanity publisher.
We can unpack that one a bit. When a writer pays the publisher more than the publisher pays the writer, the writer is said to be subsidizing the publisher's bottom line. That publisher is a subsidy publisher. By legal term of art, a subsidy publisher is a vanity publisher. Using subsidy publisher means you are self-published.
A subsidy publisher (let's use the kinder term) can be a terrific way to publish a book that has a limited audience. It would be foolish to expect Random House to publish a book of your grandmother's lutefisk recipes. But it might be just what every Lutheran chef in Minnesota wants. The comparison in Hollywood would be to expect Warner Brothers to produce a tiny art film. One of their divisions might, but WB can't market a film at that small scale. So know what would be a good fit for a subsidy publisher.
But if you have a book that you wish to see reach a wide audience, don't pay to get that published.
But what if you try to get an agent, and then try to find a commercial publisher, and are turned away? What if you query one hundred people (considered fairly definitive) and none of them wants your book? Should you self-publish (subsidy publish) then?
No. You put that book away and write another. You probably won't like that second book so well as the first. You're still in love with the first. It's called the sophomore slump. Try to land an agent with that one anyway. If that fails, try to land a commercial publisher. And if that doesn't work? Write another book.
If you are a writer, you will do this as many times as it takes. It took me a dozen novels to get here with number twelve. Do I want anyone reading my first novel? Oh no, no, no. And if your first novel cannot attract the attention of anyone in the world of commercial publishing, you shouldn't either.