Hi Sometimes, You are right of course, those forces do have an effect, and it is accumulative, but they are so small when compared to any objects sizeable enough to be seen from such a distance.
Do you remember Comet Shoemaker/Levy? That did get tugged out of it's orbit, but it took a planet the size of Jupiter and even then it took many orbital passes before the cumulative effect caused the break-up of the comet, and then it's fatal plunge right into Jupiter itself. It completed it's final orbit around the Sun in bits.
Go to :-
Comet Shoemaker-Levy 9 FAQ
It records the end of that Comet which broke into over 70 pieces, the largest of which was still over 2 kilometers wide, and impacted Jupiter with a fireball that would have incinerated most of the Earth.
Not something easily pushed about, even in pieces.
Now of course, it has become another one of those ancient wanderers now gone. In the early days of the Solar System, there were masses of bits and pieces flying everywhere, aggregating together to form the Planets themsleves. When the Planets became established they were still very much in the firing line of all the remaining uncaptured bits. These were responsible for the masses of impact craters on Planets and Moons throughout our system. These impacts becoming less and less frequent as the objects diminished in number.
Those remaining are less and less likely to fall foul of anything since their orbits have avoided problems for billions of years.