Re: "we'd probably already be dead"
The only question is, at what distance are we doomed by a passing star or black hole ? If it is two light-years away, are we safe ?
While Matthew McConaughey or Beowulf Schaeffer might experience extremes of gravity skimming the black hole, a black hole doesn't have a gravitational field of larger dimensions than a star of the same mass because gravitational force is a result of the distance between two objects and their masses. Same mass, same distance, same force. For example, instantly swapping Sol with a 1-solar mass black hole would leave all of the solar system's planets and lesser objects in unchanged orbits.
The most likely black hole to pass by is a stellar mass black hole, which would be 5 to 30 solar masses (give or take a few kilograms). Taking the worst case of 30 solar masses at 128,000 astronomical units (2 light-years with some rounding)...
1. The black hole has 443 times as much gravitational force on Earth as Pluto at 40AU
2. The black hole has 70 times as much gravitational force on Earth as Alpha Centauri A+B at 275,000AU
3. The black hole has 1/30th the gravitational force on Earth as Neptune at 30AU
4. The black hole has 1/650th the gravitational force on Earth as Mars at 0.5AU
5. The black hole has 1/14,806th the gravitational force on Earth as Venus at 0.3AU
6. The black hole has 1/20,850th the gravitational force on Earth as Jupiter at 5AU
7. The black hole has 1/1,720,000th the gravitational force on Earth as Luna at 0.0026AU
8. The black hole has 1/540,000,000th the gravitational force on Earth as Sol at 1AU
Another way to look at this is: a star 30 times as massive as the sun would need to get within about 5.2AU (Jupiter's orbit) to match Sol's gravitational force on Earth. It'd be a pain at some thousands of AU, too, probably bothering the Kuiper Belt.
So, while Jupiter and Venus warp Earth's orbit a bit over a 405,000-year period, a singular passage by a large black hole at interstellar distances won't do squat to Earth or the solar system. If the Oort cloud actually is out there then it might be disturbed and we'd need to deal with a flood of comets in a few million years, but close passages by stellar-mass objects have happened before. Scholz's Star buzzed Earth 70,000 years ago at a distance of 52,000 astronomical units.
There is a matter of radiation from the black hole's accretion disk, but that shouldn't bother Earth much unless the polar jets swept over us.