It's actually an area that's pretty easy to explore, with good access due to being so close to Bristol city centre (and in handily easy reach of a good pub to either celebrate a good visit or drown your sorrows if you've missed what you were looking for...

)
It'll no doubt be my local patch for next season, so I'm more than happy to show anyone around to where I've found particular plants - but in general there are two main access points. The first is on the Clifton side, where just before you cross the Suspension Bridge there's a block of public toilets and, by those, a path leading upwards to a funny-looking building called the Observatory (there's also a very interesting planted flowerbed by the bridge, with specimens of most of the special plants of the Gorge including Wilmott's Whitebeam

)
The part of the Gorge viewable from around the Observatory is known as St Vincent's Rocks, and is one of the best areas for the rarities; there are several excellent vantage points from the tarmac path. In July the ledges are covered in Spiked Speedwell, you should also find other unusual limestone plants (Basil Thyme, White Horehound, Large Thyme etc) plus interesting species that aren't exactly super-rare but still worth seeing; Common Rock-rose, Harebell, Golden-rod, stuff like that. The grassy areas are full of the introduced Keeled Garlic in July-August, with Garden Parsley on the "speedwell ledges" and Sweet Bay, Himalayan Tree-cotoneaster, Hybrid Cockspur-thorn, Shallon and Lauristinus amongst the usual shrubs & trees.
From there you can just continue heading eastwards along the same path, where after about a mile you'll come to a road heading down towards the River Avon (Bridge Valley Road - head straight on here and you'll come to Bristol Zoo after about 100 yards

) Stay on the path and it'll take you to a parking area, complete with ice-cream van, known as Seawalls, where you've got some of the best views of the Gorge as well as of ledges that are covered with Broad-leaved Everlasting-pea from June to August. This is also meant to be where the Bristol Rock-cress can be found, plus Perennial Candytuft, in March-April.
The other access point is to head straight on across the Suspension Bridge, then turn right along North Road into Leigh Woods (it's worth actually walking it; the locals chuck garden rubbish out over the sides of the Gorge here so I've found quite a few interesting aliens, including Rosy Evening-primrose this summer). After a mile or so you'll come to the National Trust car park, where you can either explore the woods (this is where a lot of the rare whitebeams can be found - Wilmott's Whitebeam is right in front of you

) or head downwards through what's called Nightingale Valley onto the River Avon towpath.
Once you're on here you can walk for miles, in either direction (though if you turn right you'll eventually end up back in the Bristol Waterfront, it's a long walk

). The rare whitebeams (
Sorbus bristoliensis,
S. eminens,
S. anglica etc) are all pretty easy to found along this path. Most of the same limestone plants that you get across the Clifton side can also be seen here, with saltmarsh stuff (Sea Aster, Wild Celery etc) by the river and several alien cotoneasters, Rhododendron, Japanese Rose etc.
It's certainly well worth a visit - in fact you'll almost certainly need several