There is a way from the West Rim, but I don't think it could be called non-technical. You would probably want ropes in at least two locations.
There is a ropeless way out of Right Fork into Phantom Valley, using the same trailhead (Lava Point) as the West Rim, which route may be what you are thinking of. Still, it's a stretch to call it non-technical and the entire route is very rugged.
There are at least two long and rugged, but non-technical routes into Phantom Valley from Coalpits/Terry Wash and Right Fork/Trail Canyon. If you want a non-technical route in, these are your best bets, but they are still very long and rugged.