Let's define "difficult"... for instance, as compared to a cruiser of similar engine displacement.
If it means hard to move around with your feet while in neutral, a sport bike generally has a shorter wheelbase, and should be easier to maneuver than a big cruiser.
If it means handling the bike at high speed through curved roads, the higher power-to-weight-ratio, better suspension, and superior brakes again give a sportbike the edge over the cruiser.
Yet riding position and ergonomics are the most important factors to many riders, as a modern sportbike's lay-on-your-belly position and board-hard seat don't really lend themselves to comfort. And when you're uncomfortable, you can be distracted, which can make riding difficult. As others said, you can't have a big gut and stretch yourself over a sportbike tank. And you can't have stiff joints either.
