Some folding bikes are quite good for long distances. Unfortunately, most of them seem to be optimized for short hops, by design (or by the fact of their design limitations). You're correct in thinking that more expensive folding bikes can be much easier to ride, where the money is going for stuff like custom configuration or (even a custom-built frame). Folding bikes aren't unique in this regard, but economies of scale place additional limitations on them.
Fit Limitations
How to properly fit a bike has been covered in detail elsewhere, so I won't get into issues of seatpost height, reach, bar height, bar width, and so on. But most mass-market folding bikes seem to be built in a one-size-fits-none design.
It is quite possible to get a folding bike that's sized properly. However, since folding bikes are a small slice of the bicycle market, they're somewhat pricier. And keeping several frame sizes in circulation is even more expensive. Until more people buy folders, I think we're gonna keep having this problem. And very tall or very short people (or very heavy people--most folders have a weight limit of 200-225 pounds) have a lot of trouble finding folding bikes that will fit.
Frame Flex
Bikes that have frame hinges--like Citizens, and Dahons (and their clones) and the Raleigh 20's--are subject to frame flex. This can be combated by keeping the hinge joint tight, but many of these bikes will never be as stiff as a bike without a frame hinge.
Essentially, when sitting in the saddle, if you can move the handlebar forward and back, you've got frame flex. Some bikes can also have a flexy stem post, creating similar problems.)
(The Raleigh 20 has an angled hinge joint that, I'm told, mitigates the problem. From test-riding one, I'm inclined to agree with that; the bike didn't feel flexy at all.)
Smaller Wheels
Also, smaller wheels do tend to be a little "squirrely", in that they're harder to control. This does tend to be more if an issue with 16" wheels than 20" wheels, however.
This is fairly self-correcting, though, and it's something that the rider learns to compensate for fairly quickly. Until that point, though, it can impact the rider's ability to use the bike for longer rides.
Handlebars
Most folding bikes come with flat bars. As with all bikes, the more potential hand positions you have, the happier you'll be on longer rides. Bar ends will do this, and you may be able to fit them on the bike without compromising its ability to fold.
Gearing
"Do you have to pedal faster with those small wheels?" It's a common misconception that small wheels make the cyclist work harder. When properly geared, small wheels can perform well.
However, many folding bikes come with three-speed hubs, for a variety of reasons. (Less maintenance, cheaper and lighter than 7 or 8 speed hubs, and derailers can get your pants leg messy when you have a folding bike on the train.) This is only an issue if you have hills, and it's not one of your specific concerns, but it is a factor for general use of folding bikes.
However, faster gears can make a longer ride more pleasurable--and shorter. My 3-speed folder tops out around 26 MPH, which means it's not so great for a long day of touring. (For some, speed isn't an issue, so this won't matter to parient riders in very flat areas.)
