It's possible that you would be better off on the smaller size frame, but determining how much improvement you would see depends on multiple factors.
The amount of time/distance and at what level you are riding obviously is a big factor. If you are riding long distances or training for racing bike fit matters a whole lot more than if you are casually riding 10 miles a week. Your relative proportions also matter. If you have relatively short legs and long arms a size above the nominal size for your height may be appropriate.
What I would do is examine your fit on the bike and see if there is anything obviously problematic. Many bike stores offer a fitting service, and many will do a rough assessment if they think they can persuade you to buy another bike.
If you want to assess your fit yourself, basic sizing advice can be found on the web. The basic procedure is to set saddle height and fore-aft position properly relative to the bottom bracket, then look at the angles of your torso and arms to see if the bars are too far away or low, or too high.
If you have have managed to get a reasonable fit by shortening the stem you may be OK. If the bike really is too big, and you are leant forward too far, or you hands are too far away it will affect the power you can output and possibly cause discomfort or pain.