Well, the usual argument is that an average person can easily lose one or two kilos of body weight, and that would cost good money if you take it from the bike.
On my experience, weight is more noticeable when riding uphill, but then, if you usually ride uphill I don't think you would want a single speed. Other issue would be speeding from a stop, or starting a sprint from a lower pace, but again, on a single speed you already have a problem with that, specially speeding from stop (or if your single gear is short, maintaining high speed for a good stretch, the eternal tradeoffs)
I rather start in a short gear with 5 kilos more and then switch to longer gear if we were to race 100 meters, or from a light stop to another, than on a single gear. I don't know, if you want speed, gearing seems to me a lot more important. On long stretches you could use longer gear.
Then, if we are comparing same gear ratio on same frame geaometries and handlebars, for the same cyclist, of course you want less weight, but is it worth the money? I don't think so.
All in all, if you were to magically create a clone of yourself, and race against him in the same magic bike, but one is 15 kilo and the other 7.5, and all conditions are magically the same, wind, body placement, whatever, physics of this planet will make the lesser weight bike faster.
Is it worth it? that's up to you, and more specially to your cash availability. If you want an opinion, it is not, I ride a road bike and a mountain bike, the mountain bike must be 3x the weight of the road, and still, I ride a lot of Strava segments that I did faster on the mountain bike (it's got no suspension though). So, yep, hunching like an egg and saying to yourself I'm going to pass that guy on the scooter or die trying, that can be more relevant to speed than triple weight. And of course, having bottom bracket destroying hams is #1 in the speed equation... sometimes I wheelie from the sheer power, in a 52-13 ratio :D
Really, diss weight and get some bike you fancy, and ride it fast if that's what you like. But don't put yourself or others in danger knowingly.
edit: only top speed?: