I know it's normal to lose a bit of air when getting the pump off but I'm not sure about this
The air you lose when taking the pump off should be just a very brief hiss. And, actually, it's mostly air coming out of the pump, not out of the tyre. You shouldn't lose any significant amount of pressure when taking the pump off. The only thing I can think of to do wrong when taking off the pump is if you have the type where a tube screws into the pump and onto the valve. You need to unscrew the valve end of the tube first, but surely that's what you'd do anyway.
Did the same thing happen to both tyres? If so, it's the pump. If one tyre inflates and the other one doesn't, it's the valve, and you need a new inner tube. I've never had a valve go bad but I guess it happens occasionally. Check if there's any grit visible, which might be holding the valve open or something.
If you want to try a different pump, go to a bike shop: they'll have one that you can use. Avoid gas station air pumps if you can: they're designed to fill the relatively huge volume of a car tyre so they dispense a lot of air very quickly and overinflate your tube before they even notice they've done anything.
A $5 pump probably isn't great but it should be fine for BMX tyres, which don't need to be inflated to high pressure.