Bicycle helmets contain crushable foam that works to extend the duration of impact by about 6 ms (milliseconds). This doesn't make the impact force disappear, instead it extends the duration of force experienced by the brain. By extending the duration, you reduce the peak force. The brain can withstand impacts to some degree, however if the impact is too forceful over too short a period of time (like your head hitting asphalt), the peak force experienced by your brain becomes too high resulting in brain damage or even death. While 6 ms may seem small, it can be enough to reduce the peak force experience from lethal and/or damaging to survivable and/or reduced damage (see Figure 1).
A)
B)
Figure 1. Impact force by time a brain sees with a helmet (A) and without a helmet (B). The total energy is the same (area under the curve), but the peak force is lower for the brain with a helmet.
(Source: http://www.bhsi.org/)
Modern military helmet are designed to protect against things like shrapnel, not absorb impacts (although newer research may change that). As a result your peak impact force in a crash will not substantially change and you will be under a similar risk for brain injury as being helmet less.
Also you mentioned having to replace bike helmets that take an impact as a downside. If you wonder why bicycle helmets are not made with foam that "bounces back" (e.g. hockey helmets) this is because it can cause more damage in an impact. With this type of foam your brain receives force in the initial hit and then it experiences a secondary "hit" as the foam returns (often these non-crushable foams return rapidly). Crushable foam however does not rebound as it remains crushed. The downside is that once it is crushed it no longer provides any protective benefit and needs to be replaced. By extension when the crushable foam degrades it also needs to be replaced. This is why it is also suggested that old bicycle helmets also need to be replaced (what constitutes old is up for debate). In this case the crushable foam has degraded over time and its impact absorption properties change for the worse and will not provide the same benefit in an impact as a newer helmet with foam that has not degraded.
Finally, I do remember hearing about a new type of foam that behaves similar to the crushable foam used in helmets, but rebounds slowly. I will have to work to find that reference. But if they are in the marketplace they will be more robust to hits and repeat impacts.