There are a lot of risks involved, so make sure you know what you are doing. It should work and is a common solution to the problems you describe.
Some risks that come to mind:
- make sure the PSU voltage is set at a level that the battery is comfortable with. for example the long-term storage voltage for lipo’s is 3.8V per cell.
- make sure the battery does not discharge through the PSU.
- Ensure the battery limits are respected
- watch-out for short circuits. add a fuse to the battery.