I have a love hate relationship with my phone camera. On the one hand, it's great to have a camera that can shoot reasonable quality images (as well as HD video) in my pocket all the time. On the other hand, it also makes it easier to be lazy. After all, why bother picking up the camera and taking it when you've already got a camera in your pocket.
So the camera stays in the bag and you slowly get more and more used to just using the phone instead of the camera despite the camera being able to almost always product better results.
I don't have a small mirrorless camera. I have a dinosaur of a DSLR but I still don't find it unweildly to carry in a small bag when walking the dog or going to the pub. Sure, it's a little extra thing to carry but in general my EDC involves carrying a bag anyway so it's not a problem to make that bag a camera bag.
The camera starts up instantly and I usually leave it in a semi-auto mode that means I can shoot right away without having to mess with settings. Sure, occasionally I've left it on manual and gone to take a quick shot before the moment is gone and realized after that I'd completely balled it up. That's the exception though.