Another thing to think about is that the recoil with all the pistol caliber rifles, using cowboy loads, will be light, so not much difference there, either.
As to your original final question...yes, it is completely acceptable.
As one who started with mixed calibers, I
can tell you that it is best to have two very different calibers, such as the 357 and the 44, (and not a 44 and a 45, or 45 and 44-40, etc.) if you are going to mix calibers, as it will be much easier to tell them apart at a glance or by feel.
P.S. ... the 357 ain't such a bad (small to medium) deer rifle if used within it's limitations, either. Check out postings on
www.marlinowners.com for more info, if interested.