I carried a 640 in the right front pocket for 8 or 10 years as daily carry including while doing carpentry and construction work, no holster, and with a speed loader in the pocket with it. The speed loader seems to help keep the gun positioned. I rarely had to adjust the guns position, mostly when crawling around in a crawl space or somethings similar. I started using a milk jug plastic holster some, but mainly to reduce pocket wear, I prefer without. I havent tried cargo pockets, I wore carhartt canvas work pants most of the time.
I retired the 640 when I got a 1980s Colt Agent, which I carry the same way.