Move and rotate two bodies (bound together with FixJoint)

Say I have two objects (A & B)
Object A: the hand.
Object B: the box I wanna grab.

It's ok to "grab" the box with the hand by using a FixJoint to bind them together. To "drop" the box is also easy by detaching the bodies from the joint.

The only problem I have now is to "move" and "rotate" the hand and the box together.
I move the hand by func setLinearVelocity. It works fine if I'm not grabbing anything. If I grab a box and move the hand, the box will also be "dragged" by the hand. When the hand stops, the box will remaining "drifting" a while. However, this is not what I want. I hope both of the bodies are "moved" in the same speed and remain in the same "orientation".

I don't want to apply force on the joint because I hope to extend the Object B to be a BIG object bound from several smaller box. (ex. three boxes bound together with 2 joints)

Maybe the way I want to achieve the hand grabbing ability is kinda stupid. I tried to search through the forum to find a better solution, but cannot find anything. Please suggest any good idea that can do the trick.

Thanks,
Mingyu

I have a pretty "tricky" solution to make the box not "drifting" while im moving the hand.

Set the mass of the hand to an extremely high value, so that it is hardly affected by the relatively "light" box.

The box now will only drift a bit, which makes sense if you put it into a "real" physics world.
But I still want to know a better solution for this.

Thanks for any feedback.