Making a slingshot

I'm having some difficulties properly using input in combination with Ogrenewt to make a sling shot like game.

My key issue is the important aspect of the slingshot itself.

I have a sphere body where I set its initial position orientation just mine and I make autofreeze false.

Now what I want to do for starters and simplicity is when the user holds down the left mouse button, the ball with move towards the camera in the Positive Z.
(later I will add the elastic bands or w/e I use visually).

And then when the user releases, I would call my respective custom force and torquecallback that would throw the ball.

Currently what I do is when the user moves the mouse some such that in the MouseMoved listener for OIS I increase a factor.
Then when I release the mouse I call the callback which sets the velocity of the body to a certain orientation * the factor.

This works ok when launching the ball, but I'm not sure how to best approach the moving of the sphere/ball when I'm winding up the "slingslot".

I've tried adjusting its position or velocity or adding force when the mouse click is initially held down, but its not working as I expected.