Exception thrown on raycastClosestBounds

I want to be able to translate my actor locations. So I am using raycastClosestBounds, casting a ray from a given face of the actor to check if it is touching another actor in the scene, thus telling me if I should translate the actor in that direction or not.

However, I noticed whenever I cast a ray to return the closest actor the ray hits, and there are no actors for it to hit, I get an exception.

I'm looking at the source, it's thrown on the second reference to NxShape(Memory access violation). I'm assuming NxShape is pointing to a region of memory that cannot be read. Further I can assume that mScene->raycastClosestBounds() is returning null, as in, nothing hit by ray(although I haven't checked s oI may be wrong.) I'm tempted to go into the source and override the method(Where it returns a int(-1 for fail, 1 for success), requring a parameter referencing a RaycastHit object to be altered according to the most recent hit)

PhysXPointer* shapePointer = pointer_cast(nxShape->userData);

RaycastHit Scene::raycastClosestBounds(const Ray& ray, Enums::ShapesType type, unsigned int group, Real maxDistance, unsigned int hintFlags) const
{

NxRay inRay;
Functions::XYZ<Vec3, NxVec3>(ray.mDirection, inRay.dir);
Functions::XYZ<Vec3, NxVec3>(ray.mOrigin, inRay.orig);

NxRaycastHit hit;
NxShape* nxShape = mScene->raycastClosestBounds(inRay, NxShapesType(int(type)), hit, group, maxDistance, hintFlags, 0);

RaycastHit out;
out.mDistance = hit.distance;
out.mFaceID = hit.faceID;
out.mFlags = hit.flags;
out.mInternalFaceID = hit.internalFaceID;
out.mMaterialIndex = hit.materialIndex;
out.mU = hit.u;
out.mV = hit.v;
Functions::XYZ<NxVec3, Vec3>(hit.worldImpact, out.mWorldImpact);
Functions::XYZ<NxVec3, Vec3>(hit.worldNormal, out.mWorldNormal);

PhysXPointer* shapePointer = pointer_cast(nxShape->userData);
out.mShape = shapePointer->get<Shape>();
out.mRigidBody = shapePointer->getParent<RigidBody>();

return out;
}

I'm pretty sure this is a bug that needs to be addressed. I'm looking at the physX documentation describing the NxRaycastHit structure. The NxOgre::Scene::raycastClosestBounds() should be checking the NxRaycastHit::flags to check which members in the NxRaycastHit structure are valid, and which are not. In the case of this bug, the NxRaycastHit::shape is not a valid member(it has also been zeroed, which is why I am getting memory access violations, I can't read for 0x00000000+offset).

The NxOgre::RaycastHit::shape is being used without checking the flags to assure it is a valid member.

I would like to go into the source and fix it, but I'm not sure what legal restrictions I may have, as I haven't reviewed any legal documents on NxOgre.

The bug exists in NxOgreScene.cpp, the function which contains it it on line 518(Scene::raycastClosestShape). Read the above for information on my suggested resolution. You should check to see if the flag (enum)NxRaycastBit::NX_RAYCAST_SHAPE is set in NxRaycastHit::flags, if it is, then the shape member is valid, otherwise it isn't, and it should not be used.

It was done incorrectly in Scene::raycastClosestShape as well(although it works), however, in this case you check to see if the pointer is null, opposed to checking the flags.

For now, I'll just write a cRayCasting class to handle raycasting without NxOgre, until the bug is fixed.

Quote from PhysX documentation:

enum NxRaycastBit

Specifies which members of NxRaycastHit which should be generated(when used as hint flags for raycasting methods) or which members have been generated when checking the flags member of NxRaycastHit.

Enumerator:
NX_RAYCAST_SHAPE "shape" member of NxRaycastHit is valid
NX_RAYCAST_IMPACT "worldImpact" member of NxRaycastHit is valid
NX_RAYCAST_NORMAL "worldNormal" member of NxRaycastHit is valid
NX_RAYCAST_FACE_INDEX "faceID" member of NxRaycastHit is valid
NX_RAYCAST_DISTANCE "distance" member of NxRaycastHit is valid
NX_RAYCAST_UV "u" and "v" members of NxRaycastHit are valid
NX_RAYCAST_FACE_NORMAL Same as NX_RAYCAST_NORMAL but computes a non-smoothed normal.
NX_RAYCAST_MATERIAL "material" member of NxRaycastHit is valid


My fix(I didn't bother fixing it for Scene::raycastClosestShape, although it's pretty much the same thing.):

RaycastHit Scene::raycastClosestBounds(const Ray& ray, Enums::ShapesType type, unsigned int group, Real maxDistance, unsigned int hintFlags) const
{

NxRay inRay;
Functions::XYZ<Vec3, NxVec3>(ray.mDirection, inRay.dir);
Functions::XYZ<Vec3, NxVec3>(ray.mOrigin, inRay.orig);

NxRaycastHit hit;
NxShape* nxShape = mScene->raycastClosestBounds(inRay, NxShapesType(int(type)), hit, group, maxDistance, hintFlags, 0);

RaycastHit out;
out.mDistance = hit.distance;
out.mFaceID = hit.faceID;
out.mFlags = hit.flags;
out.mInternalFaceID = hit.internalFaceID;
out.mMaterialIndex = hit.materialIndex;
out.mU = hit.u;
out.mV = hit.v;
Functions::XYZ<NxVec3, Vec3>(hit.worldImpact, out.mWorldImpact);
Functions::XYZ<NxVec3, Vec3>(hit.worldNormal, out.mWorldNormal);

if(hit.flags & NxRaycastBit::NX_RAYCAST_SHAPE)
{
PhysXPointer* shapePointer = pointer_cast(nxShape->userData);

out.mShape = shapePointer->get<Shape>();
out.mRigidBody = shapePointer->getParent<RigidBody>();
} else {
out.mShape = 0;
out.mRigidBody = 0;
}

return out;
}

Although the above fix hasn't been tested, and I can't test it because my computer is messing up(it really needs a reformat, there's a virus in here that isn't being picked up..)

I finally got a chance to test this, it turns out PhysX isn't setting the flags as described in the documentation(Or I'm missing something really big). So you'll just have to check for a null pointer.

Check the PhysX document yourself. The flag which represents whether the shape member is valid, is set, even when mShape = 0x00000000.