Big book quant question. How the answer is B? Cannot R be on X-axis, R(5,0) ? They have only mentioned that R is a point in the rectangular coordinate system. If they had given R is not on axis, then it would be true that B is correct answer. but they have not mentioned it.

It is clear from the diagram that R is not on the y axis, which negates the (5, 0) choice. See the GRE Math Conventions.

so if they are showing a point in a quadrant, tgen can we for sure assume that it cannot be on any axis?

where can i see these GRE math conventions?

Indeed, unless the question suggests otherwise. See above.

case 1: let, R coordinate is (x, y) then x is the horizontal distance, and y is the vertical distance of R, respectively. so x and y are perpendicular to each other forming a right triangle. in this right triangle we have given the hypotenuse OR = 5. so it would be 3-4-5 right triangle. so then, x coordinate either could be 3 or 4 and in both cases, quantity A is less than B.

Case 2: if this diagram is moveable in any direction either could be possible that the distance of x is positive or negative. as this diagram clearly indicates that it is not on the x-axis. so x coordinate can not be 5, could be less than that.