Along with Earwicker's reply, which I generally concur with.
Additionally, I would posit that it is rather appropriate because & is the "address-of" operator.
Since sending a reference frequently resembles passing an address rather than a value.
Additionally, taking a variable's address is frequently referred to as "referencing."
(Yes, I am aware that references do not necessarily need to be implemented via pointers; I am referring to the way they function theoretically.) But this is just conjecture.