So, RLPx is a protocol suite. It mainains the p2p overlay network, with the means of a modified version of the kademlia node discovery protocol (using UDP messages). We could name it RLPx node discovery protocol. The first time two peers meet each other they should perform a two-phase handshake (i.e. two different handshakes).
In the first handshake they exchange some cryptographic secrets (the details are explained here (in the rlpx specification)) that are used for the subsequent communications between them. The subsequent messages are therefore encrypted and authenticated.
In the second handshake they exchange their capabilities, i.e. which devp2p subprotocols they support (the peers can support multiple protocols). This second handshake could be considered already part of devp2p wire protocol (the Hello message). Therefore RLPx messages are encrypted and authenticated and on top of them you can communicate with different protocols. We can say that RLPx is the transport layer protocol and devp2p and its subprotocols are the application level protocols.