Certificate authentication is technically a part of the public-key authentication method. The signature created with the private key and the verification of the signature using the public key (contained in the X.509 certificate when doing certificate authentication) are done identically with conventional public keys and certificates. The major difference is in determining whether a specific user is allowed to log in with a specific public key or certificate. With conventional public keys, every server must have every user's public key, whereas with certificates the users' public keys do not have to be distributed to the servers - distributing the public key of the CA (self-signed certificate) is enough.
In brief, certificate authentication works as follows:
The client sends the user certificate (which includes the user's public key) to the server. The packet also contains data unique to the session and it is signed by the user's private key.
The server uses the CA certificate (and external resources as required) to check that the user's certificate is valid.
The server verifies that the user has a valid private key by checking the signature in the initial packet.
The server matches the user certificate against the rules in the server configuration file to decide whether login is allowed or not.