How to Use Custom Variables with PayPal Instant Payment Notification (IPN)

Ever wondered how you can get a record of PayPal transactions as they happen in real-time and store them in a database? What about passing along custom variables to the PayPal form, and then catching them after the payment has been completed - such as for a referral system?