For Keybase, read Following, Server Security, Keybase and Bitcoin, Chat, and Chat cryptography.
For KBFS, read Understanding the Keybase File System and Crypto spec: the Keybase filesystem (KBFS) (feel free to merely skim the serious crypto aspects).
Keybase's explanations of the problems they solve, and how they solve them, are spread over many web pages. Here are some of the core issues that naming and key management systems face; look out for them as you read Keybase's documents. 1) How does a user manage their own devices and keys? How do they add a new device that acts as them? What if a device is stolen? What if a user loses/forgets everything and wants to still use their ID? 2) How does one user find out the correct ID for a specific other person? Among all the people with the same name, how do they know it's the ID of the right one? What if bad people are pretending to be the desired user, e.g. have registered with the same name? 3) Once I know someone's ID, how do I find out their current set of valid public keys? How do I find out if a user's old public key has been revoked? Can the service I ask about this lie to me, e.g. omit to tell me about a recent revocation, or a recent device/key addition? 4) How can I use the key material I learn to achieve private and authenticated interaction with other users?
When I first start using Keybase, I tell them my user name and public key. Does Keybase verify the correctness of that information? Or can anyone sign up for any name on Keybase?
Is Keybase better than SSL certificates, in which a certificate authority verifies identity (e.g. in which MIT verifies that I'm really rtm@mit.edu before giving me a certificate for that name signed by MIT)?
What is the point of involving a user's Twitter, Github, &c accounts?
What is the point of a user signing statements that they follow certain other users?
What has to happen when a user acquires a new device (e.g. smartphone) and wants to use it with Keybase? Could Keybase forge this operation, i.e. pretend that I've added a new device when I really haven't?
What are all the private keys involved, and where do they live?
Suppose I use Keybase on my iPhone while eating lunch, but leave my iPhone lying on the table when I leave. What sensitive Keybase-related secret key information does my iPhone likely contain? If someone takes my iPhone, what bad things could the thief do (related to my Keybase identity)?
What should a user do if a device that they had used with Keybase is stolen? What's the result in terms of information stored and served by Keybase?
After a user has revoked Keybase access for a stolen device, can the stolen device still read messages/files intended for that user, but created before the theft?
If I simultaneously lose all of the devices I use with Keybase, can I still use my keybase identity, on new devices?
How do applications use Keybase key material to sign and encrypt data meant for others to read?
How do applications decrypt and check signatures on data produced by other Keybase users?
For a given user A with multiple devices, does each of A's devices have a different public/private key pair? Or does A have just a single pair, and all of A's devices know the private key? If the former, and someone sends a private message to A, can A read (decrypt) that message on any of A's devices? How does that work?
Could Keybase trick user A into using the wrong public key to encrypt a message sent to user B?
Suppose user A asks Keybase to publish a revocation for one of A's old keys (because it might have been stolen). Then user B asks Keybase for information about user A. Could Keybase's servers leave out A's revocation statement, so that B continues to think the revoked key is valid?
What does Keybase gain by its use of Bitcoin?
Do Keybase ID names have global meaning? If keybase user A tells user B "please send a copy of the document to Keybase ID rtm", does "rtm" have the same meaning for B as it does for A? What prevents Keybase from handing out the same ID to two different people?
Does Keybase ID "x" retain its meaning over time (i.e. refer to the same person), even as user "x" changes keys and changes devices? If user "x" upgrades to an entirely new set of devices and device keys, how can someone else verify that it's still the same "x" as before? Will data sent to/from "x" before "x" upgraded still be decryptable and verifiable?
What are Keybase's teams?
Would it make sense to use KBFS as infrastructure for e.g. a decentralized Piazza?
Does KBFS guarantee to serve up fresh data? What would happen if a server tried to serve an outdated version of a file?
What about KBFS equivocating or forking?
If a co-worker and I are sharing files in /keybase/private/rtm,sam how can we allow a third person to read/write our files?
Is there a notion of least-privilege execution, so that my photo editor can look at /keybase/private/rtm/photos but cannot look at /keybase/private/rtm/mail?
KBFS allows conflicting updates to data shared read/write by multiple users or devices, and then merges the updates later. Will that be awkward for decentralized applications?
How do reads and writes work with respect to encryption?
What does Section 4.1.4 of the KBFS encryption document mean about Eve blackmailing Alice?
In Section 5, how does a client check that the server is returning the correct data? That is, what does a client do with the Merkle trees?
In Section 5.4, how does a client determine if a write to KBFS happened before of after a key is revoked?
In Section 6.1, how does Bob find all the relevant reader and writer blocks to which to add his new device's key?