In "From Imperial China to Cyberspace: Contracting Without the State" I discussed the problem of contracts without court enforcement in the context of past practice in Imperial China and future practice online. There are at least two ways of doing it.
One is to rely on reputation. If both parties have sufficiently valuable reputations there is no problem, since the loss to either of being known to have broken its contract is more than the gain from doing so. All you have to do is to arrange things so that if one party does break the contract, the other can easily prove the fact to interested third parties. If only one party has a valuable reputation, the contract can be structured so that only that party has the opportunity to gain by failing to fulfill its obligations—most obviously, having the other party pay in advance. If neither party has a reputation, they can use a third party who does, an escrow agent or equivalent, to enforce the contract.
The other solution is to structure the contract in such a way that a party is always worse off pulling out than carrying his obligations to completion. In building a house, for instance, if the purchaser pays in advance the builder can pocket the money and leave. If the purchaser pays on completion, he can insist on "renegotiating" the price at that point. They solve the problem by a schedule of payments made as the house is built.
This may be difficult or impossible to do in a world of uncertainty, since we may not be able to predict how much a party will gain by fulfilling the contract or, alternatively, failing to do so. We can make reneging more costly to one party by having him put down a deposit in advance which the other can seize if the contract is not fulfilled. But doing so increases the incentive to the other party to reneg, since if he breaks the contract he can keep the deposit—all of this is happening in a world without courts to make him give it back. A solution to that problem is to replace the deposit with a hostage—something valuable to the party who gives it but not to the party who holds it. If I break the contract you destroy the hostage, but you cannot gain by yourself breaking the contract and holding the hostage.
Someone recently pointed me at a blog post proposing an elegant bilateral version of the hostage solution: "Contracts Without Trust and Third Parties", by Oleg Andreev. It is a very simple idea. I make a contract with you which gives you the opportunity to cheat me out of ten dollars, perhaps by accepting an online payment but not delivering the goods or services paid for. Before making the payment, we use bitcoin technology to deposit twenty dollars each in a form that is locked against both of us unless each of us releases it. If you never deliver the goods I refuse to unlock the deposit. I am worse off as a result but so are you, which is a good reason for you to fulfill your half of the contract.
My daughter, reading a draft of my article, pointed out a problem with the hostage solution. The hostage I deposit with you may be of no use to you but it is valuable to me, so after you break the contract you sell the hostage back to me for some substantial fraction of its value. The same problem exists for Andreev's proposal. After you fail to deliver the goods, you point out to me that leaving our deposit forever unclaimed hurts me as well as you, unlock your side of it, and wait for me to unlock mine.
Nonetheless, hostage have, historically, been used to guarantee fulfillment of promises in a variety of contexts, perhaps in part because selling my hostage back to me is a bilateral monopoly transaction with potentially high transaction costs and uncertain outcome, perhaps in part because someone who has just been cheated may be willing to accept a cost to punish the person who cheated him.
I have one improvement to offer to Andreev's scheme, in order to eliminate the deadweight cost of deposits that are never unlocked. Arrange that if the deposit is not unlocked after (say) ten days, it forfeits to a third party. Sell the right, in advance, to be that third party.