When there is a problem, something must be done--more precisely, those deemed responsible must be seen to be doing something. What is needed is some clearly visible action. Ideally one does something that everyone will notice but that will not impose sufficient costs on any reasonably well organized group to cause political problems to those doing it. The policy I observed met those requirements. It might be a minor nuisance for the restaurant and a minor inconvenience for patrons, but nobody was seriously inconvenienced and everyone could see that the water problem was being dealt with.
Unfortunately, one requirement that such policies do not face is that they actually do anything to solve the problem. I suspect that very few restaurant owners or patrons bothered to do the arithmetic to see how important the waste being prevented was in the overall scheme of things. I would not be surprised if the people who instituted the policy didn't bother either.
The calculation is pretty simple. I don't have figures for San Jose, but a quick google produces a figure for U.S. daily water consumption of about 400 billion gallons--well over a thousand gallons a day per capita, most of it used for irrigation or cooling power plants. If we estimate, I think generously, that the average person eats at the sort of restaurant that puts filled water glasses on the table one day out of five, that half of the customers don't want the water, and that the average glass holds a cup, the policy saves about a tenth of a cup of water per capita per day, reducing total demand by about one part in 200,000.
But something is visibly being done, which is the important thing.