grammar - "Shouldn't you be at work?"

Say you should be at work. Would the correct answer be "yes" or "no"? The common answer would be "yes", but isn't that really saying, "yes, I should not be at work"?