Nullable vs optional: what a missing JSON field really means
A JSON field can fail to have a value in two different ways, and they don't mean the same thing:
{"id": 1, "nickname": null} // present, and explicitly null
{"id": 2} // absent: the key isn't there
APIs routinely use the difference: in a PATCH, "nickname":
null means "clear it" while an absent key means "don't touch it". A
type model that collapses the two states can't express that request, and a
parser that treats them as one will happily erase data.
TypeScript: two independent axes
TypeScript separates them cleanly, which is why it's the best place to see the distinction:
interface User {
nickname?: string; // may be absent; if present, a string
nickname: string | null; // always present; may be null
nickname?: string | null; // anything goes
}
? talks about the key, | null talks about the
value. They compose. When Schemint infers a model from sample payloads, a field
that's missing in some records gets ?, and a field that appears
with an explicit null gets | null; a field that shows
both gets both. If you're hand-writing the interface, pick deliberately: a
response model where the server always sends every key should use
| null and no ?, so a typo'd property access fails to
compile instead of quietly reading undefined.
Pydantic: Optional is about null, not absence
The naming trips everyone once: Optional[str] means
"str or None", it says nothing about the key being
optional. Whether a field may be absent is decided by having a default:
class User(BaseModel):
nickname: str | None # key REQUIRED, null allowed
nickname: str = "anon" # key optional, null NOT allowed
nickname: str | None = None # key optional, null allowed
The first form rejects {"id": 2} with a validation error, which
surprises people who read Optional as "may be missing". The third
form accepts both failure modes but can no longer tell them apart after
parsing, both land as None. When the difference carries meaning
(the PATCH case), Pydantic keeps the receipt:
patch = UserPatch.model_validate_json(body)
if "nickname" in patch.model_fields_set:
user.nickname = patch.nickname # explicit null clears the field
# absent key: leave it alone
model_fields_set holds the keys that actually appeared in the
input, so None-because-null and None-because-default
stay distinguishable.
Databases only have one kind of missing
A SQL column is either NULL or it isn't; there is no "absent".
So on the way into a table the two JSON states must merge, and the only
question is whether that merge is a decision or an accident. Generating models
from your real DDL keeps the boundary honest: a nickname VARCHAR(40)
nullable column becomes nickname?: string | null in the interface
and str | None = None in the Pydantic model, and anything stricter
is the API layer's own promise, stated in its own schema.
Rules that hold up
- Response models: every key present, nullable where the data is nullable.
| null, no?. - PATCH bodies: everything optional, and read
model_fields_set(or its equivalent) instead of comparing toNone. - Never encode "missing" as an in-band value (
"",0,"null"). You'll meet it again in a report. - When inferring types from samples, feed several records; one payload can't show you which fields come and go.