Glonce

Go 1.27 makes the new encoding/json/v2 API stable

toolsThe Glonce editorsHacker News🔥71Glonce rating71Impact55Novelty70Relevance70Surprise35Players55Credibility95

The Go Blog announces that the new encoding/json/v2 and encoding/json/jsontext packages are stable and generally available in Go 1.27. Last year the team wrote about them as a new experimental API for JSON; after additional feedback and refinement, they are now out of the experiment. The post is an updated version of the article that introduced the experiment. It describes the problems with encoding/json (called v1) and introduces the new packages.

The Go team's verdict on v1 is that it has held up well overall: marshaling and unmarshaling arbitrary Go types with a default JSON representation, plus the ability to customize it, has proven highly flexible. But users have identified shortcomings in three groups.

Behavior flaws. v1 handles JSON syntax imprecisely, while decoders elsewhere have become stricter about ambiguous input. It accepts invalid UTF-8, although RFC 8259 requires valid UTF-8; the post says the default should report an error instead of causing silent data corruption downstream. It accepts objects with duplicate member names. RFC 8259 does not say how to treat them, so an implementation may pick a value, merge, discard or fail. The post says this could be exploited by attackers in security applications and has been exploited before (CVE-2017-12635), so the default should reject duplicates. v1 also marshals a nil slice or map as JSON null, which can break other implementations that refuse null where an array or object is expected; a survey indicated most Go users prefer an empty array or object by default. Unmarshaling matches object member names to struct fields case-insensitively, which the post calls a surprising default, a potential security vulnerability and a performance limitation. Finally, MarshalJSON methods declared on a pointer receiver are called inconsistently. That is regarded as a bug, but it cannot be fixed because too many applications depend on it.

API deficiencies. Unmarshaling correctly from an io.Reader is hard: the common json.NewDecoder(r).Decode(v) fails to reject trailing junk. Options can be set on Encoder and Decoder but not on the Marshal and Unmarshal functions, and custom Marshaler and Unmarshaler implementations cannot use options or pass them down the call stack; Decoder.DisallowUnknownFields, for example, loses its effect inside a custom UnmarshalJSON. Compact, Indent and HTMLEscape write to a bytes.Buffer rather than a []byte or io.Writer.

Performance limits. MarshalJSON forces the implementation to allocate the returned []byte, and encoding/json must verify the result is valid JSON and reformat it to the requested indentation. UnmarshalJSON needs a complete JSON value, so the package parses the value once to find its end and the method parses it again. Encoder and Decoder buffer the entire JSON value in memory despite working on an io.Writer or io.Reader; Decoder.Token is allocation-heavy and there is no token-writing counterpart. If a MarshalJSON or UnmarshalJSON method recursively calls Marshal or Unmarshal, performance becomes quadratic.

Why a new package. The post calls a new incompatible major version a heavy consideration and says fixing the existing package would be preferable. But these problems are inherent consequences of the existing API and practically impossible to fix within the Go 1 compatibility promise. Adding names such as MarshalV2 would be tantamount to a parallel namespace inside the same package, which leads to encoding/json/v2.

The jsontext foundation. JSON serialization is split into syntax and semantics. "Encode" and "decode" describe syntactic work; "marshal" and "unmarshal" describe the mapping between JSON values and Go values. The jsontext package handles only syntax and does not depend on Go reflection; its name comes from the JSON-text grammar in RFC 8259, section 2. It provides Encoder (WriteValue, WriteToken) and Decoder (ReadValue, ReadToken), built with NewEncoder and NewDecoder, which take variadic options. Unlike the v1 types, they keep syntax and semantics apart and operate in a truly streaming manner. A Value is a complete JSON unit, represented as a named []byte, identical to v1's RawMessage, which is now a type alias to jsontext.Value. A Token is an opaque type designed to represent any JSON token without allocation. To fix the MarshalJSON and UnmarshalJSON performance problems, v2 adds MarshalJSONTo and UnmarshalJSONFrom, which work on an Encoder or Decoder so implementations can process JSON in a purely streaming way. The json package then no longer has to validate or format a returned value, or find the boundaries of a provided one.

The v2 API. The package is designed to fix v1's problems while staying familiar, and the stated goal is that v1 usages will operate mostly the same if directly migrated to v2. Its basic functions are Marshal, MarshalWrite and MarshalEncode, and Unmarshal, UnmarshalRead and UnmarshalDecode. Marshal and Unmarshal look like v1 but accept options. MarshalWrite and UnmarshalRead work directly on an io.Writer or io.Reader, with no temporary Encoder or Decoder. MarshalEncode and UnmarshalDecode take a jsontext.Encoder or Decoder and are the underlying implementation of the others. Unlike v1, options are a first-class argument to each of these functions, and the post says several v2 options are not covered in the article. For custom types, the Marshaler and Unmarshaler interfaces are identical to those in v1, and v2 adds MarshalerTo and UnmarshalerFrom, which let a type write or read itself as JSON through a jsontext.Encoder or Decoder. The post points readers to the examples in the v2 package, the "Working with JSON" tutorial, and Anton Zhiyanov's blog for usage.

Key facts

  • encoding/json/v2 and encoding/json/jsontext are stable and generally available in Go 1.27, after last year's experimental release was refined based on feedback.
  • The Go team lists v1 flaws: it accepts invalid UTF-8 and duplicate member names (cited exploit: CVE-2017-12635), marshals nil slices and maps as null, and matches struct fields case-insensitively.
  • These problems are practically impossible to fix within the Go 1 compatibility promise, which is why the fix is a separate v2 package.
  • jsontext handles JSON only at the syntactic level, without Go reflection, and offers streaming Encoder and Decoder types; v2 adds MarshalJSONTo and UnmarshalJSONFrom for streaming custom types.
  • Options become a first-class argument to every marshal and unmarshal function, and the stated goal is that v1 usages operate mostly the same when migrated to v2.

Why it matters

encoding/json has been in the Go standard library since well before Go 1.0, and the post calls JSON the most popular data format on the internet, widely read and written by Go programs. The new API replaces a package whose flaws could not be repaired in place because of the Go 1 compatibility promise. With Go 1.27 the v2 packages stop being an experiment. The fixes the post names span correctness (invalid UTF-8, duplicate names, trailing junk), security (duplicate names, case-insensitive matching) and performance (streaming, no double parsing, no quadratic recursion).

Who it affects

Any Go programmer who reads or writes JSON, which the post describes as widely done. The groups with the most to gain are those who write custom MarshalJSON or UnmarshalJSON methods, those who decode from an io.Reader, and those who handle security-sensitive input where duplicate member names or invalid UTF-8 matter. Programs that talk to JSON implementations rejecting null for arrays or objects are also affected by the nil slice and map behavior.

How to use it

The new packages are encoding/json/v2 (package name json) and encoding/json/jsontext, available in Go 1.27. For most code, the post says direct migration from v1 should behave mostly the same. The main entry points are Marshal, MarshalWrite and MarshalEncode for output, and Unmarshal, UnmarshalRead and UnmarshalDecode for input, each accepting options. Custom types can keep the v1 Marshaler and Unmarshaler interfaces, which are identical in v2, or implement MarshalerTo and UnmarshalerFrom to stream through a jsontext.Encoder or Decoder. For learning material the post recommends the examples in the v2 package, the "Working with JSON" tutorial, and Anton Zhiyanov's blog on the topic.

How solid is it

This is the official Go Blog, written by the team behind the package, and it states the packages are stable and generally available in Go 1.27 after a year of feedback and refinement. It is an updated version of the article introducing the experiment. The claims about v1 flaws are specific and tied to RFC 8259 and a named CVE. The post gives no benchmark numbers or measured performance gains in the portion available, so the performance claims rest on the described API design rather than on measurements. The article text available ends partway through the v2 API description, so default behaviors, struct tags, options and migration details are not covered here.

Risks and caveats

The post's goal is that v1 usages operate mostly the same under v2, which is not the same as identical. Behavior changes aimed at safety, such as rejecting duplicate names and invalid UTF-8 by default and different nil slice and map output, are exactly the kind of difference a migration can run into. The post also says several v2 options are not covered in the article. The visible text does not say whether v1 will be deprecated or removed. It also gives no release date for Go 1.27.

“Our goal is that usages of v1 will operate mostly the same if directly migrated to v2.”

— The Go Blog