pyspark.sql.functions.to_variant_object#

pyspark.sql.functions.to_variant_object(col)[source]#

Converts a column containing nested inputs (array/map/struct) into a variants where maps and structs are converted to variant objects which are unordered unlike SQL structs. Input maps can only have string keys.

New in version 4.0.0.

Parameters

colColumn or str: a column with a nested schema or column name

Returns

Column: a new column of VariantType.

Examples

Example 1: Converting an array containing a nested struct into a variant

>>> from pyspark.sql import functions as sf
>>> from pyspark.sql.types import ArrayType, StructType, StructField, StringType, MapType
>>> schema = StructType([
...     StructField("i", StringType(), True),
...     StructField("v", ArrayType(StructType([
...         StructField("a", MapType(StringType(), StringType()), True)
...     ]), True))
... ])
>>> data = [("1", [{"a": {"b": 2}}])]
>>> df = spark.createDataFrame(data, schema)
>>> df.select(sf.to_variant_object(df.v))
DataFrame[to_variant_object(v): variant]
>>> df.select(sf.to_variant_object(df.v)).show(truncate=False)
+--------------------+
|to_variant_object(v)|
+--------------------+
|[{"a":{"b":"2"}}]   |
+--------------------+