How Avro encodes data
The binary primitives used to build the .avro file — all implemented in JavaScript.
Zig-zag varint (long/int)
Integers are encoded with zig-zag encoding then as a variable-length base-128 varint. Small integers take 1-2 bytes regardless of sign. Large integers take up to 10 bytes.
32 → zigzag(64) → [0x40]
-1 → zigzag(1) → [0x01]
String (UTF-8)
Strings are encoded as a zig-zag varint length prefix followed by raw UTF-8 bytes. The length is the byte count, not character count.
"hi" → varint(2) + [0x68, 0x69]
Double (IEEE 754)
Doubles are encoded as 8 bytes, little-endian IEEE 754 format — the same as JavaScript's native float64. No transformation needed.
3.14 → 8 bytes LE float64
Union ["null", type]
Each CSV field is a nullable union. Empty cells → varint(0) (null branch). Values → varint(1) + encoded value. This allows any field to be absent.
null → [0x00]
"foo" → [0x02] + encStr("foo")
How the Avro OCF binary is built
1
Schema generation + OCF header
CSVShift infers an Avro record schema with nullable union fields (["null", "string"], ["null", "long"] etc.) from the CSV columns. The schema JSON is embedded in the Avro file metadata map, preceded by the 4-byte magic Obj\x01 and followed by a 16-byte random sync marker.
2
Record serialisation with varint
Each CSV row is serialised as an Avro record. For each field, the union branch index is written as a zig-zag varint (0 for null, 1 for value). Then the value itself — string as length + UTF-8 bytes, long as zig-zag varint, double as 8-byte LE float64, boolean as single byte.
3
Data block + EOF
All serialised records are wrapped in an Avro data block: record count (varint) + byte size (varint) + record bytes + sync marker (16 bytes). A final empty block (varint 0) marks the end of file. The output is a standard Avro OCF readable by any Avro-compatible tool.
Read the Avro file in Python
Python — fastavro (recommended)
import fastavro
# Read all records:
with open('output.avro', 'rb') as f:
reader = fastavro.reader(f)
schema = reader.schema
records = list(reader)
print(f"Schema: {schema}")
print(f"First record: {records[0]}")
# Convert to pandas:
import pandas as pd
df = pd.DataFrame(records)
print(df.head())
Requires: pip install fastavro. fastavro reads the embedded schema automatically — no need to supply it separately. It is 10-100x faster than the official Apache Avro Python library.
Python — convert CSV to Avro with fastavro
import csv, fastavro
schema = fastavro.parse_schema({
'type': 'record',
'name': 'CSVRecord',
'namespace': 'com.example',
'fields': [
{'name': 'name', 'type': ['null', 'string'], 'default': None},
{'name': 'age', 'type': ['null', 'long'], 'default': None},
{'name': 'score', 'type': ['null', 'double'], 'default': None},
]
})
with open('data.csv') as f_in, open('output.avro', 'wb') as f_out:
rows = [
{'name': r['name'], 'age': int(r['age']), 'score': float(r['score'])}
for r in csv.DictReader(f_in)
]
fastavro.writer(f_out, schema, rows)
For full type control, define the schema explicitly in Python. CSVShift generates the same schema structure — copy the schema from the result preview.
Apache Spark — read Avro
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("avro").getOrCreate()
# Read Avro file (Spark 3.x):
df = spark.read.format("avro").load("path/to/output.avro")
df.show()
df.printSchema()
Spark 3.x includes built-in Avro support. For Spark 2.x: add spark-avro JAR and use .format("com.databricks.spark.avro").
When do you need CSV to Avro?
Apache Kafka data pipelines
Kafka producers and consumers in data pipelines use Avro with Confluent Schema Registry for schema enforcement. Converting CSV data to Avro produces messages compatible with Kafka's Avro serializer without writing a producer script.
Hadoop and data lake ingestion
HDFS, S3 and Azure Data Lake Store data lakes often use Avro for raw data storage — especially in streaming ingestion pipelines from Kafka or Flume. Converting CSV exports to Avro produces files compatible with Hive external tables and Spark jobs.
Schema evolution
Unlike CSV, Avro stores the schema with the data. When downstream consumers need to handle schema changes (adding or removing columns), Avro's schema evolution rules handle backward and forward compatibility — making it more robust than CSV for long-lived pipelines.
fastavro workflows
Python data engineers using fastavro for reading Avro files from S3 or HDFS can use CSVShift to create test fixtures and seed data in Avro format — matching the schema of production files without setting up a full Avro producer.
Related CSV tools
Popular searches
csv to avro converter
convert csv to avro
csv to avro online
csv to avro python
csv to avro free
fastavro csv to avro
csv to avro spark
csv to avro kafka
convert csv to apache avro
csv to avro schema
avro file from csv
csv to avro hadoop
Avro OCF written
in your browser. Zero libraries.
CSVShift writes the Apache Avro Object Container File format directly in JavaScript — zig-zag varint encoding, Avro map encoding for file metadata, nullable union types, random sync marker. No fastavro, no avro-js, no WebAssembly. The output is byte-compatible with files produced by the official Apache Avro SDK.
Standard Avro OCF format
Magic "Obj\x01" + metadata map + sync marker + data block — the full Object Container File specification.
Nullable union types
All fields are ["null", type] unions — empty CSV cells become Avro null, values are typed correctly.
Schema in result preview
The generated Avro schema JSON is shown in the result — copy it for use in fastavro writers or Schema Registry.
Free, no conditions
No row limit, no watermark, no account. Funded by display advertising only.
How do I convert CSV to Avro?
Upload your CSV, set the record name and namespace if needed, enable type detection for numeric columns, and click Create Avro File. Download the .avro and verify with fastavro.reader(open('f.avro','rb')) in Python.
What is Apache Avro and why is it used?
Avro is a binary serialisation format from the Apache Hadoop project. It stores the schema alongside the data (unlike Parquet which stores it in a footer), making every Avro file self-describing. It is the standard format for Apache Kafka messages and Confluent Schema Registry, used in data streaming pipelines where schema evolution and backward compatibility matter.
How do I read the Avro file in Python?
import fastavro; records = list(fastavro.reader(open('f.avro','rb'))). Requires pip install fastavro. The schema is read automatically from the file — no need to provide it separately. Convert to pandas: import pandas as pd; df = pd.DataFrame(records).
Is my data safe when converting CSV to Avro online?
Yes. The Avro binary is created entirely in your browser. Your CSV is never uploaded. Open the Network inspector during conversion — zero outbound data requests are made.
