Data API Reference
The exaloop.data API provides arrays, tables, file readers, and native buffer
exchange. Signatures below assume import exaloop.data as data. For complete
programs and the conversion model, see the Data guide.
A type parameter such as T is a Codon type, for example int, float, or
Optional[str]. Codon resolves it at compile time; the API checks the incoming
column's schema at runtime.
Construction
data.array(values: list[T], copy: bool = True) -> Array
data.array(values: np.ndarray[T, 1], copy: bool = True) -> Array
data.table(columns: dict[str, Array], num_rows: int = -1) -> Table
array() accepts a typed list or a one-dimensional NumPy array. NumPy input is
suited to fixed-width numeric and boolean data. Typed lists also represent text,
bytes, nested lists, and optional elements, which are converted to Arrow's
corresponding buffer layouts.
table() assembles named columns of equal length. The default num_rows=-1
infers that length (zero for an empty table); an explicit row count must agree
with every column. Names are UTF-8 without embedded NULs. A dictionary naturally
has unique names, although tables read from files may contain duplicates.
| Native type | Arrow representation produced |
|---|---|
NoneType | Null |
bool | Packed boolean |
i8/u8, i16/u16, i32/u32 | Corresponding signed/unsigned integer width |
int, i64, u64 | 64-bit integer |
float16, float32, float | 16/32/64-bit floating point |
str, bytes | Large UTF-8 / large binary |
list[T] | Large list with recursively constructed child |
Optional[T] | Nullable underlying column |
Construction validates the resulting Arrow data, even when the input is empty. For shared input storage, see ownership and copies.
Storage and Copies
| Operation | Storage behavior |
|---|---|
data.array(values) | Snapshot by default |
data.array(values, copy=False) | Borrows fixed-width numeric list or contiguous ndarray storage |
| Noncontiguous ndarray input | Creates contiguous storage first |
| Boolean, text, nullable, or nested input | Converts layout even with copy=False |
item, to_list, to_numpy, to_numpy_masked | Native outputs are independent of Arrow storage |
| Slices, extracted columns, C exports | Retain storage independently of the parent |
| Multi-chunk column extraction | Combines chunks, potentially copying |
to_bytes | Serialized output without a final copy |
from_bytes | Retains input; decoding or decompression can allocate |
Borrowed inputs must remain unchanged while Arrow uses them. Sharing an array does not make concurrent mutation or closing safe.
Array
An Array represents one logical column. Conversion methods return owned Codon
values; slicing and C Data export retain the Arrow storage instead.
| Member | Return / contract |
|---|---|
len(column) | Number of logical elements |
column.format | Arrow C Data format string; dictionary columns report their index format |
column.null_count | Physical null count, not decoded dictionary-value nulls |
column.is_null(index: int) | Physical validity; supports negative indices |
column.item(index: int, T) | One native value; negative indices supported |
column.to_list(T) | Independent, resizable list[T] |
column.to_numpy(T) | Independent 1D numeric ndarray; rejects nulls |
column.to_numpy_masked(T) | (numeric_values, boolean_missing_mask); missing data slots are zero |
column.cast(T) | New array using Arrow's safe cast; target is a supported primitive type, not a nested list/optional specification |
column.slice(offset: int, length: int = -1) | Shared-storage view; -1 means remaining length |
column.export_c() | New CData owner with populated ABI structs |
Array.from_c(data: CData) | Consumes a live C export into a new array |
column.close() | Idempotently closes this handle and its aliases |
with column as entered: | Checks the handle and closes it on exit |
item/to_list accept exact primitive, text/binary, NoneType, and nested
list/optional targets. NumPy methods accept only primitive numeric/bool types.
String/binary conversion accepts both 32-bit and 64-bit offset encodings.
List conversion accepts normal, large, and fixed-size lists. Dictionary indices
of every supported integer width are decoded before value conversion.
slice(offset, length) selects a contiguous region, with a nonnegative offset
and a length within the array's bounds. Leaving length=-1 selects the rest of
the array. This method does not support strides or Python's negative slice
indices.
Table
Tables provide named or positional access to columns. A name lookup is convenient for ordinary schemas; use a column index when a file contains duplicate names.
| Member | Return / contract |
|---|---|
len(dataset), dataset.num_rows | Row count |
dataset.num_columns | Column count |
dataset.column_names | Independent list[str] in schema order, including duplicates |
dataset.column(index: int) | Array; supports negative indices |
dataset.column(name: str) | Array; name must identify exactly one column |
dataset[key] | Same positional/name lookup as column |
dataset.slice(offset: int, length: int = -1) | Shared-storage row slice with bounds checks |
dataset.write(path: str, format: str = "", batch_size: int = 65536) | Writes a local file or S3 object; infers format from extension when empty |
dataset.write_avro(path, writer_schema, codec="null", block_size=65536, batch_size=65536, max_block_bytes=67108864) | Writes Avro with an explicit Schema; see table writers |
dataset.to_bytes(format: str = "ipc", batch_size: int = 65536) | Serialized bytes |
Table.from_bytes(data: bytes, format: str = "ipc") | New table retaining the input bytes as needed |
dataset.close() | Idempotently closes this handle and its aliases |
with dataset as entered: | Checks the handle and closes it on exit |
Column extraction combines all of a column's chunks, so it can allocate rather than simply return a view. For dictionary columns, combining chunks may widen the index type to 64 bits when the dictionaries cannot otherwise be unified. The logical values are preserved, including the distinction between a null index and an index pointing to a null dictionary entry.
File Readers
data.read_table(path: str, format: str = "") -> Table
data.read_ipc(path: str) -> Table
data.read_feather(path: str) -> Table
data.read_parquet(path: str) -> Table
data.read_orc(path: str) -> Table
data.read_avro(path: str) -> Table
data.read_csv(path: str) -> Table
data.read_json(path: str) -> Table
Convenience readers specify the format, regardless of filename extension.
read_table infers it when format is empty. Use read_table(path, "stream")
for IPC streams. Format names are case-insensitive. Supported names and
format-specific restrictions are listed in the guide.
For Avro projection, bounded batches, schema resolution, and generated records,
see the Avro API.
Readers and writers also accept an optional filesystem (Optional[S3FileSystem],
default None). Omit it for ordinary calls: plain paths are local and
s3:// paths use the lazily cached default AWS client. With a filesystem, both
s3://bucket/key and raw bucket/key paths refer to objects on that service.
Only s3:// URIs are supported; credentials, ports, queries, and fragments in
URIs are rejected. Percent-decode keys only through URI syntax, not raw paths.
S3 key segments equal to . or .. are rejected, including percent-encoded
forms in URIs, to prevent HTTP path normalization from changing the target object.
Other dots in names are allowed; local filesystem paths are unaffected.
S3FileSystem
S3FileSystem(region="", endpoint_override="", scheme="https",
access_key="", secret_key="", session_token="",
anonymous=False, role_arn="", session_name="", external_id="",
load_frequency=900, force_virtual_addressing=False,
connect_timeout=3.0, request_timeout=30.0, max_attempts=3,
tls_ca_file_path="")
s3_available() -> bool
filesystem.close()
| Option | Contract |
|---|---|
region | Empty uses AWS SDK region resolution; supply the bucket's region when known |
endpoint_override | Optional S3-compatible host and port; no scheme, credentials, or path |
scheme | https by default; http for trusted test services |
access_key, secret_key, session_token | Explicit credentials; both keys are required, token optional; these values do not refresh automatically |
anonymous | Unsigned access; mutually exclusive with explicit credentials and role_arn |
role_arn, session_name, external_id | Assume-role configuration using source credentials from the default chain; mutually exclusive with explicit keys |
load_frequency | Requested STS assume-role session duration in seconds, 900 through 43,200; not a refresh interval |
force_virtual_addressing | Force bucket-in-host addressing; otherwise custom endpoints use path-style addressing |
connect_timeout, request_timeout | Positive finite seconds, at most 2,147,483.647; request timeout is platform-dependent and ignored on Linux by this backend, not a total-operation deadline |
max_attempts | AWS standard retry attempts, including the initial request; 1 through 100 |
tls_ca_file_path | Optional CA bundle path; certificate verification cannot be disabled |
load_frequency retains the underlying Arrow option name but is sent to STS as
DurationSeconds. The requested duration must also fit the role's configured
maximum; role chaining is limited to 3,600 seconds. The AWS provider refreshes
credentials before their returned expiration rather than at a fixed interval.
With no authentication mode specified, Arrow uses the AWS default credential provider chain, including environment credentials, shared profiles, and supported workload identity providers. Set AWS configuration before creating the first client. Region discovery, assume-role, credential refresh, and workload identity can make additional network requests; validate them in the target AWS environment. No credential configuration is read when only local paths are used.
s3_available() reports build capability without initializing AWS. A build with
CODON_DATA_S3=OFF rejects S3 operations. A filesystem is context-managed;
closing it is idempotent, closes aliases, and prevents new operations through
that handle. Already-open Avro streams retain their client until they close.
Do not close handles concurrently with use. Do not reuse S3 clients or streams
after fork; use fork-plus-exec, or initialize S3 only after forking.
The SDK supports individual objects, not directories, datasets, listing, presigned URLs, explicit versions, append, or bucket management. Objects must remain unchanged during reads. See S3 objects for commit, abort, overwrite, and ambiguous-completion behavior. Stream-based uploads may also retain multipart buffering beyond Avro's block budget. For AWS policies, reads require object read permission and writes require object write and multipart-abort permission, plus any applicable KMS permissions.
CData
| Member | Contract |
|---|---|
CData() | Allocates empty Arrow C ABI array/schema structs |
data.array_ptr | Ptr[ArrowArray] |
data.schema_ptr | Ptr[ArrowSchema] |
data.released | True when both release callbacks are null |
data.close() | Releases any live callbacks, safely repeatable |
with data as entered: | Releases remaining callbacks on exit |
An export is independently owned. Successful Array.from_c consumes it and
clears release callbacks; importing it twice fails. Raw ABI pointers are a trusted
interop interface, not validation of arbitrary memory. The Array(handle) and
Table(handle) constructors are for internal runtime handles, not user pointers.
Errors
| Condition | Result |
|---|---|
| Unsupported static native type | Compile-time error |
| Incompatible schema, non-optional null, invalid cast, invalid/released C Data | ValueError |
Invalid file/format, malformed input, closed handle, nonpositive batch_size | ValueError from the native boundary |
Out-of-range item, is_null, or column index | IndexError |
| Missing column name | KeyError |
| Duplicate matching column names | ValueError |
| Invalid array/table slice | ValueError from native range validation |
| Invalid S3 URI/options rejected by the Codon wrapper | ValueError or OSError |
| Native filesystem setup or Avro stream I/O failure | OSError |
Catch the exception type rather than matching its message; diagnostic text may
change between releases. Local file writes can leave a partial destination if a format
or I/O error occurs. When replacing an existing file must be atomic, write to a
temporary path and rename it after a successful write.
S3 failures follow the object-write contract, not local
rename semantics. Arrow table I/O retains its existing ValueError boundary,
including remote service errors.
The Data guide demonstrates these operations in complete applications.