-   [](https://platform.exaloop.io/)
-   Data APIs
-   Data API Reference

# Data API Reference

The `exaloop.data` API provides arrays, tables, file readers, and native buffer exchange. Signatures below assume `import exaloop.data as data`. For complete programs and the conversion model, see the [Data guide](https://platform.exaloop.io/data/data/).

A type parameter such as `T` is a Codon type, for example `int`, `float`, or `Optional[str]`. Codon resolves it at compile time; the API checks the incoming column's schema at runtime.

## Construction

```python title="Source"
data.array(values: list[T], copy: bool = True) -> Array
data.array(values: np.ndarray[T, 1], copy: bool = True) -> Array
data.table(columns: dict[str, Array], num_rows: int = -1) -> Table
```

`array()` accepts a typed list or a one-dimensional NumPy array. NumPy input is suited to fixed-width numeric and boolean data. Typed lists also represent text, bytes, nested lists, and optional elements, which are converted to Arrow's corresponding buffer layouts.

`table()` assembles named columns of equal length. The default `num_rows=-1` infers that length (zero for an empty table); an explicit row count must agree with every column. Names are UTF-8 without embedded NULs. A dictionary naturally has unique names, although tables read from files may contain duplicates.

| Native type | Arrow representation produced |
| --- | --- |
| `NoneType` | Null |
| `bool` | Packed boolean |
| `i8/u8`, `i16/u16`, `i32/u32` | Corresponding signed/unsigned integer width |
| `int`, `i64`, `u64` | 64-bit integer |
| `float16`, `float32`, `float` | 16/32/64-bit floating point |
| `str`, `bytes` | Large UTF-8 / large binary |
| `list[T]` | Large list with recursively constructed child |
| `Optional[T]` | Nullable underlying column |

Construction validates the resulting Arrow data, even when the input is empty. For shared input storage, see [ownership and copies](https://platform.exaloop.io/data/data/#ownership-and-copies).

## Storage and Copies

| Operation | Storage behavior |
| --- | --- |
| `data.array(values)` | Snapshot by default |
| `data.array(values, copy=False)` | Borrows fixed-width numeric list or contiguous ndarray storage |
| Noncontiguous ndarray input | Creates contiguous storage first |
| Boolean, text, nullable, or nested input | Converts layout even with `copy=False` |
| `item`, `to_list`, `to_numpy`, `to_numpy_masked` | Native outputs are independent of Arrow storage |
| Slices, extracted columns, C exports | Retain storage independently of the parent |
| Multi-chunk column extraction | Combines chunks, potentially copying |
| `to_bytes` | Serialized output without a final copy |
| `from_bytes` | Retains input; decoding or decompression can allocate |

Borrowed inputs must remain unchanged while Arrow uses them. Sharing an array does not make concurrent mutation or closing safe.

## Array

An `Array` represents one logical column. Conversion methods return owned Codon values; slicing and C Data export retain the Arrow storage instead.

| Member | Return / contract |
| --- | --- |
| `len(column)` | Number of logical elements |
| `column.format` | Arrow C Data format string; dictionary columns report their index format |
| `column.null_count` | Physical null count, not decoded dictionary-value nulls |
| `column.is_null(index: int)` | Physical validity; supports negative indices |
| `column.item(index: int, T)` | One native value; negative indices supported |
| `column.to_list(T)` | Independent, resizable `list[T]` |
| `column.to_numpy(T)` | Independent 1D numeric ndarray; rejects nulls |
| `column.to_numpy_masked(T)` | `(numeric_values, boolean_missing_mask)`; missing data slots are zero |
| `column.cast(T)` | New array using Arrow's safe cast; target is a supported primitive type, not a nested list/optional specification |
| `column.slice(offset: int, length: int = -1)` | Shared-storage view; `-1` means remaining length |
| `column.export_c()` | New `CData` owner with populated ABI structs |
| `Array.from_c(data: CData)` | Consumes a live C export into a new array |
| `column.close()` | Idempotently closes this handle and its aliases |
| `with column as entered:` | Checks the handle and closes it on exit |

`item`/`to_list` accept exact primitive, text/binary, `NoneType`, and nested list/optional targets. NumPy methods accept only primitive numeric/bool types. String/binary conversion accepts both 32-bit and 64-bit offset encodings. List conversion accepts normal, large, and fixed-size lists. Dictionary indices of every supported integer width are decoded before value conversion.

`slice(offset, length)` selects a contiguous region, with a nonnegative offset and a length within the array's bounds. Leaving `length=-1` selects the rest of the array. This method does not support strides or Python's negative slice indices.

## Table

Tables provide named or positional access to columns. A name lookup is convenient for ordinary schemas; use a column index when a file contains duplicate names.

| Member | Return / contract |
| --- | --- |
| `len(dataset)`, `dataset.num_rows` | Row count |
| `dataset.num_columns` | Column count |
| `dataset.column_names` | Independent `list[str]` in schema order, including duplicates |
| `dataset.column(index: int)` | `Array`; supports negative indices |
| `dataset.column(name: str)` | `Array`; name must identify exactly one column |
| `dataset[key]` | Same positional/name lookup as `column` |
| `dataset.slice(offset: int, length: int = -1)` | Shared-storage row slice with bounds checks |
| `dataset.write(path: str, format: str = "", batch_size: int = 65536)` | Writes a local file or S3 object; infers format from extension when empty |
| `dataset.write_avro(path, writer_schema, codec="null", block_size=65536, batch_size=65536, max_block_bytes=67108864)` | Writes Avro with an explicit `Schema`; see [table writers](https://platform.exaloop.io/data/avro-api/#table-writers) |
| `dataset.to_bytes(format: str = "ipc", batch_size: int = 65536)` | Serialized `bytes` |
| `Table.from_bytes(data: bytes, format: str = "ipc")` | New table retaining the input bytes as needed |
| `dataset.close()` | Idempotently closes this handle and its aliases |
| `with dataset as entered:` | Checks the handle and closes it on exit |

Column extraction combines all of a column's chunks, so it can allocate rather than simply return a view. For dictionary columns, combining chunks may widen the index type to 64 bits when the dictionaries cannot otherwise be unified. The logical values are preserved, including the distinction between a null index and an index pointing to a null dictionary entry.

## File Readers

```python title="Source"
data.read_table(path: str, format: str = "") -> Table
data.read_ipc(path: str) -> Table
data.read_feather(path: str) -> Table
data.read_parquet(path: str) -> Table
data.read_orc(path: str) -> Table
data.read_avro(path: str) -> Table
data.read_csv(path: str) -> Table
data.read_json(path: str) -> Table
```

Convenience readers specify the format, regardless of filename extension. `read_table` infers it when `format` is empty. Use `read_table(path, "stream")` for IPC streams. Format names are case-insensitive. Supported names and format-specific restrictions are listed in the [guide](https://platform.exaloop.io/data/data/#formats). For Avro projection, bounded batches, schema resolution, and generated records, see the [Avro API](https://platform.exaloop.io/data/avro/#api).

Readers and writers also accept an optional `filesystem` (`Optional[S3FileSystem]`, default `None`). Omit it for ordinary calls: plain paths are local and `s3://` paths use the lazily cached default AWS client. With a filesystem, both `s3://bucket/key` and raw `bucket/key` paths refer to objects on that service. Only `s3://` URIs are supported; credentials, ports, queries, and fragments in URIs are rejected. Percent-decode keys only through URI syntax, not raw paths. S3 key segments equal to `.` or `..` are rejected, including percent-encoded forms in URIs, to prevent HTTP path normalization from changing the target object. Other dots in names are allowed; local filesystem paths are unaffected.

## S3FileSystem

```text title="Text"
S3FileSystem(region="", endpoint_override="", scheme="https",
			 access_key="", secret_key="", session_token="",
			 anonymous=False, role_arn="", session_name="", external_id="",
			 load_frequency=900, force_virtual_addressing=False,
			 connect_timeout=3.0, request_timeout=30.0, max_attempts=3,
			 tls_ca_file_path="")
s3_available() -> bool
filesystem.close()
```

| Option | Contract |
| --- | --- |
| `region` | Empty uses AWS SDK region resolution; supply the bucket's region when known |
| `endpoint_override` | Optional S3-compatible host and port; no scheme, credentials, or path |
| `scheme` | `https` by default; `http` for trusted test services |
| `access_key`, `secret_key`, `session_token` | Explicit credentials; both keys are required, token optional; these values do not refresh automatically |
| `anonymous` | Unsigned access; mutually exclusive with explicit credentials and `role_arn` |
| `role_arn`, `session_name`, `external_id` | Assume-role configuration using source credentials from the default chain; mutually exclusive with explicit keys |
| `load_frequency` | Requested STS assume-role session duration in seconds, 900 through 43,200; not a refresh interval |
| `force_virtual_addressing` | Force bucket-in-host addressing; otherwise custom endpoints use path-style addressing |
| `connect_timeout`, `request_timeout` | Positive finite seconds, at most 2,147,483.647; request timeout is platform-dependent and ignored on Linux by this backend, not a total-operation deadline |
| `max_attempts` | AWS standard retry attempts, including the initial request; 1 through 100 |
| `tls_ca_file_path` | Optional CA bundle path; certificate verification cannot be disabled |

`load_frequency` retains the underlying Arrow option name but is sent to STS as `DurationSeconds`. The requested duration must also fit the role's configured maximum; role chaining is limited to 3,600 seconds. The AWS provider refreshes credentials before their returned expiration rather than at a fixed interval.

With no authentication mode specified, Arrow uses the AWS default credential provider chain, including environment credentials, shared profiles, and supported workload identity providers. Set AWS configuration before creating the first client. Region discovery, assume-role, credential refresh, and workload identity can make additional network requests; validate them in the target AWS environment. No credential configuration is read when only local paths are used.

`s3_available()` reports build capability without initializing AWS. A build with `CODON_DATA_S3=OFF` rejects S3 operations. A filesystem is context-managed; closing it is idempotent, closes aliases, and prevents new operations through that handle. Already-open Avro streams retain their client until they close. Do not close handles concurrently with use. Do not reuse S3 clients or streams after `fork`; use fork-plus-exec, or initialize S3 only after forking.

The SDK supports individual objects, not directories, datasets, listing, presigned URLs, explicit versions, append, or bucket management. Objects must remain unchanged during reads. See [S3 objects](https://platform.exaloop.io/data/data/#s3-objects) for commit, abort, overwrite, and ambiguous-completion behavior. Stream-based uploads may also retain multipart buffering beyond Avro's block budget. For AWS policies, reads require object read permission and writes require object write and multipart-abort permission, plus any applicable KMS permissions.

## CData

| Member | Contract |
| --- | --- |
| `CData()` | Allocates empty Arrow C ABI array/schema structs |
| `data.array_ptr` | `Ptr[ArrowArray]` |
| `data.schema_ptr` | `Ptr[ArrowSchema]` |
| `data.released` | True when both release callbacks are null |
| `data.close()` | Releases any live callbacks, safely repeatable |
| `with data as entered:` | Releases remaining callbacks on exit |

An export is independently owned. Successful `Array.from_c` consumes it and clears release callbacks; importing it twice fails. Raw ABI pointers are a trusted interop interface, not validation of arbitrary memory. The `Array(handle)` and `Table(handle)` constructors are for internal runtime handles, not user pointers.

## Errors

| Condition | Result |
| --- | --- |
| Unsupported static native type | Compile-time error |
| Incompatible schema, non-optional null, invalid cast, invalid/released C Data | `ValueError` |
| Invalid file/format, malformed input, closed handle, nonpositive `batch_size` | `ValueError` from the native boundary |
| Out-of-range `item`, `is_null`, or column index | `IndexError` |
| Missing column name | `KeyError` |
| Duplicate matching column names | `ValueError` |
| Invalid array/table slice | `ValueError` from native range validation |
| Invalid S3 URI/options rejected by the Codon wrapper | `ValueError` or `OSError` |
| Native filesystem setup or Avro stream I/O failure | `OSError` |

Catch the exception type rather than matching its message; diagnostic text may change between releases. Local file writes can leave a partial destination if a format or I/O error occurs. When replacing an existing file must be atomic, write to a temporary path and rename it after a successful write. S3 failures follow the [object-write contract](https://platform.exaloop.io/data/data/#s3-objects), not local rename semantics. Arrow table I/O retains its existing `ValueError` boundary, including remote service errors.

The [Data guide](https://platform.exaloop.io/data/data/) demonstrates these operations in complete applications.

Source: https://platform.exaloop.io/data/data-api/