Introducing nanoarrow

(the world’s tiniest Arrow implementation)

Dewey Dunnington

Voltron Data, Inc.



Snowflake Python Connector


GeoArrow


cudf

Snowflake Python Connector

  • Wire protocol is Arrow IPC-based, but is more like an Arrow representation of the Snowflake internal representation
  • Needs compute functions to export Pandas, Arrow, Python objects (dbapi)
  • Used Arrow C++, which resulted in a pinned pyarrow version dependency
import snowflake.connector

con = snowflake.connector.connect(...)
cur = con.cursor()
cur.execute("SELECT * FROM testtable")
cur.fetchall().to_arrow()

GeoArrow

  • Geospatial data can be represented in Arrow
  • Libraries that implement geospatial manipulation are poor candidates to build in to Arrow implementations
  • Compute functions for geospatial data need to be shipped in separate Python packages
import geoarrow.pyarrow as ga

ga.as_wkb(["POINT (0 1)"]).storage
<pyarrow.lib.BinaryArray object at 0x15bd2bfa0>
[
  01010000000000000000000000000000000000F03F
]

cudf

  • Dataframe library implementing interoperability with pyarrow (CPU)
  • Would like to implement GPU interoperability for all supported types via the ArrowDeviceArray
  • Memory is already Arrow memory!

Sharing data among packages

Sharing data among packages

PyBuffer DLPack DataFrame Arrow
Integers ✅ ✅ ✅ ✅
Floats ✅ ✅ ✅ ✅
Strings ❌ ❌ ✅ ✅
Nested ❌ ❌ ❌ ✅
GPU ❌ ✅ ❌ ✅
ndarray ✅ ✅ ❌ *
Serializable ❌ ❌ ❌ ✅

Sharing data among runtimes

PyBuffer DLPack DataFrame Arrow
Go ✅
Java ✅
C# ✅
Julia ✅
JavaScript ✅
Rust ✅

Sharing data among ecosystems

PyBuffer DLPack DataFrame Arrow
GDAL ✅
DuckDB ✅

Apache Arrow

Create arrays from Python objects

import nanoarrow as na

na.Array(["banana", "orange", "apple"], na.string())
nanoarrow.Array<string>[3]
'banana'
'orange'
'apple'

Read arrays as Python objects

array = na.Array(["banana", "orange", "apple"], na.string())
array.to_pylist()
['banana', 'orange', 'apple']

Create arrays from buffers

c_array = na.c_array_from_buffers(
  na.string(),
  3,
  [
    None,
    na.c_buffer([0, 6, 12, 17], na.int32()),
    b"bananaorangeapple"
  ]
)
na.Array(c_array)
nanoarrow.Array<string>[3]
'banana'
'orange'
'apple'

Read arrays as buffers

array = na.Array(["banana", "orange", "apple"], na.string())
for buffer in array.buffers:
  print(buffer)
nanoarrow.c_lib.CBufferView(bool[0 b] )
nanoarrow.c_lib.CBufferView(int32[16 b] 0 6 12 17)
nanoarrow.c_lib.CBufferView(string[17 b] b'bananaorangeapple')

Create arrays from serialized bytes

url = "https://github.com/apache/arrow-experiments/raw/main/data/arrow-commits/arrow-commits.arrows"
na.ArrayStream.from_url(url).read_all()
nanoarrow.Array<non-nullable struct<commit: string, time: timestamp('...>[15487]
{'commit': '49cdb0fe4e98fda19031c864a18e6156c6edbf3c', 'time': datetime.datet...
{'commit': '1d966e98e41ce817d1f8c5159c0b9caa4de75816', 'time': datetime.datet...
{'commit': '96f26a89bd73997f7532643cdb27d04b70971530', 'time': datetime.datet...
{'commit': 'ee1a8c39a55f3543a82fed900dadca791f6e9f88', 'time': datetime.datet...
{'commit': '3d467ac7bfae03cf2db09807054c5672e1959aec', 'time': datetime.datet...
{'commit': 'ef6ea6beed071ed070daf03508f4c14b4072d6f2', 'time': datetime.datet...
{'commit': '53e0c745ad491af98a5bf18b67541b12d7790beb', 'time': datetime.datet...
{'commit': '3ba6d286caad328b8572a3b9228045da8c8d2043', 'time': datetime.datet...
{'commit': '4ce9a5edd2710fb8bf0c642fd0e3863b01c2ea20', 'time': datetime.datet...
{'commit': '2445975162905bd8d9a42ffc9cd0daa0e19d3251', 'time': datetime.datet...
...and 15477 more items

(Writing arrays as serialized bytes is in progress)

Create arrays from C objects

#include "nanoarrow.hpp"

int make_array(ArrowSchema* out_schema, ArrowArray* out_array, ArrowError* error) {
  NANOARROW_RETURN_NOT_OK(ArrowSchemaInitFromType(out_schema, NANOARROW_TYPE_STRING));

  NANOARROW_RETURN_NOT_OK(ArrowArrayInitFromSchema(out_array, out_schema, error));
  NANOARROW_RETURN_NOT_OK(ArrowArrayStartAppending(out_array));
  NANOARROW_RETURN_NOT_OK(ArrowArrayAppendString(out_array, "banana"_asv));
  NANOARROW_RETURN_NOT_OK(ArrowArrayAppendString(out_array, "orange"_asv));
  NANOARROW_RETURN_NOT_OK(ArrowArrayAppendString(out_array, "apple"_asv));
  NANOARROW_RETURN_NOT_OK(ArrowArrayFinishBuildingDefault(out_array, error));

  return NANOARROW_OK;
}

Read arrays as C objects

#include <iostream>
#include "nanoarrow.hpp"

int read_array(const ArrowSchema* out_schema, const ArrowArray* out_array, ArrowError* error) {
  nanoarrow::UniqueArrayView view;
  NANOARROW_RETURN_NOT_OK(ArrowArrayViewInitFromSchema(view.get(), out_schema, error));

  ArrowStringView item;
  item = ArrowArrayViewGetStringUnsafe(view.get(), 0);
  std::cout << std::string(item.data, item.size_bytes);

  return NANOARROW_OK;
}

Create arrays from buffers

#include <string>
#include <vector>
#include "nanoarrow.hpp"

int make_array(ArrowSchema* out_schema, ArrowArray* out_array, ArrowError* error) {
  NANOARROW_RETURN_NOT_OK(ArrowSchemaInitFromType(out_schema, NANOARROW_TYPE_STRING));

  NANOARROW_RETURN_NOT_OK(ArrowArrayInitFromSchema(out_array, out_schema, error));
  nanoarrow::BufferInitSequence<std::vector<int32_t>>(
    ArrowArrayBuffer(out_array, 1), {0, 6, 12, 17});
  nanoarrow::BufferInitSequence<std::string>(
    ArrowArrayBuffer(out_array, 2), "bananaorangeapple");
  out_array->length = 3;
  out_array->null_count = 0;
  NANOARROW_RETURN_NOT_OK(ArrowArrayFinishBuildingDefault(out_array, error));

  return NANOARROW_OK;
}

Read arrays as C buffers

#include <iostream>
#include "nanoarrow.hpp"

int read_array(const ArrowSchema* out_schema, const ArrowArray* out_array, ArrowError* error) {
  nanoarrow::UniqueArrayView view;
  NANOARROW_RETURN_NOT_OK(ArrowArrayViewInitFromSchema(view.get(), out_schema, error));

  // view->buffer_views[1].data.as_int32, view->buffer_views[1].size_bytes
  // view->buffer_views[2].data.as_char, view->buffer_views[2].size_bytes

  return NANOARROW_OK;
}



Snowflake Python Connector


GeoArrow


cudf

Snowflake Python Connector

import snowflake.connector

con = snowflake.connector.connect(...)
cur = con.cursor()
cur.execute("SELECT * FROM testtable")
cur.fetchall().to_arrow()

Example consuming by item

GeoArrow

import geoarrow.pyarrow as ga

ga.as_wkb(["POINT (0 1)"]).storage
<pyarrow.lib.BinaryArray object at 0x15c0f7e20>
[
  01010000000000000000000000000000000000F03F
]

Example consuming by item

Example building buffers

Example building arrays from buffers

cudf

  • Dataframe library implementing interoperability with pyarrow (CPU)
  • Would like to implement GPU interoperability for all supported types via the ArrowDeviceArray
  • Memory is already Arrow memory!

Example type computations

Example building by buffer

Arrow C++/PyArrow

  • Produce Arrow Arrays from C/Py types
  • Consume Arrow Arrays and convert them to C/Py types
  • Transform Arrow Arrays

Distributing a Python package (other than pyarrow) with an Arrow C++ dependency is difficult

nanoarrow

  • Produce Arrow Arrays from C/Py types
  • Consume Arrow Arrays and convert them to C/Py types
  • Transform Arrow Arrays

Copy nanoarrow.c and nanoarrow.h into any existing setup

nanoarrow