Skip to content

Dremio SQL Lakehouse Arrow Flight Client Installation Guide

Arrow Flight is a high-speed, distributed protocol designed to handle big data, providing increase in throughput between client applications and Dremio.

This Dremio Arrow Flight Client is based on python Official examples.

Disclaimer: This project is not affliated to dremio in any way. It is a tool that I developed while at CIFOR-ICRAF and now we have decided to open source it for wider community use. While I may not have enough time to actively maintain it, the tool is stable enough to sustain future use cases. Besides, community contribution is warmly welcome in form of PRs and forks.

dremio-arrow package is available on PyPI and can be installed with pip, poetry, conda or from GitHub

python

Installation Requirements

Experimental Git Branches

The default main branch contains stable release. Please install the latest release on pypi or use code from the main branch when in doubt. For experimentations, feel free to use any branch accessible to you 😄

Installing the package

Supported Python Versions

This package is thorougly tested against python3.7, python3.8, python3.9 or python3.10. Other python3 versions might work but just note they have not been tested.

Virtual Environment or Install on OS Filesystem?

Personally I discourage installation on the python base libraries path. I consider it an evil act because it may cause unprecedented issues if the package is compromised (we are always onlook for security vulnerabilities but it is always good to be prepared for unexpected eventualities). In my opinion therefore, it is better to install the package (not just this but also all other packages that you use) in a virtual environment. Using a virtual environment also allows one to experiment with different python and/or package versions. If you are curious about this, please look at pyenv.

Installing from PyPI (recomended)

Create a virtual environment and activate it.

1
2
virtualenv -p python3 venv
source venv/bin/activate
  • Using python3's venv
1
2
python3 -m venv venv
source venv/bin/activate

Install the package using pip

1
pip install dremio-arrow

Install the package using poetry

1
poetry add dremio-arrow

Install the package using conda

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
- Create a conda environment
```bash
conda create -n myenv python=3.10
source activate myenv
```

- Install the package
```bash
conda install dremio-arrow
```

Installing from github source

Clone the repository

1
git clone https://github.com/jaysnm/dremio-arrow.git

Change to the source directory 

1
cd dremio-arrow

Install the package using pip 

1
pip install -e .
Install the package using poetry 
1
poetry install

Not using Linux or MacOS?

This example assumes you are on MacOSX or Linux. If using Windows or any other OS, kindly look for their respective documentations on how to use a virtual environment in the command prompt.

Development Dependencies

Besides pyarrow and pandas, the package ships with optional dependencies used during development. Installation of these dependencies is only required if you intend to contribute some changes into the package. Please see Contributing Guidlines.

Last update: July 4, 2023
Created: July 4, 2023