Benchmark Jina

Benchmark Jina #

We are currenty considering time metrics to benchmark Jina features and using pytest to run these tests.

Playbook #

Prepare environment #

pip install -r requirements.txt
pip install pre-commit==2.13.0
pre-commit install
git submodule update --init

Run Locally #

pytest

Run on Docker #

JINA_VER=master
docker build --build-arg JINA_VER=$JINA_VER -t bechmark .
docker run -v $(pwd):/app bechmark:latest

Generate docs locally and run server #

python scripts/site_generator.py
cd docs
hugo server -D

Machine #

We are running all tests sequentially for a version on a single machine of following properties:

ItemValue
Cloud VendorAWS
Instancec5.xlarge
Memory8 GiB
vCPU4
ProcessorIntel Xeon Platinum 8124M
Clock Speed3 GHz
StorageEBS (gp2)

Contributing #

We welcome all kinds of contributions from the open-source community, individuals and partners. We owe our success to your active involvement.

Here’re some quick notes you need to know before starting to contribute:

  • Please keep all of your tests under src folder and ensure they behave as expected with pytest.
  • Please save the benchmarking artifacts in JSON format in docs/static/artifacts/${JINA_VERSION}/report.json file.
  • Please enlist any Python dependency to requirements.txt file.
  • Please run scripts/site_generator.py to generate the website everytime you generate new benchmarking artifacts.
  • report.json file should have the following shema:
[
  {
    "name": "document_array_append/test_docarray_append",
    "iterations": 5,
    "mean_time": 0.007944801799999368,
    "std_time": 0.0012715548259231583,
    "metadata": {
      "num_docs_append": 10000
    }
  }
]