Skip to content

Commit b01f944

Browse files
ramoncelestinoramoncelestino
authored
Stainless sdk changes (#15)
* Including release notes * Testing new sdk * Update tests and notebook * Fix readmes
1 parent 484a492 commit b01f944

File tree

105 files changed

+11707
-904
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

105 files changed

+11707
-904
lines changed

‎.DS_Store‎

6 KB
Binary file not shown.

‎.github/workflows/publish.yml‎

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -8,26 +8,26 @@ jobs:
88
runs-on: ubuntu-latest
99

1010
steps:
11-
- name: Checkout code
12-
uses: actions/checkout@v2
11+
- name: Checkout code
12+
uses: actions/checkout@v2
1313

14-
- name: Set up Python
15-
uses: actions/setup-python@v2
16-
with:
17-
python-version: '3.x'
14+
- name: Set up Python
15+
uses: actions/setup-python@v2
16+
with:
17+
python-version: "3.x"
1818

19-
- name: Install dependencies
20-
run: |
21-
python -m pip install --upgrade pip
22-
pip install setuptools wheel twine
19+
- name: Install dependencies
20+
run: |
21+
python -m pip install --upgrade pip
22+
pip install setuptools wheel twine
2323
24-
- name: Build package
25-
run: |
26-
python setup.py sdist bdist_wheel
24+
- name: Build package
25+
run: |
26+
python setup.py sdist bdist_wheel
2727
28-
- name: Publish package
29-
env:
30-
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
31-
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
32-
run: |
33-
twine upload dist/*
28+
- name: Publish package
29+
env:
30+
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
31+
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
32+
run: |
33+
twine upload dist/*

‎README.md‎

Lines changed: 29 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -13,17 +13,16 @@ AIMon Rely helps developers build, ship, and monitor LLM Apps more confidently a
1313
The following is a list of quality metrics that are currently available and on our roadmap. Please reach out to express your interest in any of these.
1414

1515
| Metric | Status |
16-
|--------------------------------------------------|--------------------------------------------------------------|
17-
| Model Hallucination (Passage and Sentence Level) | <span style="font-size: 24px; color: green;">&#10003;</span> |
18-
| Completeness | <span style="font-size: 24px; color: green;">&#10003;</span> |
19-
| Conciseness | <span style="font-size: 24px; color: green;">&#10003;</span> |
20-
| Toxicity | <span style="font-size: 24px; color: green;">&#10003;</span> |
21-
| Instruction Adherence | <span style="font-size: 24px; color: green;">&#10003;</span> |
22-
| Semantic Similarity | <span style="font-size: 24px;">⌛</span> |
23-
| Sentiment | <span style="font-size: 24px;">⌛</span> |
24-
| Coherence | <span style="font-size: 24px;">⌛</span> |
25-
| Sensitive Data (PII/PHI/PCI) | <span style="font-size: 24px;">⌛</span> |
26-
16+
| ------------------------------------------------ | ------------------------------------------------------------ |
17+
| Model Hallucination (Passage and Sentence Level) | <span style="font-size: 24px; color: green;">&#10003;</span> |
18+
| Completeness | <span style="font-size: 24px; color: green;">&#10003;</span> |
19+
| Conciseness | <span style="font-size: 24px; color: green;">&#10003;</span> |
20+
| Toxicity | <span style="font-size: 24px; color: green;">&#10003;</span> |
21+
| Instruction Adherence | <span style="font-size: 24px; color: green;">&#10003;</span> |
22+
| Semantic Similarity | <span style="font-size: 24px;">⌛</span> |
23+
| Sentiment | <span style="font-size: 24px;">⌛</span> |
24+
| Coherence | <span style="font-size: 24px;">⌛</span> |
25+
| Sensitive Data (PII/PHI/PCI) | <span style="font-size: 24px;">⌛</span> |
2726

2827
## Getting Started
2928

@@ -35,10 +34,25 @@ AIMon supports asynchronous instrumentation or synchronous detections for the me
3534
to get started with using the AIMon SDK and the product.
3635

3736
- Step 1: Get access to the beta product by joining the waitlist on our [website](https://aimon.ai/) or by requesting
38-
it on [Slack](https://join.slack.com/t/generativeair/shared_invite/zt-2jab62lsj-xM9a_s~Qweu8lf3YS2cANg) or sending an email to info@aimon.ai
37+
it on [Slack](https://join.slack.com/t/generativeair/shared_invite/zt-2jab62lsj-xM9a_s~Qweu8lf3YS2cANg) or sending an email to info@aimon.ai
3938
- Step 2: Install the AIMon SDK by running `pip install aimon` in your terminal.
40-
- Step 3: For an example of how to instrument an LLM application **asynchronously** using the SDK, please refer to the [sample notebook](examples/notebooks/aimon_sdk_langchain_summarization.ipynb)
41-
- Step 4: For an example of synchronous detections using the SDK, please refer to the sample [streamlit application](examples/langchain_summarization_app.py)
39+
- Step 3: For an axample of quick start usage
40+
41+
```python
42+
from aimon import Client
43+
44+
API_KEY = "YOUR_API_KEY"
45+
client = Client(auth_header=f"Bearer {API_KEY}")
46+
data_to_send = [{
47+
"context": "This is the context",
48+
"generated_text": "This is the context",
49+
"config": {'hallucination': {'detector_name': 'default'}}
50+
}]
51+
response = client.inference.detect(body=data_to_send)[0]
52+
```
53+
54+
- Step 4: For an example of how to instrument an LLM application **asynchronously** using the SDK, please refer to the [sample notebook](examples/notebooks/aimon_sdk_langchain_summarization.ipynb)
55+
- Step 5: For an example of synchronous detections using the SDK, please refer to the sample [streamlit application](examples/langchain_summarization_app.py)
4256

4357
<div align="center">
4458
<img src="images/product_apps_page.png" alt="AIMon Product">
@@ -59,6 +73,7 @@ You can try our [Sandbox](https://aimon.ai/tryproduct) that is available on our
5973
## Benchmarks
6074

6175
### Hallucination Detection
76+
6277
To demonstrate the effectiveness of our system, we benchmarked it against popular industry benchmarks for the
6378
hallucination detection task. The table below shows our results.
6479

‎README_PyPi.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# 🎉**Welcome to AIMon Rely**
2+
3+
AIMon Rely helps developers build, ship, and monitor LLM Apps more confidently and reliably with its state-of-the-art, multi-model system for detecting LLM quality issues. It helps seamlessly with both offline evaluations and continuous production monitoring. AIMon Rely offers fast, reliable, and cost-effective hallucination detection. It also supports other important quality metrics such as completeness, conciseness, and toxicity. Read our [blog post](https://aimon.ai/blogs/introducing-rely) for more details.

‎aimon/__init__.py‎

Lines changed: 81 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,81 @@
1-
from .client import Client
2-
from .client import ApplicationStage
3-
from .dataset import Dataset, DatasetCollection, DatasetRecord
4-
from .evaluation import Evaluation, Run
5-
from .models import MLModel
6-
from .simple_client import SimpleAimonRelyClient
7-
from .metrics_config import Config
8-
from .utils import InvalidAPIKeyError
1+
# File generated from our OpenAPI spec by Stainless. See CONTRIBUTING.md for details.
2+
3+
from . import types
4+
from ._types import NOT_GIVEN, NoneType, NotGiven, Transport, ProxiesTypes
5+
from ._utils import file_from_path
6+
from ._client import Client, Stream, Timeout, Transport, AsyncClient, AsyncStream, RequestOptions
7+
from ._models import BaseModel
8+
from ._version import __title__, __version__
9+
from ._response import APIResponse as APIResponse, AsyncAPIResponse as AsyncAPIResponse
10+
from ._constants import DEFAULT_TIMEOUT, DEFAULT_MAX_RETRIES, DEFAULT_CONNECTION_LIMITS
11+
from ._exceptions import (
12+
APIError,
13+
ClientError,
14+
ConflictError,
15+
NotFoundError,
16+
APIStatusError,
17+
RateLimitError,
18+
APITimeoutError,
19+
BadRequestError,
20+
APIConnectionError,
21+
AuthenticationError,
22+
InternalServerError,
23+
PermissionDeniedError,
24+
UnprocessableEntityError,
25+
APIResponseValidationError,
26+
)
27+
from ._base_client import DefaultHttpxClient, DefaultAsyncHttpxClient
28+
from ._utils._logs import setup_logging as _setup_logging
29+
30+
__all__ = [
31+
"types",
32+
"__version__",
33+
"__title__",
34+
"NoneType",
35+
"Transport",
36+
"ProxiesTypes",
37+
"NotGiven",
38+
"NOT_GIVEN",
39+
"ClientError",
40+
"APIError",
41+
"APIStatusError",
42+
"APITimeoutError",
43+
"APIConnectionError",
44+
"APIResponseValidationError",
45+
"BadRequestError",
46+
"AuthenticationError",
47+
"PermissionDeniedError",
48+
"NotFoundError",
49+
"ConflictError",
50+
"UnprocessableEntityError",
51+
"RateLimitError",
52+
"InternalServerError",
53+
"Timeout",
54+
"RequestOptions",
55+
"Client",
56+
"AsyncClient",
57+
"Stream",
58+
"AsyncStream",
59+
"file_from_path",
60+
"BaseModel",
61+
"DEFAULT_TIMEOUT",
62+
"DEFAULT_MAX_RETRIES",
63+
"DEFAULT_CONNECTION_LIMITS",
64+
"DefaultHttpxClient",
65+
"DefaultAsyncHttpxClient",
66+
]
67+
68+
_setup_logging()
69+
70+
# Update the __module__ attribute for exported symbols so that
71+
# error messages point to this module instead of the module
72+
# it was originally defined in, e.g.
73+
# aimon._exceptions.NotFoundError -> aimon.NotFoundError
74+
__locals = locals()
75+
for __name in __all__:
76+
if not __name.startswith("__"):
77+
try:
78+
__locals[__name].__module__ = "aimon"
79+
except (TypeError, AttributeError):
80+
# Some of our exported symbols are builtins which we can't set attributes for.
81+
pass

0 commit comments

Comments
 (0)