Skip to content

Commit 80793ec

Browse files
harsh97harsh97
committed
Adding feature store docs
1 parent e032cd1 commit 80793ec

File tree

5 files changed

+278
-49
lines changed

5 files changed

+278
-49
lines changed

‎ads/feature_store/docs/source/index.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,9 @@ Feature Store
2727
.. toctree::
2828
:maxdepth: 2
2929
:caption: Getting started:
30-
30+
3131
overview
32-
user_guides.setup.feature_store_operator
32+
setup_feature_store
3333
quickstart
3434

3535
.. toctree::

‎ads/feature_store/docs/source/setup_feature_store.rst‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
====================
2+
Setup Feature Store
3+
====================
4+
5+
Feature store is being provided by OCI as a stack based offering in user's own tenancy via OCI marketplace. It can be configured by the user primarily in two ways:
6+
7+
:doc:`Setup via Feature Store Marketplace Operator (Recommended) <./user_guides.setup.feature_store_operator>`
8+
__________________________________________________________
9+
10+
The feature store marketplace operator can be used to setup the feature store api server in an existing OKE cluster while also utilising an existing MySQL database. It will also help setup authentication and authorization using OCI. For more details, see :doc:`Marketplace operator <./user_guides.setup.feature_store_operator>`
11+
12+
:doc:`Setup via Helm Charts <./user_guides.setup.helm_chart>`
13+
_______________________________
14+
15+
We can manually export images to OCIR using Marketplace UI and then deploy the obtained Helm Chart. Optionally, we can also setup Feature Store API Gateway stack for authentication and authorization.
16+
17+
.. _Database configuration:
18+
19+
Database configuration
20+
-----------------------
21+
22+
Feature Store can be configured to use your existing MySQL database. It supports two types of authentication:
23+
24+
1. Basic (Not recommended): The password is stored as plaintext in the API server.
25+
2. Vault (Recommended): The password is stored in an encrypted format inside `OCI Vault <https://docs.oracle.com/en-us/iaas/Content/KeyManagement/Concepts/keyoverview.htm>`_.
26+
27+
Storing the password in Vault:
28+
29+
1. (Optional) Create a new Vault.
30+
2. (Required) Create a secret of plain-text type containing the db password.
31+
3. (Required) Additional policies for the Feature Store API dynamic group to allow reading the secret from Vault:
32+
- ``Allow dynamic-group <feature-store-dynamic-group> to use secret-family in tenancy``
33+
34+
Here ``feature-store-dynamic-group`` is the dynamic group corresponding to the instances of the OKE nodepool where the server is deployed. `Dynamic groups <https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm#:~:text=Dynamic%20groups%20allow%20you%20to,against%20Oracle%20Cloud%20Infrastructure%20services.>`_
35+
36+
37+
.. _Policies:
38+
Policies
39+
---------
40+
41+
Policies required by the user deploying:
42+
43+
.. code-block:: text
44+
45+
allow group <user-group> to manage clusters in compartment <compartment-name>
46+
allow group <user-group> to use repos in compartment <compartment-name>
47+
allow group <user-group> to manage marketplace-listings in compartment <compartment-name>
48+
allow group <user-group> to read compartments in compartment <compartment-name>
49+
allow group <user-group> to manage app-catalog-listing in compartment <compartment-name>
50+
allow group <user-group> to read object-family in compartment <compartment-name>
51+
52+
53+
The policies required by the Feature Store API server are:
54+
55+
.. code-block:: text
56+
57+
allow dynamic-group <feature-store-dynamic-group> to read compartments in tenancy
58+
59+
allow dynamic-group <feature-store-dynamic-group> to manage data-catalog-family in tenancy
60+
61+
allow dynamic-group <feature-store-dynamic-group> to inspect data-science-models in tenancy
62+
63+
64+
Here ``feature-store-dynamic-group`` is the dynamic group corresponding to the instances of the OKE nodepool where the server is deployed. `Dynamic groups <https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm#:~:text=Dynamic%20groups%20allow%20you%20to,against%20Oracle%20Cloud%20Infrastructure%20services.>`_
65+
66+
.. _Known Issues:
67+
68+
Known Issues
69+
-------------
70+
71+
1. Deployment doesn't work in Virtual Nodepool as the Feature Store API server relies on Instance Principal authentication.
72+
73+
74+
.. toctree::
75+
:maxdepth: 1
76+
77+
./user_guides.setup.feature_store_operator
78+
./user_guides.setup.helm_chart

‎ads/feature_store/docs/source/user_guides.setup.feature_store_operator.rst‎

Lines changed: 7 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
=================================
2-
Feature Store Deployment Operator
2+
Feature Store Deployment Operator (Recommended)
33
=================================
44

55
The Feature Store marketplace operator uses your current infrastructure to set up a Feature Store. It helps in setting up the Feature Store API server in your exisiting OKE cluster and MySQL database.
@@ -12,22 +12,19 @@ The Feature Store Marketplace Operator can be installed from PyPi using the foll
1212

1313
.. code-block:: bash
1414
15-
python3 -m pip install "https://github.com/oracle/accelerated-data-science.git@feature-store-marketplace[feature-store-marketplace]"
15+
python3 -m pip install "https://github.com/oracle/accelerated-data-science.git@oracle-ads[feature-store-marketplace]"
1616
1717
1818
After that, the Operator is ready to go!
1919

2020
Configuration
2121
-------------
2222

23-
After having set up ads opctl on your desired machine using ads opctl configure, you are ready to begin setting up Feature Store. At a minimum, you need to provide the following details about your infrastructure:
23+
After having set up `ads opctl` on your desired machine using `ads opctl configure`, you are ready to begin setting up Feature Store. At a minimum, you need to provide the following details about your infrastructure:
2424

2525
- The path to the OCIR repository where Feature Store container images are cloned.
2626
- The compartment ID where Feature Store is set up.
27-
28-
.. seealso::
29-
:ref:`Database configuration`
30-
27+
- :ref:`Database configuration details <Database configuration>`
3128
- The app name to use for Helm.
3229
- The namespace to use in the Kubernetes cluster.
3330
- The version of the Feature Store stack to install.
@@ -47,8 +44,8 @@ Before running the operator you need to configure the following requirements:
4744

4845
1. Helm: Helm is required to be installed on the machine for deploying Feature Store helm chart to the Kubernetes cluster. Ref: `Installing Helm <https://helm.sh/docs/intro/install/>`_
4946
2. Kubectl: Kubectl is required to be installed to deploy the helm chart to the cluster. Ref: `Installing Kubectl <https://kubernetes.io/docs/tasks/tools/>`_
50-
3. :ref:`Policies`: Required policies for API server.
51-
4. `Setup cluster access locally: <https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengdownloadkubeconfigfile.htm#:~:text=Under%20Containers%20%26%20Artifacts%2C%20click%20Kubernetes,shows%20details%20of%20the%20cluster>`_
47+
3. :ref:`Policies`: Required policies for API server and user running the operator.
48+
4. `Setup cluster access locally <https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengdownloadkubeconfigfile.htm#:~:text=Under%20Containers%20%26%20Artifacts%2C%20click%20Kubernetes,shows%20details%20of%20the%20cluster>`_
5249

5350

5451
Run
@@ -58,45 +55,9 @@ After the feature_store_marketplace.yaml is written using the init step above, y
5855

5956
.. code-block:: bash
6057
61-
ads operator run -f feature_store_marketplace.yaml
62-
63-
64-
**Common Issues**
65-
-- TODO --
66-
67-
68-
.. _Policies:
69-
70-
Policies
71-
---------
72-
73-
The policies required by the Feature Store API server are:
74-
75-
.. code-block:: text
76-
77-
allow dynamic-group <feature-store-dynamic-group> to read compartments in tenancy
78-
79-
allow dynamic-group <feature-store-dynamic-group> to manage data-catalog-family in tenancy
80-
81-
allow dynamic-group <feature-store-dynamic-group> to insect data-science-models in tenancy
82-
83-
Here ``feature-store-dynamic-group`` is the dynamic group corresponding to the instances of the OKE nodepool where the server is deployed. `Dynamic groups <https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm#:~:text=Dynamic%20groups%20allow%20you%20to,against%20Oracle%20Cloud%20Infrastructure%20services.>`_
84-
85-
.. _Database configuration:
86-
87-
Database configuration
88-
-----------------------
58+
ads operator run -f feature_store_marketplace.yaml -b marketplace.python
8959
90-
Feature Store can be configured to use your existing MySQL database. It supports two types of authentication:
9160
92-
1. Basic (Not recommended): The password is stored as plaintext in the API server.
93-
2. Vault (Recommended): The password is stored in an encrypted format inside `OCI Vault <https://docs.oracle.com/en-us/iaas/Content/KeyManagement/Concepts/keyoverview.htm>`_.
9461
95-
Storing the password in Vault:
9662
97-
1. (Optional) Create a new Vault.
98-
2. (Required) Create a secret of plain-text type containing the db password.
99-
3. (Required) Additional policies for the Feature Store API dynamic group to allow reading the secret from Vault:
100-
- ``Allow dynamic-group <feature-store-dynamic-group> to use secret-family in tenancy``
10163
102-
Here ``feature-store-dynamic-group`` is the dynamic group corresponding to the instances of the OKE nodepool where the server is deployed. `Dynamic groups <https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm#:~:text=Dynamic%20groups%20allow%20you%20to,against%20Oracle%20Cloud%20Infrastructure%20services.>`_

‎ads/feature_store/docs/source/user_guides.setup.helm_chart.rst‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
=================================
2+
Setup using Helm Charts
3+
=================================
4+
5+
We always suggest to deploy feature store via the :doc:`Feature Store Operator <./user_guides.setup.feature_store_operator>` to setup Feature Store API server in OKE Cluster. This method should preferably be used only when the operator can not satisfy your requirements
6+
as it is much simpler to do the setup via the operator.
7+
8+
9+
Prerequisites
10+
_____________
11+
12+
- Setup `MySQL Database <https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/mysql-database/doc/overview-mysql-database-service.html>`_ so that it is reachable from the API server
13+
.. seealso::
14+
:ref:`Database configuration`
15+
- :ref:`Helm Setup`
16+
- :ref:`Policies`
17+
- `Setup cluster access locally <https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengdownloadkubeconfigfile.htm#:~:text=Under%20Containers%20%26%20Artifacts%2C%20click%20Kubernetes,shows%20details%20of%20the%20cluster>`_
18+
19+
Steps to deploy API server image
20+
________________________________
21+
22+
- Export the package to OCIR using the `Feature store marketplace listing <https://cloud.oracle.com/marketplace/application/ocid1.mktpublisting.oc1.iad.amaaaaaabiudgxya26lzh2dsyvg7cfzgllvdl6xo5phz4mnsoktxeutecrvq>`_
23+
- Wait for export work request to complete
24+
- Identify the Helm chart and API images exported to OCIR:
25+
- Helm chart image would be of format: ``<ocir-image>:<version>``
26+
- API image would be of format: ``<ocir-image>:<export-number>-<image-id>-<tenancy-namespace>-feature-store-api-<version>``
27+
28+
- :ref:`Create kubernetes docker secret <Kubernetes secret>`
29+
30+
- :ref:`Generate custom values.yaml for deployment <Helm values>`
31+
32+
- Install the helm chart
33+
34+
.. code-block:: bash
35+
36+
helm upgrade feature-store-api oci://<helm-chart-image-path> --namespace feature-store --values <path-to-values-yaml> --timeout 300s --wait -i
37+
- (Optional) `Setup Feature Store API Gateway <https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/feature_store/apigw_terraform>`_
38+
39+
40+
Appendix
41+
________
42+
43+
.. _Helm Setup:
44+
45+
Setup Helm to use OCIR
46+
______________________
47+
48+
To login to Container Registry using the Helm CLI:
49+
50+
- If you already have an auth token, go to the next step. Otherwise:
51+
- In the top-right corner of the Console, open the Profile menu and then click User settings to view the details.
52+
- On the Auth Tokens page, click Generate Token.
53+
- Enter a friendly description for the auth token. Avoid entering confidential information.
54+
- Click Generate Token. The new auth token is displayed.
55+
- Copy the auth token immediately to a secure location from where you can retrieve it later, because you won't see the auth token again in the Console.
56+
- Close the Generate Token dialog.
57+
58+
- In a terminal window on the client machine running Docker, log in to Container Registry by entering ``helm registry login <region-key>.ocir.io``, where <region-key> corresponds to the key for the Container Registry region you're using. For example, ``helm registry login iad.ocir.io``. See `Availability by Region <https://docs.oracle.com/en-us/iaas/Content/Registry/Concepts/registryprerequisites.htm#regional-availability>`_.
59+
- When prompted for a username, enter your username in the format <tenancy-namespace>/<username>, where <tenancy-namespace> is the auto-generated Object Storage namespace string of your tenancy (as shown on the Tenancy Information page). For example, ansh81vru1zp/jdoe@acme.com. If your tenancy is federated with Oracle Identity Cloud Service, use the format <tenancy-namespace>/oracleidentitycloudservice/<username>.
60+
- When prompted for a password, enter the auth token you copied earlier.
61+
62+
.. _Kubernetes secret:
63+
64+
Kubernetes Docker Secret Configuration
65+
__________________________________________________________
66+
- If you don't already have an auth token refer :ref:`<helm setup>`
67+
- `Login to Kubernetes cluster <https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengdownloadkubeconfigfile.htm#:~:text=Under%20Containers%20%26%20Artifacts%2C%20click%20Kubernetes,shows%20details%20of%20the%20cluster>`_
68+
- Run command
69+
70+
.. code-block:: bash
71+
72+
kubectl create secret docker-registry <secret-name> --docker-server=<region-key>.ocir.io --docker-username=<tenancy-namespace>/<username> --docker-password=<auth token>
73+
74+
.. _Helm values:
75+
76+
Helm values configuration:
77+
__________________________________________________________
78+
79+
- Minimal Helm values example for getting started:
80+
81+
.. code-block:: yaml
82+
83+
db:
84+
configuredDB: MYSQL
85+
mysql:
86+
authType: BASIC
87+
basic:
88+
password: #enter-db-password-here
89+
jdbcURL: jdbc:mysql://<db-ip>:3306/FeatureStore?createDatabaseIfNotExist=true
90+
username: #enter-db-username-here
91+
imagePullSecrets:
92+
- name: #enter secret name containing docker secret here
93+
oci_meta:
94+
images:
95+
repo: #ocir repo: <region-key>.ocir.io/<tenancy-namespace>/repository
96+
api:
97+
image: #ocir image: The name of image>
98+
tag: #API Image tag
99+
authoriser:
100+
image: na
101+
tag: na
102+
103+
104+
105+
- All available Helm values
106+
107+
.. code-block:: yaml
108+
109+
oci_meta:
110+
repo: #ocir repo: <region-key>.ocir.io/<tenancy-namespace>/repository
111+
images:
112+
api:
113+
image: #ocir image: The name of image
114+
tag: #API Image tag
115+
authoriser: # We don't want to deploy this image. This image will be deployed with OCI functions
116+
image: na
117+
tag: na
118+
119+
imagePullSecrets:
120+
- name: #name-of-docker-secret-with-credentials
121+
122+
db:
123+
configuredDB: #type of DB configured. possible values: "MYSQL"
124+
mysql:
125+
authType: #Type of authentication to use for connecting to database.
126+
# Possible values: 'BASIC', 'VAULT'
127+
jdbcURL: #JDBC URL of the MySQL server
128+
username: #Name of the user on MySQL server
129+
basic:
130+
password: #Password to mysql server in plain-text format
131+
vault:
132+
vaultOcid: #OCID of the vault where the secret is kept
133+
secretName: #Name of the secret used for connecting to vault
134+
135+
resources: #https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
136+
137+
nameOverride: #Value for label app.kubernetes.io/name
138+
139+
podSecurityContext: #Pod security #https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
140+
141+
securityContext: #Container Security context #https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
142+
143+
deploymentStrategy: #This block is directly inserted into pod spec
144+
#https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
145+
146+
147+
nodeSelector: {} #Pod node selector
148+
#https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
149+
150+
tolerations: [] #Pod tolerations
151+
#https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
152+
153+
affinity: {} #Pod affinity
154+
#https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
155+
156+
replicaCount: #Pod replicas
157+
#https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
158+
159+
autoscaling: #Horizontal pod scaling details
160+
#https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
161+
enabled:
162+
minReplicas:
163+
maxReplicas:
164+
targetCPUUtilizationPercentage:
165+
targetMemoryUtilizationPercentage:
166+
scaleUp:
167+
stabilizationWindowSeconds:
168+
periodSeconds:
169+
podCount:
170+
percentage:
171+
scaleDown:
172+
stabilizationWindowSeconds:
173+
periodSeconds:
174+
podCount:
175+
percentage:
176+
177+
applicationEnv:
178+
containerName: #Container name
179+
180+
livenessProbe: # Liveness probe details
181+
initialDelaySeconds:
182+
periodSeconds:
183+
timeoutSeconds:
184+
failureThreshold:
185+
186+
readinessProbe: # Readiness probe details
187+
initialDelaySeconds:
188+
periodSeconds:
189+
timeoutSeconds:
190+
failureThreshold:
191+

‎ads/opctl/backend/marketplace/marketplace_utils.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,6 @@ def list_container_images(
146146
artifact_client = OCIClientFactory(**authutil.default_signer()).artifacts
147147
list_container_images_response = artifact_client.list_container_images(
148148
compartment_id=compartment_id,
149-
compartment_id_in_subtree=True,
150149
sort_by="TIMECREATED",
151150
repository_name=ocir_image_path,
152151
)

0 commit comments

Comments
 (0)