|
| 1 | +Talk Bot App with Transformers |
| 2 | +============================== |
| 3 | + |
| 4 | +`Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.` |
| 5 | + |
| 6 | +In this article, we'll demonstrate how straightforward it is to leverage the extensive capabilities |
| 7 | +of the `Transformers <https://github.com/huggingface/transformers>`_ library in your Nextcloud application. |
| 8 | + |
| 9 | +Specifically, we'll cover: |
| 10 | + |
| 11 | +* Setting the cache path for the Transformers library |
| 12 | +* Downloading AI models during the application initialization step |
| 13 | +* Receiving messages from Nextcloud Talk Chat and sending them to a language model |
| 14 | +* Sending the language model's reply back to the Nextcloud Talk Chat |
| 15 | + |
| 16 | +Packaging the Application |
| 17 | +""""""""""""""""""""""""" |
| 18 | + |
| 19 | +Firstly, let's touch upon the somewhat mundane topic of application packaging. |
| 20 | + |
| 21 | +For this example, we've chosen Debian as the base image because it simplifies the installation of required Python packages. |
| 22 | + |
| 23 | +.. code-block:: |
| 24 | +
|
| 25 | + FROM python:3.11-bookworm |
| 26 | +
|
| 27 | +
|
| 28 | +While Alpine might be a better choice in some situations, that's not the focus of this example. |
| 29 | + |
| 30 | +.. note:: The selection of a suitable base image for an application is a complex topic that merits its own in-depth discussion. |
| 31 | + |
| 32 | +Requirements |
| 33 | +"""""""""""" |
| 34 | + |
| 35 | +.. literalinclude:: ../examples/as_app/talk_bot_ai/requirements.txt |
| 36 | + |
| 37 | +We opt for the latest version of the Transformers library. |
| 38 | +Because the example was developed on a Mac, we ended up using Torchvision. |
| 39 | + |
| 40 | +`If you're working solely with Nvidia, you're free to use TensorFlow instead of PyTorch.` |
| 41 | + |
| 42 | +Next, we integrate the latest version of `nc_py_api` to minimize code redundancy and focus on the application's logic. |
| 43 | + |
| 44 | +Model Downloading |
| 45 | +""""""""""""""""" |
| 46 | + |
| 47 | +**When Should We Download the Language Model?** |
| 48 | + |
| 49 | +Although the example uses the smallest model available, weighing in at 300 megabytes, it's common knowledge that larger language models can be substantially bigger. |
| 50 | +Downloading such models should not begin when a processing request is already received. |
| 51 | + |
| 52 | +So we have two options: |
| 53 | + |
| 54 | +* Heartbeat |
| 55 | +* enabled_handler |
| 56 | + |
| 57 | +This can't be accomplished in the **app on/off handler** as Nextcloud expects an immediate response regarding the app's operability. |
| 58 | + |
| 59 | +Thus, we place the model downloading within the Heartbeat: |
| 60 | + |
| 61 | +.. code-block:: |
| 62 | +
|
| 63 | + # Thread that performs model download. |
| 64 | + def download_models(): |
| 65 | + pipeline("text2text-generation", model=MODEL_NAME) # this will download model |
| 66 | +
|
| 67 | +
|
| 68 | + def heartbeat_handler() -> str: |
| 69 | + global MODEL_INIT_THREAD |
| 70 | + print("heartbeat_handler: called") # for debug |
| 71 | + # if it is the first heartbeat, then start background thread to download a model |
| 72 | + if MODEL_INIT_THREAD is None: |
| 73 | + MODEL_INIT_THREAD = Thread(target=download_models) |
| 74 | + MODEL_INIT_THREAD.start() |
| 75 | + print("heartbeat_handler: started initialization thread") # for debug |
| 76 | + # if thread is finished then we will have "ok" in response, and AppAPI will consider that program is ready. |
| 77 | + r = "init" if MODEL_INIT_THREAD.is_alive() else "ok" |
| 78 | + print(f"heartbeat_handler: result={r}") # for debug |
| 79 | + return r |
| 80 | +
|
| 81 | +
|
| 82 | + @APP.on_event("startup") |
| 83 | + def initialization(): |
| 84 | + # Provide our custom **heartbeat_handler** to set_handlers |
| 85 | + set_handlers(APP, enabled_handler, heartbeat_handler) |
| 86 | +
|
| 87 | +
|
| 88 | +.. note:: While this may not be the most ideal way to download models, it remains a viable method. |
| 89 | + In the future, a more efficient wrapper for model downloading is planned to make the process even more convenient. |
| 90 | + |
| 91 | +Model Storage |
| 92 | +""""""""""""" |
| 93 | + |
| 94 | +By default, models will be downloaded to a directory that's removed when updating the app. |
| 95 | +o persistently store the models even after updates, add the following line to your code: |
| 96 | + |
| 97 | +.. code-block:: |
| 98 | +
|
| 99 | + from nc_py_api.ex_app import persist_transformers_cache # noqa # isort:skip |
| 100 | +
|
| 101 | +This will set ``TRANSFORMERS_CACHE`` environment variable to point to the application persistent storage. |
| 102 | +Import of this **must be** on top before importing any code that perform the import of the ``transformers`` library. |
| 103 | + |
| 104 | +And that is all, ``transformers`` will automatically download all |
| 105 | +models you use to the **Application Persistent Storage** and AppAPI will keep it between updates. |
| 106 | + |
| 107 | +Working with Language Models |
| 108 | +"""""""""""""""""""""""""""" |
| 109 | + |
| 110 | +Finally, we arrive at the core aspect of the application, where we interact with the **Language Model**: |
| 111 | + |
| 112 | +.. code-block:: |
| 113 | +
|
| 114 | + def ai_talk_bot_process_request(message: talk_bot.TalkBotMessage): |
| 115 | + # Process only messages started with **@ai** |
| 116 | + r = re.search(r"@ai\s(.*)", message.object_content["message"], re.IGNORECASE) |
| 117 | + if r is None: |
| 118 | + return |
| 119 | + model = pipeline("text2text-generation", model=MODEL_NAME) |
| 120 | + # Pass all text after **@ai** we to the Language model. |
| 121 | + response_text = model(r.group(1), max_length=64, do_sample=True)[0]["generated_text"] |
| 122 | + AI_BOT.send_message(response_text, message) |
| 123 | +
|
| 124 | +
|
| 125 | +Simply put, the AI logic is just two lines of code when using Transformers, which is incredibly efficient and cool. |
| 126 | + |
| 127 | +Messages from the AI model are then sent back to Talk Chat as you would expect from a typical chatbot. |
| 128 | + |
| 129 | +That's it for now! Stay tuned—this is merely the start of an exciting journey into the integration of AI and chat functionality in Nextcloud. |
0 commit comments