Improve Performance: CoPilot: users experience #110

quge009 · 2025-10-27T22:54:50Z

This PR is mainly about improving the user experience.

Changes made to optimize users perceived latency and reading experience:
- Implement streaming output for LLMSession class, change the final answer generation call to streaming output, to post the answer to user as-soon-as the first few tokens are ready.
- Implement the push_frontend method to leverage the steaming output to feedback the CoPilot progress status message to user in real-time, to manage users' experience during waiting for the answer.
- Add auto scroll feature for frontend plugin to enhance readability.
Changes made to reduce the average_response_latency (defined as time between question receival and answer posting):
- Refactor several components' (SmartHelp, LTP, ...) implementation into classes, to make it possible to preserve the states when necessary.
- Reuse the same llm_session instance for requests within the same conversation, by avoiding unnecessary https re-connection in initialization.
- Implement a new question parsing function to combine contextualization and classification llm calls into one efficient call, to reduce time.
- Move prompt reading to instance initialization, to avoid unnecessary file I/O operations.
Also a minor bug fix is included:
- Change the assignment of 'turnId' to frontend.

Effectiveness of this PR:

Impact on accuracy
- No change
Impact on response latency
- ~15% response time reduction on average
- ~50% response time reduction for extreme simple question

…thorized users

…, pre-load prompts for classifier, contextualizer

…gement

…sification session

…same username

…nt route

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

hippogr · 2025-11-07T03:01:09Z

contrib/copilot-plugin/src/app/ChatBox.tsx

+
+        // Process all complete SSE messages in buffer
+        let sepIndex;
+        while ((sepIndex = buffer.indexOf('\n\n')) !== -1) {


I am not sure if this will cause infinite loop here when buffer.indexOf('\n\n') !== -1, please make sure that the loop can be breaked from the loop no matter which parts will be executed

hippogr · 2025-11-07T03:13:10Z

src/copilot-chat/src/copilot_agent/ltp/ltp.py

-SUB_FEATURE = 'ltp'
-
-SKIP_LUCIA_CONTROLLER_EXECUTION = True
+class LTP:


just curious, why naming this as LTP? From the comments below, LtpQueryEngine is a better name than the project name. 😁

hippogr · 2025-11-07T03:17:09Z

src/pylon/deploy/pylon-config/location.conf.template

  proxy_send_timeout 2m;
 }

+location ~ ^/copilot/api/stream(.*)$ {


do we need to remove the original part above?

hippogr · 2025-11-07T03:20:30Z

src/copilot-chat/src/copilot_agent/ltp/ltp.py

+        """
+        self.llm_session = llm_session
+        self.feature_skipped = True
+        self.ltp_documentation = get_prompt_from(os.path.join(PROMPT_DIR, self.SUB_FEATURE, 'ltp_documentation.txt'))


I checked the file and find that this function may through an exception. Is that the exception you want in init function?

hippogr · 2025-11-07T03:22:28Z

src/copilot-chat/src/copilot_agent/ltp/ltp.py

+            query, end_time_stamp, parallel, param = gen_promql_query(self.SUB_FEATURE, question, self.llm_session)
+
+            if not query:
+                logger.info(f'No query found in the response, query is {query}')


just a suggestion and you can keep the code. 😊 Is this a warning instead of info?

hippogr · 2025-11-07T04:34:17Z

src/copilot-chat/src/copilot_agent/utils/smart_help.py

+                                  help_msg['sku'] + 
+                                  help_msg['workload'])
+        else:
+            self.capability_str = help_msg['feature']


a kind reminder, both verion f3 and f4 use this one?

hippogr · 2025-11-07T04:37:57Z

src/copilot-chat/src/copilot_agent/copilot_service.py

+        except Exception as e:
+            logger.error(f"Failed to parse JSON body for stream_operation: {e}")
+            return jsonify({"status": "error", "message": "invalid json"}), 400
+


remove extra empty line

hippogr · 2025-11-07T04:39:29Z

src/copilot-chat/src/copilot_agent/copilot_service.py

+                try:
+                    llm_session.clear_instance_stream_callback()
+                except Exception:
+                    logger.debug('Failed to clear instance stream callback')


What will happen if exception happens here?

hippogr · 2025-11-07T04:42:28Z

src/copilot-chat/src/copilot_agent/copilot_turn.py

-        # verion f3, resolves objective 8 (Lucia Training Platform)
-        if self._version == 'f3':
-            if obj.count('8') > 0:
+        # debug only


should we remove these code which are for debug only?

hippogr · 2025-11-07T04:44:22Z

src/copilot-chat/src/copilot_agent/copilot_turn.py

+                help_keys = ['unsupported_question']
+                answer = self.smart_help.generate(question, help_keys, True)
+                debug = {}
+            elif obj.count('8') > 0:


I see many magic numbers like 8, 3, 9 here, plase add some comments or use const variable with meaningful name to replace them so other developers can understand them well.

quge009 and others added 20 commits October 27, 2025 15:40

add: stream output

d79b3c5

change: use frontend to assign turnId

4278f23

modify: state update help message

7e01649

improve: auto scroll

434d8ec

adding more status update message

b68c194

minor: add new status update message

1adf126

Add feature: multi user concurrency

92d2869

fix: minor bug for non-streaming api

a0c50a0

improve user experience: post unauthorized access information to unau…

f4c3f1b

…thorized users

code cleanup: remove unnecessary llmsession instances

7776476

code refactor: stage 0

1e1285c

code refactor: stage 1

99c280a

code refactor: stage 2

7f989e7

code refactor: stage 3, use the same llmsession for each conversation…

856f861

…, pre-load prompts for classifier, contextualizer

code refactor: stage 4, change ltp into a class for easier state mana…

9f8a8b0

…gement

code refactor: stage 5, fix chuck accumulation bug

798d738

code refactor: stage 6, smart help into a class

173a9b2

minor bug fix

01384ec

improve: response latency by merge small llm chat calls into one clas…

16f4c63

…sification session

improve: response latency, by reuse llmsession for requests from the …

27d8f64

…same username

quge009 temporarily deployed to auto-test October 27, 2025 22:54 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 27, 2025 23:44 — with GitHub Actions Inactive

fix bug: missing import, changed prompt file name

138ba06

quge009 had a problem deploying to auto-test October 28, 2025 00:22 — with GitHub Actions Failure

quge009 temporarily deployed to auto-test October 28, 2025 19:11 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 19:21 — with GitHub Actions Inactive

quge009 changed the title ~~tmp~~ Improve Performance: CoPilot, response latency, user expectation Oct 28, 2025

quge009 changed the title ~~Improve Performance: CoPilot, response latency, user expectation~~ Improve Performance: CoPilot: response latency, user expectation Oct 28, 2025

quge009 changed the title ~~Improve Performance: CoPilot: response latency, user expectation~~ Improve Performance: CoPilot: users' perceived response latency Oct 28, 2025

quge009 marked this pull request as ready for review October 28, 2025 20:04

quge009 added 2 commits October 28, 2025 13:35

resolve review comment: add appropriate null handling

b9f3d52

update: nginx configuration to add the new /copilot/api/stream endpoi…

e42bea9

…nt route

quge009 temporarily deployed to auto-test October 28, 2025 20:36 — with GitHub Actions Inactive

Update src/copilot-chat/src/copilot_agent/copilot_conversation.py

372cfe8

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

quge009 temporarily deployed to auto-test October 28, 2025 20:41 — with GitHub Actions Inactive

remove unnecessary comment

3c41e32

quge009 temporarily deployed to auto-test October 28, 2025 20:46 — with GitHub Actions Inactive

Update src/copilot-chat/src/copilot_agent/copilot_turn.py

32a2906

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

quge009 temporarily deployed to auto-test October 28, 2025 20:48 — with GitHub Actions Inactive

resolve review comment: remove consle log

4019f8a

quge009 temporarily deployed to auto-test October 28, 2025 20:56 — with GitHub Actions Inactive

Update src/copilot-chat/src/copilot_agent/copilot_service.py

f8a5211

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

quge009 temporarily deployed to auto-test October 28, 2025 20:59 — with GitHub Actions Inactive

Update src/copilot-chat/src/copilot_agent/ltp/ltp.py

66d3a90

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

quge009 temporarily deployed to auto-test October 28, 2025 21:00 — with GitHub Actions Inactive

Update src/copilot-chat/src/copilot_agent/copilot_conversation.py

735ec38

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

quge009 temporarily deployed to auto-test October 28, 2025 21:00 — with GitHub Actions Inactive

update: remove unused function

53ddf4d

quge009 temporarily deployed to auto-test October 28, 2025 21:02 — with GitHub Actions Inactive

quge009 changed the title ~~Improve Performance: CoPilot: users' perceived response latency~~ Improve Performance: CoPilot: users experience Oct 28, 2025

improve: robustness, gracefully handling if classification fail

34b95ae

quge009 temporarily deployed to auto-test October 28, 2025 21:46 — with GitHub Actions Inactive

quge009 had a problem deploying to auto-test October 28, 2025 22:18 — with GitHub Actions Failure

quge009 had a problem deploying to auto-test October 28, 2025 22:20 — with GitHub Actions Failure

quge009 had a problem deploying to auto-test October 28, 2025 22:24 — with GitHub Actions Failure

change classifier version for deployment

cb34b54

quge009 temporarily deployed to auto-test October 28, 2025 23:34 — with GitHub Actions Inactive

quge009 had a problem deploying to auto-test October 28, 2025 23:38 — with GitHub Actions Failure

quge009 temporarily deployed to auto-test October 29, 2025 18:33 — with GitHub Actions Inactive

hippogr reviewed Nov 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve Performance: CoPilot: users experience #110

Improve Performance: CoPilot: users experience #110

Uh oh!

quge009 commented Oct 27, 2025 •

edited

Loading

Uh oh!

hippogr Nov 7, 2025

Uh oh!

hippogr Nov 7, 2025

Uh oh!

hippogr Nov 7, 2025

Uh oh!

hippogr Nov 7, 2025

Uh oh!

hippogr Nov 7, 2025

Uh oh!

hippogr Nov 7, 2025

Uh oh!

hippogr Nov 7, 2025

Uh oh!

hippogr Nov 7, 2025

Uh oh!

hippogr Nov 7, 2025

Uh oh!

hippogr Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Improve Performance: CoPilot: users experience #110

Are you sure you want to change the base?

Improve Performance: CoPilot: users experience #110

Uh oh!

Conversation

quge009 commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR is mainly about improving the user experience.

Effectiveness of this PR:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

quge009 commented Oct 27, 2025 •

edited

Loading