Session API integration

README.md

@@ -24,17 +24,23 @@ lk app create --template agent-starter-swift --sandbox <token_server_sandbox_id>
 Then, build and run the app from Xcode by opening `VoiceAgent.xcodeproj`. You may need to adjust your app signing settings to run the app on your device.
 
 > [!NOTE]
-> To setup without the LiveKit CLI, clone the repository and then either create a `VoiceAgent/.env.xcconfig` with a `LIVEKIT_SANDBOX_ID` (if using a [Sandbox Token Server](https://cloud.livekit.io/projects/p_/sandbox/templates/token-server)), or open `TokenService.swift` and add your [manually generated](#token-generation) URL and token.
+> To setup without the LiveKit CLI, clone the repository and then either create a `VoiceAgent/.env.xcconfig` with a `LIVEKIT_SANDBOX_ID` (if using a [Sandbox Token Server](https://cloud.livekit.io/projects/p_/sandbox/templates/token-server)), or modify `VoiceAgent/VoiceAgentApp.swift` to replace the `SandboxTokenSource` with a custom token source implementation.
 
 ## Feature overview
 
 This starter app has support for a number of features of the agents framework, and is configurable to easily enable or disable them in code based on your needs as you adapt this template to your own use case.
 
 ### Text, video, and voice input
 
-This app supports text, video, and/or voice input according to the needs of your agent. To update the features enabled in the app, edit `VoiceAgent/VoiceAgentApp.swift` and update `AgentFeatures.current` to include or exclude the features you need.
+This app supports text, video, and/or voice input according to the needs of your agent. To update the features enabled in the app, edit `VoiceAgent/VoiceAgentApp.swift` and modify the `.environment()` modifiers to enable or disable features.
 
-By default, only voice and text input are enabled.
+By default, all features (voice, video, and text input) are enabled. To disable a feature, change the value from `true` to `false`:
+
+```swift
+.environment(\.voiceEnabled, true)   // Enable voice input
+.environment(\.videoEnabled, false)  // Disable video input
+.environment(\.textEnabled, true)    // Enable text input
+```
 
 Available input types:
 - `.voice`: Allows the user to speak to the agent using their microphone. **Requires microphone permissions.**
@@ -43,23 +49,23 @@ Available input types:
 
 If you have trouble with screensharing, refer to [the docs](https://docs.livekit.io/home/client/tracks/screenshare/) for more setup instructions.
 
-### Preconnect audio buffer
+### Session
 
-This app uses `withPreConnectAudio` to capture and buffer audio before the room connection completes. This allows the connection to appear "instant" from the user's perspective and makes your app more responsive. To disable this feature, remove the call to `withPreConnectAudio` as below:
+The app is built on top of two main observable components from the [LiveKit Swift SDK](https://github.com/livekit/client-sdk-swift):
+- `Session` object to connect to the LiveKit infrastructure, interact with the `Agent`, its local state, and send/receive text messages.
+- `LocalMedia` object to manage the local media tracks (audio, video, screen sharing) and their lifecycle.
+
+### Preconnect audio buffer
 
-- Location: `VoiceAgent/App/AppViewModel.swift` → `connectWithVoice()`
-- To disable preconnect buffering but keep voice:
-  - Replace the `withPreConnectAudio { ... }` block with a standard `room.connect` call and enable the microphone after connect, for example:
-    - Connect with `connectOptions: .init(enableMicrophone: true)` without wrapping in `withPreConnectAudio`, or
-    - Connect with microphone disabled and call `room.localParticipant.setMicrophone(enabled: true)` after connection.
+This app enables `preConnectAudio` by default to capture and buffer audio before the room connection completes. This allows the connection to appear "instant" from the user's perspective and makes your app more responsive. To disable this feature, set `preConnectAudio` to `false` in `SessionOptions` when creating the `Session`.
 
 ### Virtual avatar support
 
-If your agent publishes a [virtual avatar](https://docs.livekit.io/agents/integrations/avatar/), this app will automatically render the avatar’s camera feed in `AgentParticipantView` when available.
+If your agent publishes a [virtual avatar](https://docs.livekit.io/agents/integrations/avatar/), this app will automatically render the avatar's camera feed in `AgentView` when available.
 
 ## Token generation in production
 
-In a production environment, you will be responsible for developing a solution to [generate tokens for your users](https://docs.livekit.io/home/server/generating-tokens/) which is integrated with your authentication solution. You should disable your sandbox token server and modify `TokenService.swift` to use your own token server.
+In a production environment, you will be responsible for developing a solution to [generate tokens for your users](https://docs.livekit.io/home/server/generating-tokens/) which is integrated with your authentication solution. You should replace your `SandboxTokenSource` with an `EndpointTokenSource` or your own `TokenSourceFixed` or `TokenSourceConfigurable` implementation. Additionally, you can use the `.cached()` extension to cache valid tokens and avoid unnecessary token requests.
 
 ## Running on Simulator
 

VoiceAgent.xcodeproj/project.pbxproj

@@ -10,7 +10,6 @@
 		ACAEBA5B2DE6EE970072E93E /* ReplayKit.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = ACAEBA5A2DE6EE970072E93E /* ReplayKit.framework */; };
 		ACAEBA622DE6EE970072E93E /* BroadcastExtension.appex in Embed Foundation Extensions */ = {isa = PBXBuildFile; fileRef = ACAEBA582DE6EE970072E93E /* BroadcastExtension.appex */; platformFilters = (ios, xros, ); settings = {ATTRIBUTES = (RemoveHeadersOnCopy, ); }; };
 		ACAEBA692DE6EF4B0072E93E /* LiveKit in Frameworks */ = {isa = PBXBuildFile; productRef = ACAEBA682DE6EF4B0072E93E /* LiveKit */; };
-		ACFBA1DB2D8D5CBE0021202B /* Collections in Frameworks */ = {isa = PBXBuildFile; productRef = ACFBA1DA2D8D5CBE0021202B /* Collections */; };
 		B5E1B90F2D14E9EC00A38CB6 /* LiveKitComponents in Frameworks */ = {isa = PBXBuildFile; productRef = B5E1B90E2D14E9EC00A38CB6 /* LiveKitComponents */; };
 		B5E1B9122D14E9F500A38CB6 /* LiveKit in Frameworks */ = {isa = PBXBuildFile; productRef = B5E1B9112D14E9F500A38CB6 /* LiveKit */; };
 /* End PBXBuildFile section */
@@ -23,13 +22,6 @@
 			remoteGlobalIDString = ACAEBA572DE6EE970072E93E;
 			remoteInfo = BroadcastExtension;
 		};
-		ACC2802B2DEDDA1D0023C137 /* PBXContainerItemProxy */ = {
-			isa = PBXContainerItemProxy;
-			containerPortal = B5B5E3AA2D124AE00099C9BE /* Project object */;
-			proxyType = 1;
-			remoteGlobalIDString = B5B5E3B12D124AE00099C9BE;
-			remoteInfo = VoiceAgent;
-		};
 /* End PBXContainerItemProxy section */
 
 /* Begin PBXCopyFilesBuildPhase section */
@@ -49,7 +41,6 @@
 /* Begin PBXFileReference section */
 		ACAEBA582DE6EE970072E93E /* BroadcastExtension.appex */ = {isa = PBXFileReference; explicitFileType = "wrapper.app-extension"; includeInIndex = 0; path = BroadcastExtension.appex; sourceTree = BUILT_PRODUCTS_DIR; };
 		ACAEBA5A2DE6EE970072E93E /* ReplayKit.framework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.framework; name = ReplayKit.framework; path = System/Library/Frameworks/ReplayKit.framework; sourceTree = SDKROOT; };
-		ACC280272DEDDA1D0023C137 /* VoiceAgentTests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = VoiceAgentTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
 		B5B5E3B22D124AE00099C9BE /* VoiceAgent.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = VoiceAgent.app; sourceTree = BUILT_PRODUCTS_DIR; };
 /* End PBXFileReference section */
 
@@ -82,11 +73,6 @@
 			path = BroadcastExtension;
 			sourceTree = "<group>";
 		};
-		ACC280282DEDDA1D0023C137 /* VoiceAgentTests */ = {
-			isa = PBXFileSystemSynchronizedRootGroup;
-			path = VoiceAgentTests;
-			sourceTree = "<group>";
-		};
 		B5B5E3B42D124AE00099C9BE /* VoiceAgent */ = {
 			isa = PBXFileSystemSynchronizedRootGroup;
 			exceptions = (
@@ -107,18 +93,10 @@
 			);
 			runOnlyForDeploymentPostprocessing = 0;
 		};
-		ACC280242DEDDA1D0023C137 /* Frameworks */ = {
-			isa = PBXFrameworksBuildPhase;
-			buildActionMask = 2147483647;
-			files = (
-			);
-			runOnlyForDeploymentPostprocessing = 0;
-		};
 		B5B5E3AF2D124AE00099C9BE /* Frameworks */ = {
 			isa = PBXFrameworksBuildPhase;
 			buildActionMask = 2147483647;
 			files = (
-				ACFBA1DB2D8D5CBE0021202B /* Collections in Frameworks */,
 				B5E1B90F2D14E9EC00A38CB6 /* LiveKitComponents in Frameworks */,
 				B5E1B9122D14E9F500A38CB6 /* LiveKit in Frameworks */,
 			);
@@ -140,7 +118,6 @@
 			children = (
 				B5B5E3B42D124AE00099C9BE /* VoiceAgent */,
 				ACAEBA5C2DE6EE970072E93E /* BroadcastExtension */,
-				ACC280282DEDDA1D0023C137 /* VoiceAgentTests */,
 				ACAEBA592DE6EE970072E93E /* Frameworks */,
 				B5B5E3B32D124AE00099C9BE /* Products */,
 			);
@@ -151,7 +128,6 @@
 			children = (
 				B5B5E3B22D124AE00099C9BE /* VoiceAgent.app */,
 				ACAEBA582DE6EE970072E93E /* BroadcastExtension.appex */,
-				ACC280272DEDDA1D0023C137 /* VoiceAgentTests.xctest */,
 			);
 			name = Products;
 			sourceTree = "<group>";
@@ -182,29 +158,6 @@
 			productReference = ACAEBA582DE6EE970072E93E /* BroadcastExtension.appex */;
 			productType = "com.apple.product-type.app-extension";
 		};
-		ACC280262DEDDA1D0023C137 /* VoiceAgentTests */ = {
-			isa = PBXNativeTarget;
-			buildConfigurationList = ACC2802F2DEDDA1D0023C137 /* Build configuration list for PBXNativeTarget "VoiceAgentTests" */;
-			buildPhases = (
-				ACC280232DEDDA1D0023C137 /* Sources */,
-				ACC280242DEDDA1D0023C137 /* Frameworks */,
-				ACC280252DEDDA1D0023C137 /* Resources */,
-			);
-			buildRules = (
-			);
-			dependencies = (
-				ACC2802C2DEDDA1D0023C137 /* PBXTargetDependency */,
-			);
-			fileSystemSynchronizedGroups = (
-				ACC280282DEDDA1D0023C137 /* VoiceAgentTests */,
-			);
-			name = VoiceAgentTests;
-			packageProductDependencies = (
-			);
-			productName = VoiceAgentTests;
-			productReference = ACC280272DEDDA1D0023C137 /* VoiceAgentTests.xctest */;
-			productType = "com.apple.product-type.bundle.unit-test";
-		};
 		B5B5E3B12D124AE00099C9BE /* VoiceAgent */ = {
 			isa = PBXNativeTarget;
 			buildConfigurationList = B5B5E3C12D124AE20099C9BE /* Build configuration list for PBXNativeTarget "VoiceAgent" */;
@@ -226,7 +179,6 @@
 			packageProductDependencies = (
 				B5E1B90E2D14E9EC00A38CB6 /* LiveKitComponents */,
 				B5E1B9112D14E9F500A38CB6 /* LiveKit */,
-				ACFBA1DA2D8D5CBE0021202B /* Collections */,
 			);
 			productName = VoiceAgent;
 			productReference = B5B5E3B22D124AE00099C9BE /* VoiceAgent.app */;
@@ -245,10 +197,6 @@
 					ACAEBA572DE6EE970072E93E = {
 						CreatedOnToolsVersion = 16.3;
 					};
-					ACC280262DEDDA1D0023C137 = {
-						CreatedOnToolsVersion = 16.3;
-						TestTargetID = B5B5E3B12D124AE00099C9BE;
-					};
 					B5B5E3B12D124AE00099C9BE = {
 						CreatedOnToolsVersion = 16.2;
 					};
@@ -266,15 +214,13 @@
 			packageReferences = (
 				B5E1B90D2D14E9EC00A38CB6 /* XCRemoteSwiftPackageReference "components-swift" */,
 				B5E1B9102D14E9F500A38CB6 /* XCRemoteSwiftPackageReference "client-sdk-swift" */,
-				ACFBA1D92D8D5CBE0021202B /* XCRemoteSwiftPackageReference "swift-collections" */,
 			);
 			preferredProjectObjectVersion = 77;
 			productRefGroup = B5B5E3B32D124AE00099C9BE /* Products */;
 			projectDirPath = "";
 			projectRoot = "";
 			targets = (
 				B5B5E3B12D124AE00099C9BE /* VoiceAgent */,
-				ACC280262DEDDA1D0023C137 /* VoiceAgentTests */,
 				ACAEBA572DE6EE970072E93E /* BroadcastExtension */,
 			);
 		};
@@ -288,13 +234,6 @@
 			);
 			runOnlyForDeploymentPostprocessing = 0;
 		};
-		ACC280252DEDDA1D0023C137 /* Resources */ = {
-			isa = PBXResourcesBuildPhase;
-			buildActionMask = 2147483647;
-			files = (
-			);
-			runOnlyForDeploymentPostprocessing = 0;
-		};
 		B5B5E3B02D124AE00099C9BE /* Resources */ = {
 			isa = PBXResourcesBuildPhase;
 			buildActionMask = 2147483647;
@@ -312,13 +251,6 @@
 			);
 			runOnlyForDeploymentPostprocessing = 0;
 		};
-		ACC280232DEDDA1D0023C137 /* Sources */ = {
-			isa = PBXSourcesBuildPhase;
-			buildActionMask = 2147483647;
-			files = (
-			);
-			runOnlyForDeploymentPostprocessing = 0;
-		};
 		B5B5E3AE2D124AE00099C9BE /* Sources */ = {
 			isa = PBXSourcesBuildPhase;
 			buildActionMask = 2147483647;
@@ -338,11 +270,6 @@
 			target = ACAEBA572DE6EE970072E93E /* BroadcastExtension */;
 			targetProxy = ACAEBA602DE6EE970072E93E /* PBXContainerItemProxy */;
 		};
-		ACC2802C2DEDDA1D0023C137 /* PBXTargetDependency */ = {
-			isa = PBXTargetDependency;
-			target = B5B5E3B12D124AE00099C9BE /* VoiceAgent */;
-			targetProxy = ACC2802B2DEDDA1D0023C137 /* PBXContainerItemProxy */;
-		};
 /* End PBXTargetDependency section */
 
 /* Begin XCBuildConfiguration section */
@@ -411,58 +338,6 @@
 			};
 			name = Release;
 		};
-		ACC2802D2DEDDA1D0023C137 /* Debug */ = {
-			isa = XCBuildConfiguration;
-			buildSettings = {
-				BUNDLE_LOADER = "$(TEST_HOST)";
-				CODE_SIGN_IDENTITY = "Apple Development";
-				"CODE_SIGN_IDENTITY[sdk=macosx*]" = "Apple Development";
-				CODE_SIGN_STYLE = Automatic;
-				DEAD_CODE_STRIPPING = YES;
-				DEVELOPMENT_TEAM = 76TVFCUKK7;
-				GENERATE_INFOPLIST_FILE = YES;
-				IPHONEOS_DEPLOYMENT_TARGET = 18.0;
-				MACOSX_DEPLOYMENT_TARGET = 15.0;
-				MARKETING_VERSION = 1.0;
-				PRODUCT_BUNDLE_IDENTIFIER = com.livekit.VoiceAgentTests;
-				PRODUCT_NAME = "$(TARGET_NAME)";
-				PROVISIONING_PROFILE_SPECIFIER = "";
-				SDKROOT = auto;
-				SUPPORTED_PLATFORMS = "iphoneos iphonesimulator macosx xros xrsimulator";
-				SWIFT_EMIT_LOC_STRINGS = NO;
-				SWIFT_VERSION = 5.0;
-				TARGETED_DEVICE_FAMILY = "1,2,7";
-				TEST_HOST = "$(BUILT_PRODUCTS_DIR)/VoiceAgent.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/VoiceAgent";
-				XROS_DEPLOYMENT_TARGET = 2.0;
-			};
-			name = Debug;
-		};
-		ACC2802E2DEDDA1D0023C137 /* Release */ = {
-			isa = XCBuildConfiguration;
-			buildSettings = {
-				BUNDLE_LOADER = "$(TEST_HOST)";
-				CODE_SIGN_IDENTITY = "Apple Development";
-				"CODE_SIGN_IDENTITY[sdk=macosx*]" = "Apple Development";
-				CODE_SIGN_STYLE = Automatic;
-				DEAD_CODE_STRIPPING = YES;
-				DEVELOPMENT_TEAM = 76TVFCUKK7;
-				GENERATE_INFOPLIST_FILE = YES;
-				IPHONEOS_DEPLOYMENT_TARGET = 18.0;
-				MACOSX_DEPLOYMENT_TARGET = 15.0;
-				MARKETING_VERSION = 1.0;
-				PRODUCT_BUNDLE_IDENTIFIER = com.livekit.VoiceAgentTests;
-				PRODUCT_NAME = "$(TARGET_NAME)";
-				PROVISIONING_PROFILE_SPECIFIER = "";
-				SDKROOT = auto;
-				SUPPORTED_PLATFORMS = "iphoneos iphonesimulator macosx xros xrsimulator";
-				SWIFT_EMIT_LOC_STRINGS = NO;
-				SWIFT_VERSION = 5.0;
-				TARGETED_DEVICE_FAMILY = "1,2,7";
-				TEST_HOST = "$(BUILT_PRODUCTS_DIR)/VoiceAgent.app/$(BUNDLE_EXECUTABLE_FOLDER_PATH)/VoiceAgent";
-				XROS_DEPLOYMENT_TARGET = 2.0;
-			};
-			name = Release;
-		};
 		B5B5E3BF2D124AE20099C9BE /* Debug */ = {
 			isa = XCBuildConfiguration;
 			buildSettings = {
@@ -689,15 +564,6 @@
 			defaultConfigurationIsVisible = 0;
 			defaultConfigurationName = Release;
 		};
-		ACC2802F2DEDDA1D0023C137 /* Build configuration list for PBXNativeTarget "VoiceAgentTests" */ = {
-			isa = XCConfigurationList;
-			buildConfigurations = (
-				ACC2802D2DEDDA1D0023C137 /* Debug */,
-				ACC2802E2DEDDA1D0023C137 /* Release */,
-			);
-			defaultConfigurationIsVisible = 0;
-			defaultConfigurationName = Release;
-		};
 		B5B5E3AD2D124AE00099C9BE /* Build configuration list for PBXProject "VoiceAgent" */ = {
 			isa = XCConfigurationList;
 			buildConfigurations = (
@@ -719,28 +585,20 @@
 /* End XCConfigurationList section */
 
 /* Begin XCRemoteSwiftPackageReference section */
-		ACFBA1D92D8D5CBE0021202B /* XCRemoteSwiftPackageReference "swift-collections" */ = {
-			isa = XCRemoteSwiftPackageReference;
-			repositoryURL = "https://github.com/apple/swift-collections";
-			requirement = {
-				kind = upToNextMajorVersion;
-				minimumVersion = 1.1.4;
-			};
-		};
 		B5E1B90D2D14E9EC00A38CB6 /* XCRemoteSwiftPackageReference "components-swift" */ = {
 			isa = XCRemoteSwiftPackageReference;
 			repositoryURL = "https://github.com/livekit/components-swift";
 			requirement = {
-				kind = upToNextMajorVersion;
-				minimumVersion = 0.1.5;
+				branch = "blaze/agent-conversation";
+				kind = branch;
 			};
 		};
 		B5E1B9102D14E9F500A38CB6 /* XCRemoteSwiftPackageReference "client-sdk-swift" */ = {
 			isa = XCRemoteSwiftPackageReference;
 			repositoryURL = "https://github.com/livekit/client-sdk-swift";
 			requirement = {
-				kind = upToNextMajorVersion;
-				minimumVersion = 2.7.1;
+				branch = "blaze/agent-conversation";
+				kind = branch;
 			};
 		};
 /* End XCRemoteSwiftPackageReference section */
@@ -750,11 +608,6 @@
 			isa = XCSwiftPackageProductDependency;
 			productName = LiveKit;
 		};
-		ACFBA1DA2D8D5CBE0021202B /* Collections */ = {
-			isa = XCSwiftPackageProductDependency;
-			package = ACFBA1D92D8D5CBE0021202B /* XCRemoteSwiftPackageReference "swift-collections" */;
-			productName = Collections;
-		};
 		B5E1B90E2D14E9EC00A38CB6 /* LiveKitComponents */ = {
 			isa = XCSwiftPackageProductDependency;
 			package = B5E1B90D2D14E9EC00A38CB6 /* XCRemoteSwiftPackageReference "components-swift" */;