Share
Explore
AI Co-Pilot Evaluation Framework
Implementation, Customization, Security & Ecosystem
Product Assessment - Part 2
Core Focus Area
Sub-Focus Area
Question
Example Answer
Cost Impact
Level of Capability
Initial Setup & Integration
7
Deployment Architecture
Does the co-pilot have a streamlined model for the agent to communicate with the model server through the customer firewall efficiently and securely?
Eg. The co-pilot software leverages an API gateway to streamline traffic through a trusted FQDN & IP range from the browser client, through the customer firewall which can be easily allow listed.
Security & Compliance Costs
Performance & Monitoring Costs
Integration Costs
Infrastructure Costs
Local Installation & Distribution
Does the co-pilot software have an efficient mechanism of distributing / installing the co-pilot on end user browsers / work stations?
Eg. The co-pilot software is integrated into Chrome as a plug-in that can be distributed via a central web app and installed on individual user machines.
Integration Costs
Infrastructure Costs
Deployment & Testing Costs
Performance & Monitoring Costs
Integration with Customer Business Systems
Does the co-pilot facilitate easy integration with existing customer databases and subsequent data transformations for both API and non-API driven systems?
Eg. The co-pilot has a connector workflow to drag and drop configurable integrations from customer systems into a workflow based creator wherein a user can define data points that need to be referenced, along with transformations needed for RAG.
Data Acquisition, Storage and Processing Costs
Integration Costs
Infrastructure Costs
Interoperability
Depending on the deployment architecture, Does the co-pilot support multiple browsers and operating systems?
Eg. The co-pilot supports browsers such as Firefox and Microsoft Edge, beyond Google Chrome. OR The co-pilot agent can run on Windows XP or higher, as well as on Mac.
Enablement & Training Costs
Integration Costs
Private Cloud / On-Premise / Hybrid Support
Does the co-pilot offer secure private cloud deployments, or hybrid deployments to interface with on premise / privately hosted source data?
Eg. The co-pilot offers on premise connectors to privately hosted DB instances, as well as systems such as SAP Hana.
Data Acquisition, Storage and Processing Costs
Data Labelling Costs
Model Development Costs
Deployment & Testing Costs
Infrastructure Costs
Integration Costs
User Management
Does the co-pilot offer a secure user management mechanism that can integrate with customer’s identity systems via SSO?
Eg. The co-pilot software offers integration with Okta, PingFed and AFS for Single-Sign-On, and also offers a manual user management mechanism.
Security & Compliance Costs
Integration Costs
Licensing
Does the co-pilot offer a flexible licensing pool, wherein licensing can be switched between multiple users?
Eg. The co-pilot offers a floating licensing model that allows admins to switch licenses between systems, users and workstations.
Enablement & Training Costs
Licensing Costs
Configuration & Change Management
5
Input & Output Formats
Does the co-pilot provide an easy way to tweak input and output formats?
Eg. A user can tweak the input format to accomodate for questions and documents that an end customer can upload. From an output standpoint, a user can define whether the co-pilot generates a JSON, XML or CSV output.
Enablement & Training Costs
Integration Costs
Additional Configuration Costs
Fine Tuning via human-in-the-loop
Does the co-pilot provide mechanisms for users to fine tune underlying models with their own data, through ad-hoc approaches as well as via human in the loop?
Eg. The document understanding co-pilot provides users the capability to upload any new invoices received in the last month. Additionally, the invoice processing review step occasionally presents values that need human review along with 3 possible outcomes, for the user to select from, or manually enter within NetSuite.
Model Development Costs
Data Labelling Costs
Deployment & Testing Costs
Enablement & Training Costs
Infrastructure Costs
Data Acquisition, Storage and Processing Costs
Additional Configuration Costs
Brand, Tonality & Moderation
Does the co-pilot provide the user with the capability to fine tune the visual and textual branding and tonality of the output? Does it provide the user the capability to do custom moderation?
Eg. The co-pilot software allows customers to upload their proprietary brand assets as building blocks or examples. The co-pilot responses can be assigned a specific tonality based on the use cases, and can also account for specific cultural considerations such as addressing employees as “team-members”
Enablement & Training Costs
Security & Compliance Costs
Additional Configuration Costs
Scalability
How do the infrastructure costs increase with a higher number of transactions, types & scope of transactions, as well as users, as compared to the value created?
Eg. Infrastructure costs increase at a slower rate than the value creation from processing the marginal transaction, due to the user of various optimization methods used, thus increasing value as the software scales.
Infrastructure Costs
Deployment & Testing Costs
Performance & Monitoring Costs
Monitoring & Management
Does the co-pilot provide a central monitoring platform / dashboard to manage workflows, track usage, and observe co-pilot activity?
Eg. Yes - The co-pilot software offers a central dashboard to activate, deactivate and edit co-pilot workflows, commission or commission users and monitor co-pilot activity.
Performance & Monitoring Costs
Security & Compliance Costs
Security and Complaince
7
Securing Model Parameters
Does the model have controls in place to secure its hyper parameters?
Eg. The AI co-pilot uses standard encryption protocols and has built in access controls to prohibit unauthorized access to parameters. Additionally, it uses zero-knowledge proof techniques to allow certain computations to be performed on the model's parameters without revealing the parameters.
Security & Compliance Costs
Performance & Monitoring Costs
Training on Sensitive Data
Does the model use techniques to minimize exposure of training on sensitive data?
Eg. The AI co-pillot uses federated learning, where the model is trained collaboratively without sharing sensitive data, reducing the exposure of confidential information.
Data Labelling Costs
Data Acquisition, Storage and Processing Costs
Security & Compliance Costs
Infrastructure Costs
Input & Output Validation & Sanitization
Does the co-pilot use techniques to filter model input and outputs to avoid malicious inputs, harmful content, injection attacks etc?
Eg. Input data to & output from the co-pilot is validated and sanitized to prevent malicious inputs via mechanisms such as data validation, sanitization, explicit whitelisting, escaping special characters, content moderation, custom filters and human review.
Security & Compliance Costs
Performance & Monitoring Costs
Sensitive Information Storage & Retention
If the co-pilot collects PII or PFI, does it securely store and retain data for a finite period? Do customers have the capability to forget their data or tweak retention periods?
Eg.The co-pilot uses techniques such as identifying and masking PII / PFI for model fine tuning and training. Any PII and PFI stored is retained for a year with clear opt out and tweak-able retention periods by customer.
Security & Compliance Costs
Performance & Monitoring Costs
Data Security
Does the co-pilot use industry standard protocols to secure data in motion / transit and data ar rest?
Eg. The AI co-pilot leverages TLS 1.2 / HTTPS protocol to encrypt data in motion and AES 256 to encrypt data at rest.
Security & Compliance Costs
Performance & Monitoring Costs
Industry Standard Compliance
Does the co-pilot software follow standard industry standard protocols such as SOC 2 and GDPR?
Eg. Yes, the co-pilot software is SOC 2 Type 2, GDPR and CCPA compliant.
Security & Compliance Costs
Performance & Monitoring Costs
Transparency and Accountability
Does the co-pilot software have the capability to expose an audit trail of decision making including events such as the identified customer intent, systems referenced, citations etc?
Eg. Yes, the support co-pilot can present an audit trail of the acknowledged customer request, classified intent, cite the knowledge base referenced and highlight the recommendation made.
Security & Compliance Costs
Performance & Monitoring Costs
Ecosystem & Network Effects
3
Model Ecosystem
Does the co-pilot offer a central marketplace to share either fine tuned models or the parameters of in-house trained model for the broader community to use?
Eg. The accounts payable co-pilot leverages federated learning for customers to share model weights from respective fine tuned model on custom invoices without sharing any vendor or company information.
Model Development Costs
Deployment & Testing Costs
Data Labelling Costs
Data Acquisition, Storage and Processing Costs
Integration Costs
Data Marketplace
Does the co-pilot offer a central marketplace for customers or partners to share non-sensitive data?
Eg. The expense management co-pilot provides the capability for customers to share any receipts uploaded by employees which do not contain non-sensitive information.
Model Development Costs
Deployment & Testing Costs
Data Labelling Costs
Data Acquisition, Storage and Processing Costs
Integration Costs
Integration Marketplace
Does the co-pilot offer the capability for customers and partners to share OOTB integrations to third party applications?
Eg. The UiPath marketplace offers the capability to upload plug & play packages for customers and partners to share their home grown integrations via the UIPath SDK. Customers can also share automation templates for the community to broadly use.
Enablement & Training Costs
Deployment & Testing Costs
Infrastructure Costs
Type of Costs
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.