Emerging-Risk Workshop: From AI Proposal to an Accountable Decision

An artificial intelligence proposal becomes manageable when the organization defines the business use, data, model behavior, suppliers, decisions, obligations, and consequences before adoption. This workshop follows an AI initiative through stakeholder objectives, risk assessmen…

In this episode, we are applying the book's emerging-risk process to a proposed artificial-intelligence service. The lesson is not that every AI proposal is dangerous or that a security manager must become a model engineer. It is about translating an unfamiliar technology into a decision involving business purpose, information flows, suppliers, uncertainty, control responsibilities, evidence, and proper authority.

Imagine a fictional company called Lakeside Support Services. It provides customer support for several manufacturers through telephone, email, and an online ticket portal. The business wants to add an AI feature that summarizes customer conversations, proposes draft responses, and identifies tickets that may require escalation. A third-party provider will host the model and process ticket content. The project team expects faster response times, but the proposal is incomplete about data retention, model training, human review, supplier access, error handling, and how the feature can be disabled.

The first management step is to state the intended business use precisely. Saying that Lakeside wants to use AI is too broad to support risk assessment. The proposed service will receive customer ticket text, create summaries, suggest responses, and assign an escalation indication for human review. It is not authorized to make binding customer commitments, change account records, or close tickets automatically. These boundaries matter because risk depends on what the system is allowed to do, what information it receives, and how people rely on its output.

Business objectives should be made visible. Lakeside wants to reduce time spent reading long ticket histories, help new agents find relevant context, and identify urgent cases earlier. Those outcomes can be measured through response quality, handling time, escalation accuracy, and customer impact. The security assessment should not become a generic debate about innovation. It should ask whether the proposal can support those objectives within the organization's risk appetite and obligations. Clear objectives also make it possible to evaluate whether the feature delivers enough value to justify its controls and dependencies.

The next step is to map information flows. Customer text enters the ticket platform, selected fields are sent to the external provider, the model generates output, and the output returns to the agent interface. Logs, prompts, responses, feedback, and administrative data may also be stored by one or more parties. The assessment should identify where information originates, where it travels, who can access it, how long it remains, and what downstream services support processing. An architecture view turns an abstract technology label into specific boundaries and control points.

Data classification guides those questions. Tickets may contain customer contact information, product details, contractual information, employee names, troubleshooting records, or other sensitive content. Some customers may submit information that Lakeside did not request. The organization should determine which categories may be processed, which must be removed, and whether the supplier can support the required handling. A policy that says confidential data must be protected is not enough. The project needs technical and procedural controls that apply that requirement to prompts, outputs, storage, and support access.

Data minimization can reduce exposure before complex controls are considered. Lakeside may not need to send full ticket histories, customer identifiers, or attachments to generate a useful summary. It may use selected text, masking, or field-level restrictions. The project should confirm that minimization does not remove context necessary for safe output. The goal is not to send as little information as possible without regard to function. It is to send only what the defined use requires and to understand why each element is needed.

The organization should distinguish provider processing from provider training or service improvement. The proposal needs clear answers about whether customer content is retained, used to improve shared models, reviewed by supplier personnel, or transferred to downstream providers. Contract language, technical settings, and operational evidence should align. A supplier statement that data is protected may not answer how it is used. Security leaders should ask for decision-relevant evidence rather than accept broad marketing assurances.

Third-party and fourth-party dependencies must be identified. The AI provider may rely on cloud hosting, monitoring, support, data-storage, or model services that are not directly contracted by Lakeside. The organization may not have direct control over every downstream party, but it remains accountable for understanding material dependencies and contractual boundaries. The assessment should determine which providers can access customer information, which incidents they must report, and what evidence Lakeside can obtain. Lack of visibility is itself a risk input, not a reason to pretend the dependency does not exist.

Legal, regulatory, and contractual requirements should be translated into project decisions with qualified internal support. The security manager should not make universal legal conclusions about every customer or jurisdiction. Instead, the team should identify the categories of information involved, customer commitments, data-location expectations, notification duties, retention requirements, and restrictions on automated decisions. Legal and privacy functions can determine which obligations apply. Security then helps design evidence and controls that make compliance operational.

The proposal also creates confidentiality risk. Sensitive ticket content could be exposed through supplier access, insecure integration, excessive logging, misconfiguration, or inappropriate reuse. The assessment should identify credible pathways rather than listing every imaginable threat. Controls may include strong authentication, least privilege, encryption, restricted logging, data minimization, supplier access controls, and monitoring. The risk decision should explain which pathways are addressed and what exposure remains.

Integrity risk is equally important. The model may produce inaccurate summaries, omit critical context, invent details, or suggest an inappropriate response. The business should decide how much reliance is permitted. A draft reviewed by a trained agent creates a different risk than an automated customer message. Human review is not a complete control if reviewers are rushed, untrained, or encouraged to accept suggestions without challenge. The control design should explain what the person must verify and how errors are detected and corrected.

Availability and resilience also matter. Agents may become dependent on the feature, and an outage at the supplier could slow operations or block access to ticket context. Lakeside should decide whether the core support process can continue without the AI service. A graceful fallback might return agents to the original ticket interface. The project should test disablement and degraded operation rather than assume they will work. A new feature should not silently become a single point of failure for a service that previously operated without it.

The organization should consider the risk of inappropriate automation. A suggestion tool may gradually become a decision tool if agents treat outputs as authoritative or if management pressures them to follow the recommendation. Scope drift can occur through configuration changes, new integrations, or informal practice. Governance should define prohibited uses, approval requirements for expanded capability, and monitoring for changes in actual behavior. The authorized use should remain visible after launch.

Transparency and accountability need practical definitions. Lakeside may not understand every internal model mechanism, but it can still require enough information to manage the service. It should know the purpose, inputs, outputs, owners, provider responsibilities, quality limits, decision boundaries, and escalation process. When a harmful output occurs, the organization needs a way to investigate what information was sent, what response was returned, and who acted on it. Accountability does not require perfect explainability, but it does require a traceable operating process.

Traditional risk models may have limited historical data for the exact use case. That does not prevent assessment. The team can use qualitative analysis, scenario reasoning, supplier evidence, pilot results, analogous incidents, and expert input. It should document uncertainty instead of disguising it with precise-looking numbers. The decision-maker needs to know which assumptions have strong evidence, which are provisional, and which could change the recommendation. Honest uncertainty supports bounded action.

A useful risk statement connects a plausible event to business consequence. For example, sensitive customer content may be retained or accessed outside approved conditions, leading to contractual breach and loss of trust. An inaccurate escalation recommendation may delay response to a safety-related ticket, affecting customers and operations. A provider outage may remove a capability on which agents have become dependent, increasing backlog. These statements are more actionable than saying that AI is risky.

Risk ownership should follow the business service, not be assigned automatically to the security team. The support-service owner is accountable for how the feature affects customer operations and for accepting residual business risk within authority. The project and technology teams own implementation tasks. Control owners maintain specific safeguards. Supplier-management functions oversee external obligations. Security coordinates assessment and provides challenge. Clear ownership prevents the novelty of the technology from becoming an excuse for vague responsibility.

The project should evaluate alternatives. It could proceed with the full feature, restrict the initial use, redesign data flows, conduct a bounded pilot, select a different provider, or defer the proposal. The strongest option is not necessarily the most restrictive. It is the one that supports the business objective while keeping risk within approved limits and generating enough evidence for the next decision. Comparing alternatives makes tradeoffs visible and reduces pressure to choose between unconditional approval and complete rejection.

A bounded pilot is often useful when uncertainty can be controlled. Lakeside could use a limited set of low-sensitivity tickets, a small group of trained agents, restricted functions, and short data-retention settings. It could prohibit automated sending, customer-account changes, and use of attachments. The pilot should have a defined duration, owner, success criteria, stop conditions, and review date. A pilot is a risk treatment and evidence-generation activity, not permission to operate indefinitely without governance.

Pilot success criteria should include business and control outcomes. Business measures may include summary usefulness, handling time, escalation quality, and agent workload. Security and governance measures may include prohibited-data detection, supplier access visibility, error reporting, incident cooperation, retention confirmation, and disablement performance. If only productivity is measured, the pilot can appear successful while risk remains unexamined. A balanced evaluation supports an accountable decision about expansion.

Stop conditions should be specific enough to act on. Examples include confirmed processing of prohibited data, repeated material inaccuracies, loss of required logging, supplier inability to explain an incident, unauthorized scope expansion, or failure of the disablement process. The organization should identify who can pause the pilot and how quickly that action can occur. Emergency authority should be predefined. A stop condition without an owner and mechanism is merely a warning label.

Human review must be designed as a control. Agents need instruction on what the model can and cannot do, which outputs require special scrutiny, and how to report errors. The interface should make suggestions distinguishable from verified facts. Workload targets should not punish agents for taking time to review. Sampling and quality assurance can determine whether human review actually catches problems. The organization should measure behavior, not assume that placing a person in the process guarantees safety.

Access management should cover administrators, developers, supplier personnel, support staff, and service identities. Privileged functions should be limited and monitored. The project should define who can change prompts, models, data connections, retention settings, and output behavior. Changes that alter risk should go through appropriate review. A well-designed user control can be undermined by weak administrative governance.

Logging should support the decisions the organization may need to make. Lakeside may need records of data sent, outputs received, administrative changes, access events, errors, and disablement actions, subject to data-minimization and retention requirements. Logging everything without purpose can create new exposure. The team should identify the events needed for monitoring, investigation, quality review, and accountability. Log access and retention should be governed like the underlying data.

Supplier assurance evidence should be evaluated for relevance, scope, period, and limitations. A certification or audit report may provide useful information about general controls but not about Lakeside's configuration, data use, or incident cooperation. Penetration-test summaries, architecture descriptions, contractual commitments, and operational demonstrations may add context. The organization should avoid treating one artifact as proof of the complete service. Assurance is strongest when several relevant sources support the specific decision.

Change management should recognize that the service can evolve quickly. The provider may update models, features, interfaces, data practices, or downstream dependencies. Lakeside should define which changes require notice, reassessment, testing, or approval. A material change may invalidate pilot evidence. Ongoing monitoring should include service changes, not only security incidents. The risk decision applies to the approved configuration and use, not to every future version automatically.

Exit planning belongs in the initial decision. Lakeside should know how to disable the integration, retrieve or delete retained data, remove supplier access, preserve necessary records, and continue customer support without the feature. It should understand whether outputs or configurations are portable and whether downstream providers retain copies. A service that is easy to start but difficult to stop can create strategic dependency. Exit evidence is part of supplier governance.

The information security architecture should translate the risk decision into requirements and control relationships. It can show the permitted data path, trust boundaries, identity controls, logging, human-review point, fallback route, and supplier interface. The architecture is not merely a diagram. It provides a basis for implementation, testing, exceptions, and future change. When a requirement cannot be met, the deviation should be visible and governed.

Control testing should examine design and operation. A policy may require prohibited data to be removed, but the team must verify whether the integration actually prevents or detects it. A contract may require deletion, but Lakeside needs evidence that deletion settings and processes operate as expected. A disablement procedure may exist, but the pilot should demonstrate that it works without disrupting the core ticket service. Testing converts planned safeguards into observed evidence.

The risk register should capture material uncertainty and treatment, not every technical detail. It should identify the business use, key scenarios, affected objectives, owner, selected treatment, conditions, indicators, and review triggers. Technical findings can be linked through supporting records. The register should be updated when pilot evidence changes likelihood, impact, or control confidence. A risk record is useful when it supports an ongoing decision, not when it becomes a static description of technology.

Communication with leadership should avoid hype in both directions. The proposal is not automatically transformational, and it is not automatically unacceptable. The security manager can explain the business purpose, information flows, credible risks, evidence limits, proposed pilot conditions, and decision required. This framing helps leaders compare the initiative with other investments and understand what they are authorizing. Fear and enthusiasm should not replace accountable analysis.

The decision may be to proceed with conditions. Lakeside could approve a limited pilot using low-sensitivity ticket categories, trained agents, no automated customer messages, restricted retention, verified logging, and a tested disablement process. Expansion could require evidence that quality and control thresholds were met, supplier responsibilities remained clear, and no material incidents occurred. This approach preserves innovation while keeping uncertainty bounded and visible.

The decision may instead be to redesign. If the provider cannot prevent shared training, cannot support required deletion, or cannot provide sufficient incident cooperation, the data flow may need to change. Lakeside might remove sensitive fields, use a different deployment model, or select another supplier. Redesign is not rejection of the business objective. It is an effort to achieve the objective through a more acceptable architecture.

Deferral may be appropriate when the organization cannot establish basic ownership, data boundaries, or control evidence. A rushed pilot without a disablement path, supplier obligations, or human-review design could create exposure that the business is not prepared to manage. The recommendation should explain what evidence or capability is needed before reconsideration. Deferral becomes constructive when it is connected to clear conditions rather than an indefinite statement that more study is required.

On the exam, emerging technology is often a context for familiar management responsibilities. Identify the business objective, assets, data, dependencies, threats, control gaps, owners, and authority. Do not assume the newest technology requires abandoning governance principles. Also do not force an old control onto a new architecture without checking fit. The strongest answer usually makes the risk actionable and integrates the decision into normal governance and program processes.

Watch for choices that respond to external headlines without connecting them to internal exposure. A broad threat report may justify investigation, but it does not prove Lakeside faces a specific material risk. The manager should determine whether the organization uses the technology, what pathways exist, and what objectives could be affected. Conversely, lack of a public incident does not prove the proposal is safe. Relevance comes from the organization's actual use and dependencies.

For a learner exercise, draw the Lakeside information flow in words. Identify the customer, ticket system, integration, provider, downstream services, agent interface, logs, and administrative path. For each boundary, state the information involved, responsible owner, required control, and evidence. Then identify one uncertainty that could change the recommendation. This exercise turns a fashionable label into a manageable service architecture.

For a second exercise, write pilot conditions. Include scope, prohibited uses, data restrictions, human review, supplier obligations, monitoring, success measures, stop conditions, duration, owner, and expansion authority. Then imagine that the provider changes its retention practice midway through the pilot. Decide whether the change requires pause, redesign, or additional evidence. This trains you to treat approval as conditional on the facts that supported it.

The project should plan for incident response before the pilot begins. The organization needs a way to recognize unauthorized data use, harmful output patterns, supplier compromise, or unexpected service behavior. It should define how agents report concerns, how logs are preserved, who contacts the provider, and who can disable the integration. Incident planning is stronger when it is based on the actual service design rather than added after launch.

Business continuity planning should also consider dependence on the feature. During a pilot, agents may still be comfortable working without it. After broader adoption, procedures and staffing may assume the summaries and suggestions are always available. Lakeside should test fallback operation and monitor whether skill or capacity is eroding. A productivity tool can become a resilience risk when the organization no longer knows how to perform the underlying work without it.

The organization should include customer and employee perspectives where appropriate. A technically acceptable design may still create trust concerns if people do not understand how their information is used or if agents cannot explain a decision influenced by the service. Communication should be accurate and aligned with actual practice. Promising that information is never retained or that a human always decides is dangerous unless the architecture, contract, and operating process support that claim.

Training should extend beyond the immediate users. Support supervisors need to recognize quality and escalation problems. Developers and administrators need to understand approved data paths and change restrictions. Procurement and legal teams need to know which supplier changes trigger review. Executives need to understand what the pilot can and cannot prove. Shared understanding reduces the chance that the feature expands through informal decisions outside the approved boundary. It also gives each function a clear reason to challenge a change, because the concern can be tied to an agreed requirement rather than to personal discomfort with the technology. That discipline keeps review focused on accountable outcomes, evidence, and scope. It also makes later reassessment easier because the original decision basis remains visible.

Finally, the risk assessment should identify what would make the service strategically unacceptable even if short-term results look good. Examples might include inability to exit the provider, loss of control over sensitive information, persistent material error, or dependence on an opaque supplier chain. These are not predictions that failure will occur. They are decision thresholds that protect the organization from allowing convenience to create an ungovernable dependency.

Lakeside Support Services should not approve the proposal merely because productivity benefits appear attractive, and it should not reject it merely because the technology is unfamiliar. It should define the use, map information flows, identify obligations and dependencies, assess credible scenarios, and choose a bounded path that generates evidence. A controlled pilot may be appropriate if the organization can limit data, preserve human authority, monitor the service, and stop it when conditions fail.

Emerging-risk management is a recurring process. New information, provider changes, business expansion, incidents, and pilot results can alter the decision. The organization should update the risk record, architecture, controls, and authority as the service evolves. What begins as a narrow suggestion feature may become a critical operational dependency if governance does not keep pace. Continuous review protects both innovation and accountability.

The central lesson is that unfamiliar technology does not remove the need for disciplined management. State the business purpose, follow the information, identify who is accountable, distinguish evidence from assumption, compare treatment choices, and authorize only the scope the organization can govern. That approach makes emerging risk actionable without pretending uncertainty can be eliminated. It also keeps the security conversation focused on decisions the business can understand and own.

The AI proposal should therefore leave the workshop with explicit conditions for data use, human oversight, supplier evidence, testing, monitoring, and reassessment. A decision that names these conditions is more useful than a broad approval or rejection because it connects innovation to accountable operation and creates observable reasons to continue, restrict, or stop.

Emerging-Risk Workshop: From AI Proposal to an Accountable Decision
Broadcast by