The Challenge: Scaling AI Beyond Proof of Concept

Key points:

• As Vice-President of a new start-up, Andrea transformed an untested proof of concept – a conversational agent to respond to frequently asked airline fight questions – to actuality and implementation.

• The project encountered initial resistance and a lack of trust to the use of a conversational agent from customer service personnel but won commitment through collaborative engagement and two-way feedback drawing from front line staff insights which built trust and enhanced system capability.

• The success of the program was conclusively demonstrated at the height of the pandemic when a European carrier adopted the system. The system demonstrated its ability to handle complex inquiries efficiently and effectively, aiding both passengers and carrier staff during a time of heightened pressure.

A Dublin-based technology firm embarked on a venture to modernise airline customer service through machine learning. As newly appointed Vice President, Andrea inherited a basic proof of concept that showed promise but required substantial development to meet the complex demands of an operational environment. The transformation from prototype to commercial SaaS platform would test the boundaries of both technical innovation and organisational change management. The basic idea was a conversational agent that could handle an airlines most frequently asked questions, for example: “Do you allow pets onboard?”.

Data: The Foundation of AI Excellence

The initial prototype operated on a limited dataset of standardised airline FAQ documentation. While functional, this constraint became apparent as the team worked toward production readiness as their was insufficient data to train the language model effectively. The initial PoC had been too narrow, and it was apparent that make it into an commercially viable operational product would require a LOT more data to train the model on. The team initiated a comprehensive data gathering strategy, incorporating diverse sources including customer service transcripts, social media interactions, detailed airline policy documentation, and crowd-sourced answers. This enhanced dataset proved essential for training the language model to handle the nuanced nature of real-world passenger inquiries.

End-to-End Process Design: Ensuring Production Success

Recognising that successful AI deployment requires more than just technical considerations, the team conducted extensive workshops focused on end-to-end digital process design. These sessions mapped out the flow of data through the production environment, identifying potential bottlenecks and integration points. Stakeholders from IT, operations, legal, and customer service collaborated to ensure the system would seamlessly integrate with existing airline infrastructure.

The Dress Rehearsal: Testing in a Live Environment

A new innovation in the deployment strategy was the implementation of full "dress rehearsals.", introduced by Andrea. These comprehensive testing sessions involved stakeholders from the organisation, airline representatives, and external vendors. Participants assumed various roles, from customers to service agents, conducting thorough quality assurance testing under realistic conditions. These rehearsals proved invaluable in identifying operational challenges and refining system responses before live deployment.

Stakeholder Integration: Building Trust and Expertise

The integration of airline staff presented unique challenges, particularly initial training requirements with customer service personnel, who needed to be trained on using and updating the large language model. However, through structured training programs and collaborative feedback sessions, the team successfully transformed this initial reluctance into active engagement. Regular workshops and feedback loops ensured that front-line staff insights were continuously incorporated into system refinements, building both trust and system capability.

Production Deployment: A Milestone Achievement

The program reached a significant milestone in 2020 when a European carrier adopted the system during the heightened pressures of the pandemic. The deployment followed a carefully measured approach, beginning with specific query categories and gradually expanding based on performance data. This methodical implementation enabled careful monitoring of the language model and systematic optimisation of responses. The careful attention to preparation and stakeholder engagement paid dividends in production. Q&A response accuracy metrics showed steady improvement. The system demonstrated its ability to handle complex inquiries effectively, validating the extensive development and testing process.

Early experiments with language models

The timing of this implementation proved particularly significant, preceding the widespread adoption of large language models in commercial applications. The project team successfully addressed complex technical challenges, secured comprehensive stakeholder support, and demonstrated the practical application of conversational AI technology in a demanding customer service environment.

The success of this project highlighted the importance of thorough preparation, stakeholder engagement, thorough legal and regulatory due diligence, and systematic, cross-company testing in AI deployment.

The innovative use of dress rehearsals and comprehensive process design workshops set new standards for AI implementation in the company. This pioneering work continues to inform Advise-AI’s approach to AI transformation across industries, demonstrating the critical balance of technical excellence and organisational change management necessary for successful AI.

Situation: A premium Asian airline faced the complex challenge of modernising its retail channels to serve both its five million loyalty programme members and global customer base through digital platforms. The transformation would fundamentally reshape how customers accessed and purchased flights and travel packages via web and mobile channels.

Result: Under Andrea's leadership as Manager of Channel and Digital Transformation, the airline embarked on an ambitious five-year programme to update its decades-old physical retail presence. Her extensive experience in channel distribution and digital transformation proved fundamental in navigating the complexities of this large-scale initiative.

Collaboration

The preliminary phase involved comprehensive stakeholder engagement across corporate executives, partners, vendors and global consultants. This collaborative approach enabled the team to construct a detailed vision of the retail travel journey, whilst carefully considering customer requirements, potential impediments, regulatory constraints and emerging digital trends in the travel sector.

Agile framework

Working closely with senior stakeholders across the airline, Andrea established an agile funding framework and developed a comprehensive iterative business case to support the 5 year program of work.  She implemented governance structures and maintained regular board-level communications throughout the programme. Her influencing skills and methodical approach secured the necessary support for a full-scale transformation of the airline's digital channel strategy.

Digital forum to manage complex rollout

To manage the programme's considerable scope, involving over 100 team members, multiple vendors and consulting houses, and 16 business units, and covering a meticulous port-by-port deployment across 23 countries, Andrea established the "Digital Retail Forum," a new portfolio management function. This cross-functional initiative required business unit heads to demonstrate return on investment for proposed digital services, enabling data-driven prioritisation of development efforts against available resources.

The transformation delivered significant capability enhancements, notably the introduction of dynamic product bundling, allowing customers to create personalised travel packages incorporating flights and accommodation. This required substantial restructuring of external vendor data sources and necessitated reorganisation of several key business functions, including Sales, IT, and global call centres to support the new digital operating model.

Differences between today’s Generative AI transformation

The programme's implementation approach remains particularly relevant to contemporary Generative AI transformation initiatives. The approach demonstrated the importance of careful use-case selection, considered stakeholder engagement, ROI assessment, process modification and data quality management—all crucial elements in current AI tech transformation programmes.

The experience gained continues to inform Andrea’s approaches to transformation across various sectors, including Generative AI implementations

Situation: An Australian print directories company faced a critical juncture as the digital age disrupted traditional industries. As print revenue declined, the company needed to adapt. Andrea, a senior digital leader, helped guide the company's transformation from a print to digital platform, with bundled product offerings.

Result : Navigating a complex legacy transition was a significant challenge for the 90-year-old phone directory company. This involved Andrea and her team collaborating with multiple departments, vendors, and government institutions, gathering user and business insights, addressing regulatory considerations, and developing new digital products and services

Change management

This transformation required a fundamental shift in the company's operations. The organisation needed to adapt its sales strategies, retrain its workforce, and realign its budget and organisational structure to support its digital future.

Despite initial resistance, and out of market necessity, the organisation eventually transitioned to a digital platform. This transformation demonstrates the importance of pragmatic leadership, stakeholder engagement, and a focus on customer needs in navigating technological disruption.

Parallels between Generative AI today

The parallels between the directory company’s digital transformation and the current wave of Generative AI adoption are notable. Both scenarios present organisations with a critical juncture, demanding a fundamental re-evaluation of their core operations and a proactive response to disruptive technological forces.

Just as the directory company faced the erosion of print media, organisations today confront the transformative potential of Generative AI, requiring some to adapt and evolve to remain competitive. This necessitates a high degree of agility and a willingness to embrace change, mirroring the adaptability required by the print company during its own digital transformation.

Furthermore, both transformations underscore the critical importance of stakeholder collaboration. The directory company’s success hinged on close collaboration across departments, gathering insights from users and businesses to inform its digital strategy. Similarly, successful AI implementation demands close collaboration between business units, IT teams, data teams,  legal and regulatory, customers and consumers.

Both scenarios also emphasise the paramount importance of understanding and addressing organisational needs. The directories company prioritised user feedback in its digital transformation, and likewise, organisations must prioritise their use-cases appropriately so as to deliver tangible value when implementing AI solutions.