A Continuously Learning AI Cognitive Assistant

Articles, News

Learning to operate a complex system, such as an agile production line, can be a daunting task. The high variability in products and frequent reconfigurations make it difficult to keep documentation up-to-date and share new knowledge amongst factory workers. We introduce CLAICA, a Continuously Learning AI Cognitive
that supports workers in the aforementioned scenario. CLAICA learns from (experienced) workers, formalizes new knowledge, stores it in a knowledge base, along with contextual information, and shares it when relevant. We conducted a user study with 83 participants who performed eight knowledge exchange tasks with CLAICA, completed a survey, and provided qualitative feedback. Our results provide a deeper understanding of how prior training, context expertise, and interaction modality affect the user experience of cognitive assistants. We draw on our results to elicit design and evaluation guidelines for cognitive assistants that support knowledge exchange in fast-paced and demanding environments, such as an agile production line.

Human-centered manufacturing

A key paradigm for modern production is the broad adoption of digital tools to increase, for example, productivity and sustainability. Thoben et al. concretized this paradigm with research challenges and application examples for Industry 4.0 and smart manufacturing (SM). The latter are two initiatives to systematically detail and implement this paradigm in manufacturing. While Industry 4.0 originated in Germany, SM emerged in the United States. Countries such as Japan, Korea, and China created similar initiatives. These initiatives are largely technology-focused and do not adequately account for human needs, involvement, and collaboration with AI systems .

The next paradigm addresses this shortcoming through human-centric manufacturing, called Industry 5.0 . Müller reports the findings of an expert group regarding technologies contributing to more human-centric manufacturing. They include, among others, individualized human-machine interaction technologies accounting for the strengths of humans and machines. Also, using AI to assist humans in understanding causalities in complex and dynamic systems is a critical technology. Romero et al. introduced similar ideas stressing the importance of socially sustainable manufacturing. They proposed the vision of Operator 4.0, which focuses on trusting and interaction-based relationships between humans and machines. Eight types of operators illustrate how technology could enhance human capabilities and skills.

Meanwhile, agile manufacturing enables product customization and on-demand production to respond quickly to customer needs and market changes. Moreover, because of the low cost of internet of things (IoT) devices, IoT devices have gained popularity in manufacturing settings to optimize business workflows and processes, improve safety and improve research and development. Nevertheless, such devices also generate unprecedented volumes of sensor data, necessitating operators’ constant attention and dramatically increasing the incurred cognitive load.

Interestingly, working in an agile connected production line is a real challenge for inexperienced workers. Even for experienced operators, operating an agile production line is a knowledge-intensive task that requires enormous cognitive resources. For example, a single worker may need to operate and fix numerous machines along a production line simultaneously, while also (re)optimizing the setup for more than 100 different products. Therefore, systems that could reduce cognitive demands and allow quick feedback are needed to effectively support workers on a personal basis.

Cognitive Assistants

Unlike systems designed to replace humans in specific tasks (e.g., industrial robots), cognitive assistants strive to complement human abilities to accomplish complex tasks, such as aiding life-long education and machine operation. In addition, such assistants often outperform human capacities for communication and memory in a variety of ways, such as simultaneously providing dependable and repeatable communication between numerous users.

To achieve the aims mentioned above, cognitive assistants should support efficient human-machine interfacing via natural language processing, interpretation of gestures, perception, vision, and sounds, augmented reality to provide additional layers of information, and others. Of these, the most widely used interaction method for cognitive assistants is conversational agent-based natural language communication, which involves natural language understanding, generation, and dialog for implementation.

Conversational agents engage with people using natural language, which could perform labor-intensive jobs at low cost in a variety of industries, such as customer service, healthcare, education, e-banking, and personal assistants. For example, recent work has proven conversational agents to be an effective educational tool for communicating breast cancer risk and suggested medical guidelines to women, leading to a significant increase in breast cancer genetics knowledge. Furthermore, advances in context awareness make it possible to improve a conversational agent’s usefulness. For example, acquiring more accurate knowledge about city locations by asking questions when users are there or inferring context from user utterances to provide more relevant tourist recommendations.

A virtual assistant called Amber was proposed by Kimani et al. using a sensing framework that could record users’ faces, speech, and app usage in order to aid users with job prioritization, provide reminders, and inhibit social media diversions.

Conversational agent-based cognitive assistants have also shown promise in supporting one’s cognitive processing. Le and Wartschinski developed LIZA, a cognitive assistant aiming at enhancing users’ reasoning and decision-making abilities. By holding a conversation with test subjects to help them solve common heuristic and bias problems, LIZA helped test subjects to improve their reasoning skills and achieve significantly higher learning gain.

Numerous application scenarios have also been proposed in which cognitive assistants have positive cognitive effects. From education and training, such as by employing lifelong learning to retrain adult workers to meet shifting technological demands, to elderly care by facilitating interaction with those suffering by cognitive decline.

Cognitive Assistants in Manufacturing

Industrial applications for cognitive assistants (CA)—similar to Alexa, Google Assistant, or Siri—are an emerging research topic. These prototypes emerged in different research communities with different names (e.g., intelligent (virtual/personal) assistants, digital assistants, software robots, or simply chatbots).

Cognitive assistants in manufacturing can bear significant benefits. These include, for instance, central access to heterogeneous information systems, the delegation of tasks, and gaze-free and hands-free interactions during work. Furthermore, cognitive assistants can be used for training workers on-the-job and adjusting machine parameters in a simulation. Most of the literature regarding CAs in manufacturing focuses on knowledge and information delivery, context-aware assistance, recommendations for predictive maintenance, decision support based on business analytics of shopfloor data.

  • For instance, Rodriguez et al. present a mixed reality assistance system to support real-time assembly operations. They evaluated the operation context through a recognition system that determines the completion of each assembly step to derive the next instruction. Belkadi et al.proposes a contextaware knowledge-based system aiming to support manufacturing operators. They integrate knowledge management, context management, and simulation management modules in order to support the decision making of the workers in real-time. As for Rodriguez et al., context management gets contextual information in order to understand the current user’s situation and implements simulation techniques to anticipate the effect of the worker decisions.
  • Büttner et al. implemented a hand-tracking algorithm in order to identify wrong picking actions and errors in the assembly process.
  • Tao et al. implements wearable device sensing and environmental sensing to capture worker activity in the workplace in order to guide them in the execution of their tasks, and Josifovska et al. integrates a context manager module which includes a Digital twin of a human that simulates specific human abilities and preferences in order to enable assistance system adaptation.
  • Longo et al. demonstrated a digital assistant integrated into an augmented reality application to train machine operators. Their prototype provides information about safety measures, potential hazards, machine status and operations, and quality control procedures. Besides, it instructs users on lubrication, greasing, cleaning, checking and restoring hydraulic pressure or fluids and lube for maintenance. Researchers have used several approaches to evaluate the effect of instruction delivery on workers’ mental workload and its effectiveness. Funk et al. evaluated the workload effect associated with the delivery of instructions on assembly work by monitoring biosignals (such as heart rate, galvanic skin response, electroencephalography, and electromyography) and indicators such as task completion time and error rates.
  • Likewise, Kosch et al. evaluated the cognitive workload produced by in situ projections during the execution of manual assembly tasks. They implemented electroencephalography (EEG) to monitor cognitive workload and compared the results with those obtained by traditional paper-based instructions. In line with the above-mentioned approaches, Funk et al. proposed a standardized experiment design for evaluating the effect of interactive instructions in heterogeneous assembly tasks.

Existing literature has also explored acquiring knowledge from workers but to a significantly lesser extent. This could be partially attributed to the less immediate benefits; however, we argue that the ability to continuously learn is necessary for the long-term success of CAs on the shop floor. Fenoglio et al. propose a system for capturing explicit and tacit knowledge (through best practices) from experienced workers in industrial domains. They implement a role-playing game where a virtual agent interacts with human experts and knowledge engineers in order to extract and represent knowledge in an iterative way.

Despite this system providing means to capture tacit knowledge, it requires the intervention of a human agent, as they argue that “is not possible to fully capture tacit knowledge (usually nonverbal and unexpressed) with a purely algorithmic approach”. Likewise, Soliman and Vanharanta suggests a model for knowledge creation and retention through artificial intelligence. However, there is no practical application of the model reported in literature, and Hoerner et al. propose a digital assistance system to support operator troubleshooting processes on the shop floor. For this purpose, a method to capture and structure expert’s tacit knowledge was developed. However, this method is not executed by the digital assistant. It is performed by human experts who are in charge of extracting and representing the obtain knowledge and to deliver it as input to the system. However, and to the best of our knowledge, no existing solution captures (tacit) knowledge from experienced operators with the purpose to structure it, store it, and re-share it with novices in real time and on the shop floor.

CLAICA eases the learning curve for novice operators by serving as a dialectic mediator among novices and current or past experienced operators. Even for experienced personnel, CLAICA provides multiple opportunities to rehearse and test acquired knowledge and skills, while constantly discovering and formalizing new knowledge or knowledge that has been overlooked.


The primary goal of CLAICA is to enable knowledge exchange between shop floor workers. It continuously learns by acquiring knowledge from workers through dialectic interactions, allowing it to efficiently share up-to-date knowledge. In addition, it can recommend existing work instructions and perform information retrieval tasks. Furthermore, workers can provide feedback on the knowledge they receive to improve recommendations over time.

What sets CLAICA apart from the state-of-the-art is its ability to efficiently acquire knowledge from workers on the shop floor without human involvement and store it in a knowledge graph along with contextual information. However, knowledge managers may still be needed to perform quality control by approving, reviewing, and removing elements of knowledge. Ultimately, CLAICA aims to reduce the burden on knowledge managers and improve knowledge sharing (see Figure 1).

Figure 1: CLAICA’s architecture (below) and the simulation (above)

Co-designing CLAICA

CLAICA was developed in close collaboration with an industrial company, a detergent producer. Their ambitions for the assistant are to result in faster training of new operators and higher overall equipment effectiveness (OEE), that is, the percentage of manufacturing time that is truly productive. During the early design phase of CLAICA, we conducted semi-structured interviews and focus groups with operators and management at two detergent factories.

We explored three main topics, namely

  • what opportunities are there to support machine operators with a cognitive assistant,
  • what wishes do factory employees have regarding the (interactive) capabilities of the assistant, and
  • what challenges the assistant might face from a user acceptance perspective.

We interviewed six machine operators, two maintenance technicians, two shift leaders, four engineers, and one factory director.

We pinpointed the following opportunities:

1) identify the best practices of operators, elicit this knowledge, and share it with others,

2) make existing work instructions more easily accessible,

3) help operators identify the root cause of problems,

4) provide operators with access to machine data anywhere on the production line, and

5) create high-quality issue reports.

The operators were primarily interested in receiving ubiquitous access to machine data, while their supervisors were also concerned with facilitating knowledge sharing among operators, providing access to instructions, and creating better issue reports. The shift leaders suggested that operators, even experts, could benefit from suggestions; however, the operators themselves disagreed. We noted that the operators were very proud of their skills and proud that they did not need instruction material.

However, they thought that novice operators could benefit from easy access to up-to-date documentation. Existing documentation resides in paper and digital form, but is poorly structured and frequently outdated. As such, novice operators are trained almost exclusively by experienced operators on-the-job. Due to the complexities of operating an agile production line, this process is lengthy and, therefore, costly.

Furthermore, it is risky from the company’s perspective, as a lot of valuable (tacit) knowledge will be lost when experienced operators leave. We collected 100 issue descriptions on one of the production lines over three days. We asked the operators to verbally describe the location, symptoms, and cause of each issue as it occurred. The operators used a mono headset during data collection (see Fig. 2).

Figure 2: An operator describing a problem with the production line

We analyzed the resulting reports to identify opportunities and challenges for CLAICA. We observed that operators used different terms and acronyms when describing the same machines (e.g., depa, palletizer, depalletizer). In addition, the production line breaks down frequently (up to 30 times in an eight-hour shift) and the operators are under intensive time pressure. As a result, many problems go undocumented. Once we built a working prototype, we presented it to the operators (see Fig. 3) to elicit further feedback.

Figure 3: An operator using CLAICA at the production line

The insights from the interviews, the focus groups, and the collection of issue descriptions resulted in the following design requirements for CLAICA.
• It must provide accurate cognitive support to novices
• It must interact efficiently and reliably
• It must support user feedback
• It must continuously learn from its (expert) users
• There must be incentives for (expert) users to share their knowledge (this responsibility is shared with the company)
• It must reduce the reliance on expert operators for training and supporting novices
• It must be transparent about what data it collects and how it provides recommendations
• It must be able to handle divergent phrasing and misunderstandings gracefully

We presented CLAICA, a continuously learning AI cognitive assistant that provides support to agile manufacturing workers by exchanging knowledge and providing quick access to instructional material. CLAICA is the product of a co-design process with factory workers and the focus of a user study with 83 participants. Our findings contribute to a deeper understanding of how prior training, context expertise, and interaction modality affect the user experience of cognitive assistants. Drawing on our findings, we elicit design and evaluation guidelines for cognitive assistants that support knowledge exchange in cognitively-demanding tasks and in challenging environments (e.g., an agile production line).

This work was supported by the European Union’s Horizon 2020 research and innovation program via the project COALA “COgnitive Assisted agile manufacturing for a LAbor force supported by trustworthy Artificial Intelligence” (Grant agreement 957296).

This article has been extracted and freely adapted from ACM Digital Library website (March 2023) – Lessons Learned from Designing and Evaluating CLAICA IUI ’23, March 27–31, 2023, Sydney, NSW, Australia

Authors: Samuel Kernan Freire, Evangelos Niforatos, Chaofan Wang, Santiago Ruiz-Arenas, Mina Foosherian, Stefan Wellsandt, Alessandro Bozzon