Australian Study Reveals AI Could Create More Work For Employees

Humans throttled AI in tackling a series of business-related tasks in a workplace setting.

According to a new study from Australia’s Securities and Investments Commission (ASIC) and Amazon, artificial intelligence might make more work for employees after researchers used Meta’s open-source Llama2-70B to complete business-related tasks in a workplace setting.

, the artificial intelligence (AI) programs and a group of 10 humans were asked to summarize submissions focusing on the ASIC, recommendations, and references to regulation. In addition, the tool was asked to include both context and page references. Both sets of responses were then blindly assessed by reviewers for coherency, length, and identification of ASIC references, as well as regulation references.

The artificial intelligence programs performed poorly compared to their human counterparts, a 47% score compared to the humans’ 81% score. According to the team that conducted the study, the AI performed especially badly when it came to finding references to the AISC in the documents it was supposed to use to create its summaries.

“Finding references in larger documents is a notoriously hard task for LLMs due to context window limitations and embedding strategies,” the team wrote. “Page references are not traditionally stored in the embedding models as the contents of PDF documents are ingested as plain text. To achieve better accuracy with this issue, substantial progress was made by splitting documents into pages and treating pages as chunks with associated metadata.”

According to the study, the reviewers often had “to refer back to the source material to confirm AI summary details,” and they “generally agreed that the AI outputs could potentially create more work if used (in current state), due to the need to fact check outputs, or because the source material presented information better.”

Although this study is limited in its application of artificial intelligence, it does imply that the expected benefit of AI could be tempered by humans having to do more work to fit AI’s specific needs in the workplace.

Several reports have indicated that Black people and AI are at odds, a group of Stanford University researchers offered some considerations for the Congressional Black Caucus in a March 2024 white paper.

According to the researchers, “While AI holds the potential to deepen racial inequalities, it can also benefit Black communities. If deployed carefully, AI has the power to improve access to healthcare and education, as well as create new economic opportunities. For example, AI can help doctors make more accurate diagnoses and provide personalized treatment plans, particularly in underserved communities where access to healthcare is limited.”