Style-Aware Bangla News Headline Generation Using Large Language Models: A Comparative Zero-Shot Evaluation

Abstract

     News headline generation is an essential task in natural language processing (NLP), as it plays an important role in conveying the main message of a news article in a concise form. Although this task has been widely studied in English and other high-resource languages, Bangla headline generation remains less explored, especially when the headline needs to follow a specific writing style. In this study, four Large Language Models (LLMs), namely GPT-OSS-120B, LLaMA-3.3-70B, Qwen3-32B, and Qwen2.5-14B-Local, are evaluated for style-aware Bangla news headline generation in a zero-shot setting. The models were not fine-tuned for this task; instead, they were guided only through style-specific prompts. For the experiment, 75 Bangla news articles were used, and headlines were generated in six styles: Political, Informative, Dramatic, Short, Analytical, and Event-Based. A total of 1,800 generation attempts were performed. The generated headlines were evaluated using 15 automatic metrics, including BLEU-1/2/4, METEOR, ROUGE-1/2/L, WER, CER, Edit Distance, TF-IDF Cosine Similarity, Jaccard Similarity, and BERTScore Precision, Recall, and F1. The results show that LLaMA-3.3-70B achieves the best overall performance, with a BLEU-1 score of 0.1454, a METEOR score of 0.1333, and a BERTScore-F1 score of 0.2064. Qwen3-32B obtains the highest BERTScore Recall score of 0.2144, indicating better coverage of the reference headline content. The results also indicate that the Informative style produces more accurate and consistent headlines than the other styles. Overall, this study demonstrates the potential of zero-shot LLMs for style-aware Bangla headline generation, while also highlighting the limitation of using a single reference headline for evaluating creative generated outputs.