Rocksolid Light - comp.lang.cobol

New meet old

<uuct5j$22r9n$1@dont-email.me>

https://news.novabbs.org/devel/article-flat.php?id=940&group=comp.lang.cobol#940

Path: i2pn2.org!i2pn.org!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: arne@vajhoej.dk (Arne Vajhøj)
Newsgroups: comp.lang.cobol
Subject: New meet old
Date: Sun, 31 Mar 2024 19:55:30 -0400
Organization: A noiseless patient Spider
Lines: 23
Message-ID: <uuct5j$22r9n$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Sun, 31 Mar 2024 23:55:31 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="2d37047cfe23e50ce3755a76b90d2c83";
logging-data="2190647"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18Dn3jGrQiOVry28ZiTBLaUPN36VnHKH8s="
User-Agent: Mozilla Thunderbird
Cancel-Lock: sha1:6KBry4SiUzq/RFIpYODSLb5Q40g=
Content-Language: en-US

by: Arne Vajhøj - Sun, 31 Mar 2024 23:55 UTC

Someone created a framework for evaluating LLM's ability
to write Cobol.

https://bloop.ai/blog/evaluating-llms-on-cobol

For those that do not bother reading the entire article,
then the conclusion at the bottom is:

<quote>
GPT-4 - the best-performing model - generates a correct solution for
10.27% of problems. Compare this to HumanEval, where it solves 67% of
problems. CodeLlama, one of the best open-source coding models, fares
even worse, with the 34b variant only clocking 2%. COBOLEval is hard.

Looking at the failure cases, we can see that state-of-the-art LLMs
struggle to generate COBOL that even compiles. Only 47.94% of GPT-4
generated solutions compile with GnuCOBOL.
</quote>

Arne

And on the seventh day, He exited from append mode.

devel / comp.lang.cobol / New meet old