From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 130881 invoked by alias); 27 Oct 2018 04:56:58 -0000 Mailing-List: contact gdb-patches-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sourceware.org Received: (qmail 130852 invoked by uid 89); 27 Oct 2018 04:56:56 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-1.8 required=5.0 tests=AWL,BAYES_00,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=no version=3.3.2 spammy=HX-Received:sk:d4-v6mr, love, H*c:HHHHHHHH, H*c:HHHH X-HELO: mail-pl1-f172.google.com Received: from mail-pl1-f172.google.com (HELO mail-pl1-f172.google.com) (209.85.214.172) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Sat, 27 Oct 2018 04:56:55 +0000 Received: by mail-pl1-f172.google.com with SMTP id t6-v6so1379212plo.9 for ; Fri, 26 Oct 2018 21:56:55 -0700 (PDT) Return-Path: Received: from resnet-31-44.resnet.ucsb.edu (ResNet-31-44.resnet.ucsb.edu. [169.231.31.44]) by smtp.gmail.com with ESMTPSA id x13-v6sm33099567pge.13.2018.10.26.21.56.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Oct 2018 21:56:52 -0700 (PDT) From: Saagar Jha Content-Type: multipart/mixed; boundary="Apple-Mail=_81B44216-F393-4C18-AC3C-6453980DC296" Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: [PATCH] Prevent overflow in rl_set_screen_size Message-Id: Date: Sat, 27 Oct 2018 04:56:00 -0000 To: gdb-patches@sourceware.org X-IsSubscribed: yes X-SW-Source: 2018-10/txt/msg00638.txt.bz2 --Apple-Mail=_81B44216-F393-4C18-AC3C-6453980DC296 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Content-length: 619 Hi, I was running GDB under the undefined behavior sanitizer and I found a sign= ed integer overflow in set_screen_size. I=E2=80=99ve attached a (IMO slight= ly clumsy, but I couldn=E2=80=99t think of a nicer way to solve this) patch= that solves this issue. I couldn=E2=80=99t figure out how to formally test= this code, but I can compile and run this on my computer running macOS Moj= ave 10.14.1. Would someone mind taking a look at this? This is my first set= of contributions to GDB, so if there=E2=80=99s anything wrong (or you have= general feedback) I=E2=80=99d love to hear about it! Regards, Saagar Jha --Apple-Mail=_81B44216-F393-4C18-AC3C-6453980DC296 Content-Disposition: attachment; filename=Prevent-overflow-in-rl_set_screen_size.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="Prevent-overflow-in-rl_set_screen_size.patch" Content-Transfer-Encoding: quoted-printable Content-length: 1762 >From 9bf98bc3963d822e33f485067907b191420bb8e4 Mon Sep 17 00:00:00 2001=0A= From: Saagar Jha =0A= Date: Tue, 22 May 2018 04:08:40 -0700=0A= Subject: [PATCH 1/5] Prevent overflow in rl_set_screen_size=0A= =0A= GDB calls rl_set_screen_size in readline with the current screen size,=0A= measured in rows and columns. To represent "infinite" sizes, GDB passes=0A= in INT_MAX; however, since rl_set_screen_size internally multiplies the=0A= number of rows and columns, this causes a signed integer overflow. To=0A= prevent this we can instead pass in the approximate square root of=0A= INT_MAX (which is still reasonably large), so that even when the number=0A= of rows and columns is "infinite" we don't overflow.=0A= =0A= gdb/ChangeLog:=0A= 2018-05-22 Saagar Jha =0A= =0A= * utils.c: Reduce "infinite" rows and columns before calling=0A= rl_set_screen_size.=0A= ---=0A= gdb/utils.c | 6 ++++--=0A= 1 file changed, 4 insertions(+), 2 deletions(-)=0A= =0A= diff --git a/gdb/utils.c b/gdb/utils.c=0A= index 8d4a744e71..56257c35cf 100644=0A= --- a/gdb/utils.c=0A= +++ b/gdb/utils.c=0A= @@ -1377,11 +1377,13 @@ set_screen_size (void)=0A= int rows =3D lines_per_page;=0A= int cols =3D chars_per_line;=0A= =20=0A= + // Use approximately sqrt(INT_MAX) instead of INT_MAX so that we don't= =0A= + // overflow in rl_set_screen_size, which multiplies rows and columns=0A= if (rows <=3D 0)=0A= - rows =3D INT_MAX;=0A= + rows =3D INT_MAX >> (sizeof(int) * 8 / 2);=0A= =20=0A= if (cols <=3D 0)=0A= - cols =3D INT_MAX;=0A= + cols =3D INT_MAX >> (sizeof(int) * 8 / 2);=0A= =20=0A= /* Update Readline's idea of the terminal size. */=0A= rl_set_screen_size (rows, cols);=0A= --=20=0A= 2.19.1=0A= =0A= --Apple-Mail=_81B44216-F393-4C18-AC3C-6453980DC296 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Content-length: 1 --Apple-Mail=_81B44216-F393-4C18-AC3C-6453980DC296--