From mboxrd@z Thu Jan 1 00:00:00 1970 From: Daniel Berlin To: Elena Zannoni Cc: Daniel Berlin , gdb-patches@sources.redhat.com Subject: Re: [RFA]: Add misc_obstack to object files. Date: Mon, 25 Jun 2001 23:21:00 -0000 Message-id: <87vgljg3rl.fsf@cgsoftware.com> References: <87lmngjc5w.fsf@dynamic-addr-83-177.resnet.rochester.edu> <15159.42589.632817.275157@kwikemart.cygnus.com> <87zoawngdr.fsf@cgsoftware.com> <15160.1620.839855.907203@kwikemart.cygnus.com> <8766djhoc9.fsf@cgsoftware.com> <15160.7951.174487.798388@kwikemart.cygnus.com> X-SW-Source: 2001-06/msg00420.html Elena Zannoni writes: > Daniel Berlin writes: > > Elena Zannoni writes: > > > > > Daniel Berlin writes: > > > > Elena Zannoni writes: > > > > > > > > > Daniel Berlin writes: > > > > > > > > > > > > This way, we stop putting things in the psymbol_obstack that don't > > > > > > belong. > > > > > > > > > > > > > > > > Dan, can you elaborate a bit on why we need this? > > > > Because these things don't belong in the psymbol obstack. > > > > They are section offsets, not partial symbols. > > > > > > > > > > Dan, I am really sorry, but still I cannot see a real reason for this > > > change emerging from your mail. > > > > Um. > > Okay. > > According to objfiles.h: > > > > struct obstack psymbol_obstack; /* Partial symbols */ > > struct obstack symbol_obstack; /* Full symbols */ > > struct obstack type_obstack; /* Types */ > > > > > > Notice it doesn't say that section offsets, or other random junk, > > should be on the psymbol_obstack. > > It says "Partial symbols". > > Now, if you want to remove the partial symbols, for that object file, > > theoretically, you just remove the psymbol_obstack. However, because > > we allocate other random things, like section offset tables, etc, > > which have zip to do with partial symbols, on that obstack, we won't > > remove just the partial symbols by removing that obstack. We'll blow > > away the section offset table, the psymtab structures, etc. > > This should not be occurring. > > It makes as much sense as allocating object file section offsets on the > > type_obstack, which is for types only. That is to say, none. > > > > Yes, and this is the bug I was alluding to. You mentioned you observed > that things like the section tables were freed too early. When does > this occur? When you attempt to free the psymbol obstack after you are done with all the psymbols for an objfile. However, gdb currently doesn't do this. This is part of the reason you can't. > Did you try to move around the call to > free_obstack(psymbol_obstack)? There are only calls to do this type of thing when we remove object files or reread symbols. It would, of course, to make sense to free the partial symbols for an objfile once we were sure all the symtabs for that objfile had been read in. This does not currently occur. The bug occurs when you attempt to do this. The actual reason it never occurs in gdb is because the stabs reader incorrectly expects partial symbols to live past the readin of full symbols (why would you want partial symbols when you have full ones?), in order to do some duplicate elimination the cheap, easy, and stabs specific, way. It does this at the expense of every other symbol reader, since if you attempt to free the psymbol obstack for stabs, even after you are done with all the partial symbols for a given objfile, you'll start seeing weird stabs reader errors. However, this I consider a seperate problem, since gdb currently doesn't ever free the psymbol obstack except when it frees objfiles. > What motivated you to try and do that? The fact that we waste effectively double the memory to read symbols for everything except stabs, because none of the other symbol readers assume the psymtabs will live past complete symtab conversion, and thus, assume they will be freed. The STABS reader happens to eliminate enough psymbols through it's cheap easy method that you don't notice the fact that it's keeping all the psymbols around, never freeing them. Besides attacking this part of problem by fixing the stabs reader, you must also make it so you can actually free the psymbol obstack when you are done with all the psymtabs for an objfile, without affecting anything else about that objfile. I.E. make it so the psymbol_obstack only contains psymbols. Otherwise, fixing the stabs reader does you no good, since you still will blow away unrelated things when trying to free the psymbols. So in order to fix the problem of keeping psymbols around until their objfile goes away, rather than until the symtabs for that objfile are all read in (Or better yet, until a given psymab is converted into a symtab), I need to start with the small things like moving the structures out of the psymbol_obstack. > That's really what I am asking. > > > It's the equivalent of having a trash bin marked "white paper only", > > and having people knowing throw all kinds of colored paper in there, > > for lack of a better place to throw colored paper. > > They don't belong. > > Not only that, but if you later wanted to remove the white paper, you'd have to sort out > > the colored paper, and do something with it, first. > > > > I'm just adding a bin for colored paper, so people will put that stuff > > in there instead in the first place, and it will be plainly clear that they are supposed > > to be doing so. > > > > > > > You are doing this in response to some bug you saw, I assume, but what that > > > bug is has not been explained. > > No, i'm doing it because partial symbols belong on the psymbol > > obstack, but section offsets, and other random things don't. > > In practice, we've not followed this, because nobody bothered to > > create somewhere *else* to put them. I'm attempting to bring us > > to doing what we should have been doing in the first place, by > > creating this place to put them. > > > > Yes, true. > > > Had I been trying to fix a bug, i would explain what the bug was, and > > where it showed up. > > > > This is not a bug, in that nothing is specifically *broken*. It's > > perfectly okay to allocate these structures on that obstack, given > > that gdb never frees the obstack, it's just not in line with what the > > comments say, and the purpose of the psymbol obstack. > > > > Isn't the psymbol_obstack freed when you re-read symbols in? Yes. And when you free objfiles. However, this is not really it's minimal lifetime. The usefulness of a psymbol_obstack for an objfile ends the second you've "converted" (i put it in quotes since conversion usually involves transforming one thing into a nother, where here we are really effectively copying, and transforming the copy, since we never remove the original) all the psymtabs into symtabs. > > > > > > > > > > > > Why are you removing > > > > > the partial symbol table from the psymbol_objstack? > > > > I'm not. At all in fact. > > > > I'm specifically trying to make it so *all* that is in the > > > > psymbol_obstack is the partial symbol table. > > > > we've been stuffing random stuff like section offsets into the > > > > psymbol_obstack, because it was handy. > > > > This is wrong. > > > > If we need a misc place to put info like section offsets, that isn't > > > > large enough to deserve it's own obstack on it's own, then all the > > > > misc stuff together should go in a misc_obstack. > > > > > > Then your patch includes a diff that is unrelated. > > Um, how so? > > Please point out the portion of the diff unrelated. > > The only thing even related to psymtabs there is the fact that they > > psymtab structure itself is allocated on the misc obstack, since it > > isn't a partial symbol either. It's a partial symbol table structure, > > containing pointers to partial symbols. > > Yes, that's the one. > > I would think that the psymtab itself is better coupled with > psymbol_objstack, than misc_obstack. Okay, that's fine. > I would think it safer to free both of them or none, than to risk > freeing the section_offsets and also the psymbol table by mistake. Sure. > > > The actual partial symbols that fill that table are still allocated on > > the psymbol_obstack. > > As they should be. > > yes, but looking at the way the symbol table is allocated, it follows > the same logic, symbol_obstack includes the symtab itself as well. Real symbols are different. Their lifetime really ends with the objfile. Partial symbols are just there until we have a need to read all the real symbols. So their lifetime is different. > > I mean, the real problem that you are trying to address is that you > need a different lifetime for psymbols vs psymtab, and for psymbols vs > section_offsets. So, why is this necessary? I think i've explained this above, but in different parts of the message so, for clarity's sake, this particular thing is necessary because: The lifetime of psymbols for an objfile is until all the symtabs for that objfile are read in (Theoretically, if you were careful, you could make the lifetime until those psymbols are converted to symbols, but it's tricky, because of how obstacks work. The whole freeing everything that came after the object you want to free stuff). Once this is done, the psymbols serve 0 purpose. Provably, no lookup that fails on the symbols could work on the psymbols, unless your reader is broken (since they are coming from the same info). > Are you addressing a > limitation that you have found while trying to do something else? Yes. This is the first step of addressing the limitation that we keep psymbols around longer than necessary. This becomes a factor for STABS when you have very large programs (IE duplicate symbol elimination doesn't save you, because you really have 500k distinct symbols and types), and for other debug formats which don't have linker support for duplicate elimination, much quicker. Even with a gdb compiled with dwarf2, where gdb isn't that large of a program in terms of psymbols (because c has few duplicates compare to C++, and because C++ programs have often have more of the things that we generate psymbols for) shows the following, after all symtabs are read in (IE when psymbols are, bye, definition, useless) (top-gdb) maintenance print statistics Statistics for '/buildspace2/dwarf2gdbtree/src/gdb/./gdbdwarf2': Number of "minimal" symbols read: 7149 Number of "partial" symbols read: 289144 Number of "full" symbols read: 445652 Number of "types" defined: 162155 Total memory used for psymbol obstack: 12389013 Total memory used for psymbol cache: 745176 Total memory used for symbol obstack: 35083179 Total memory used for type obstack: 28099722 Out of 75 meg of memory, we are wasting 1/6th on psymbols. This is probably average for C programs. For C++ programs, you tend to see a psymbol obstack equal in size to the symbol obstack. In other words, we waste double the memory on symbols. > Why > is it a problem that section_offsets, psymbols and psymtab have the > same lifetime? section_offsets logically have the lifetime of the objfile. psymbols and the associated psymtabs have a lifetime that lasts exactly until the full symbols for those psymbols are read in (IE the psymtab is converted to a symtab). Right now, the we treat them like they last until the objfile goes away. I'm working on treating them like they last until all the symbols for a given objfile are read in (which happens fairly quickly unless you have a huge, monolithic program). This is because treating them like they have the lifetime they really do would require being able to free all the psymbols for a given psymtab without touching any other psymbols. Because of how obstacks work, doing this would require not using obstacks for psymbols, or rather, making the obstack part of the psymtab (so you can free all the memory associated with the psymtab without touching anything else) Eventually, that is the goal, but that's way off. Since it is a complex change to do that anyway, and it's more of an incremental step from being able to free psymbols at *some particular* point before the objfile goes away, this actually gets us closer to being able to do that. > [I am not arguing that section offsets are logically related to psymbols] > > > > > > > > > > > > > Was this intended? > > > > > There are several other things that get allocated on the > > > > > psymbol_objstack, for instance look in xcoffread.c, somread.c, > > > > > etc. Why just remove the section offset stuff? > > > > Because I haven't gotten to the others yet. > > > > And there is no harm in taking time to make sure it gets done right. > > > > > > > > > > If you have a roadmap in mind, you should explain it when you submit > > > the patch. > > > > I have no roadmap for when i'd be getting to that. > > Eventually. > > I could do them all now if it helps, but it'll make the patch bigger. > > > > I would suggest that you put the section offsets stuff on its own > separate obstack for now, but do that across the board, to avoid > inconsistencies. That would be a self contained patch. > Follow up patches would move other pieces. Okeydokey. > > Elena -- "When I was a kid, I went to the store and asked the guy, "Do you have any toy train schedules?" "-Steven Wright