Hello cyberpals.
I've been doing a lot of
classic Mac projects lately.
One of them is the PowerBook 100
series batteries that I talked about in the last video.
These things. I ended up making
60 of these things and sold all of them,
except I think I have like four left, and so I'm done with
that. I'm not going to be making any more because I don't really
want to be in the battery assembling business.
So maybe somebody else can pick up that project. I still had
a lot of requests for them in Europe,
but I wasn't able to find a way
to ship them. No one seems to want to do that
because they're batteries, even though they're not lithium ion. But if
somebody, I guess over in Europe wants to make them, you'd probably
sell a whole bunch. But anyway,
so as far as my Mac software
projects-- Subtext, my BBS, I just launched
version four yesterday,
has a bunch of kind of internal changes, and I added syslog
support so you don't have to look at the console window all the
time. It can also send every logged
event to a syslog server that you can watch through
SSH or have it e-mail you the logs every day or whatever.
And I'm actually using that on my server to watch when
people log in and out. And then I have it posting to a Mastodon account
each time somebody logs in and out of my bbs.
So I'll put a link to that in the description if
you want to follow that account, I guess.
And then I've been mostly working on Carl, which is my IMAP
client that I started this whole video series
with. I kind of got sidetracked by all my other projects and
hadn't worked on it in a while, but I started working on it at
the beginning of the end of last year with
a silly goal that I was going to do all of my e-mail reading
and writing through Carl. So I
was doing a whole bunch of work on it, and then I kept running into
one more thing that I needed to add to be able to decode a
particular message, and it just kind of kept snowballing. And so
it was already like January, and... now
it's almost mid-late February
and it's still not done. So anyway,
I am still working on it and got
a lot of progress done. So I thought I would make a video showing
that. I don't remember what state it was
in the last time I made a video about it, but it was pretty rough
because when I went back to it, now. Or maybe it was just looking at
it with fresh eyes, but I was like man, this code is all kind of
in disarray. But this
util c that I carry around all my projects, it's gotten so much bigger
now because of all the little utility functions that I've had to
write. But anyway, so if
I just run this now,
it's at the point where it can pretty much read most of my non
HTML e-mails. And that's
actually harder than it sounds because there's so much encoding
and stuff that's going on under the hood that you don't even really notice that
an IMF client has to account for.
So this is running a little bit slower than normal because I have Malloc
debugging enabled, which I'll talk about in a little bit.
So this is the main screen. I think it probably looks similar to what I
had before. And you can view a message and
it downloads it and decodes it and prints it.
There's a settings screen now,
so you can define all these various things,
supports masked password fields that I
wrote for Wallops, I think. And then I had to fix it for
the Wi-Fi DA-- the Desk Accessory
to make password fields work on like macOS 8 and newer.
And then you can define a time zone now because it properly accounts
for time zone offsets when converting dates
and stuff. And then it has the SOCKS5
proxy support. And so this is actually running--
we're talking to a TLS
decrypting proxy that I have on my firewall.
So I talk plain text to the SOCKS5 proxy and then
it makes a TLS connection to FastMail
and then does all the decryption and then passes
back plain text to me.
So this all works now. And it reads all the settings out of there.
Yeah, so basically I've had to do a
whole bunch of work kind of under the hood.
So to just give an example... let me get out
of here.
If you look at what is in a message,
like an e-mail message,
there's so much crap in it that you don't even need.
And if you're on a slow connection with limited CPU power like this
is, you want to avoid having to download the whole
thing and then parse it and then just take out what you need.
So this is a structure of a tiny e-mail. It has a one line body
with some formatting. I think I bolded a
word and it has UTF-8 text in it,
and there was a small image attached to it.
And so this is the actual e-mail. These are all headers.
This is all just garbage that you don't need,
that the e-mail program is not going to do anything with. So you
would need to download all of this just
to... it just keeps going and going...
just to get to
useful things like an in reply to header the,
from, the date, the message id, and subject. And so
these are the only things that you really need for
what Carl needs to display the e-mail and reply
to it. So you'd have to download all those headers and then
parse them all. And then once you get to the body, it's a
MIME encoded body because it's
a multipart/alternative message.
Because most modern e-mail clients,
when you do formatting, it encodes it as HTML,
and then usefully, helpfully, it also
sends a text/plain part, which is a text/plain
representation of the encoded or of the HTML message just
with the formatting stripped out. And so that's what Carl is displaying,
but it would have to download all of this and parse it just to get
that text out. And then once you get to the body, you only want
to display this one single line,
but then you get stuff like this, which is quoted
printable. And so this is how you encode a
UTF-8, or any character that's not seven bit.
Because for whatever reason, in 2024,
SMTP is still restricted to seven bits.
So anything that's not, that's over seven bits, like high-ASCII
or a UTF-8 character or whatever, has to be encoded
with something that like quoted-printable or base64
which is what the, the actual
image attachment is.
So you have to do all of that to be able
to print these two characters,
these two, and then a
program like Carl has to parse that,
know that it's quoted printable, decode these into their actual characters,
and then convert the UTF-8 characters
into MacRoman, which is the character set that
Classic Mac OS uses. So if you're displaying like accented
characters, which these are, these are UTF-8 characters,
but Mac OS can't display them. So it
basically needs to like download, it needs to find the
part that it needs to download, then it needs to download it, then it needs
to unquoted-printable the
characters, then it needs to run it through a UTF-8 decoder,
convert them, find the UTF-8 characters,
then run them through another utility function that
converts UTF-8 to MacRoman. And so there's like a big
table of every MacRoman character
and its possible UTF-8 counterpart. And so it needs
to swap them back and forth. So that it can display an accented
character on a Classic Mac OS. Then you type
your reply because-- say you're quoting
this message in your reply and then you type something and
then you send it through SMTP. You need to do all
of that again in reverse. You need to convert MacRoman
characters that you're typing or are seeing
displayed, and then you need to convert them into UTF-8
Then you need to either encode it quoted-printable, or in base64,
and then send that to the SMTP
server so that it arrives at the recipient
so that their e-mail client can do all that in reverse again.
It's kind of dumb, but e-mail and
SMTP is like I don't know how many decades old at
this point, but it all works and it's all what we're still using,
but it's just kind of gotten really bloated. So anyway,
this is the text/plain version. So this one single line is all I need
to pull out of that body. And I guess
the rest of this is the quoted response that I also want to display.
But then it needs to then ignore this HTML version
of it and then it doesn't do it
yet. But it would need to know that this final part is an
attachment that's not meant to be displayed in line and then show like
a button or a link that says download or save this
attachment. And then to do that it would need to
download and decode this final section
and then de-base64
and then save the raw binary data.
So that's a lot of work to do for every message.
And helpfully, IMAP has
a MIME decoder. IMAP-- the specification for IMAP4
requires that you do MIME decoding on the server. And so
a client like Carl can ask the server to
give it the body structure of that entire message that
I just showed. So the server reads
that whole e-mail and then runs it
through its MIME parser and then gives me back this one line which
is also a pain in the ass to parse. So this is basically saying
that this is one part. There's a text/plain part,
it's encoded in UTF-8,
which is encoded in quoted printable,
and then it has this size,
but then there's also this part which
is text/html, in UTF-8, it's quoted-
printable, 702 bytes. And then
the third part of this says that it is alternative.
And so of this
closing parentheses closes this
one. And so you're supposed to decode those three and
know that the, the final one says it's alternative. And so the first two
are not meant to be displayed together. You're supposed to
pick one or the other. And then there's another part
that is an image/png
with a name, it's base64 encoded with this size.
And yeah,
so you can basically do this for every message. So you ask the server
to basically parse it on its side and just send me back and so you
can tell it. So this is actually like part one one.
And so instead of fetching that whole message, Carl tells the
server send me part 1.1. And the server
would only reply with that text/plain part
of that one line with the quoted text.
So that's really helpful because you don't have to download the entire message and parse
it only to show a small part of it. And for a lot of these
messages, like if you send an e-mail-- image attachment,
that one image attachment can exceed the whole memory size of my Mac
Plus. So there's no way that I'm going to be able to efficiently download that
whole thing and then parse it just to show like
a link that says download that part. Anyway.
So all of that is working. And so there's a lot of
having to parse all of these things from the IMAP
server and just to get the small bits
of text that I want to display.
And unfortunately a lot of e-mails,
a lot of automated e-mails that come from eBay or your
bank or whatever, they don't bother to include that alternative part
that is text/plain, they only include like a text/html version.
And so in those cases Carl would
ask the server to parse the body structure. The body structure would
come back and just say there's one text/html part and
in that case I can't do anything. I just have to display a message that
says there's no plain
part of this message to view.
So that's unfortunate. And I don't really want to write like an entire HTML parser
just to try and extract some text, but I might have to because
otherwise half the e-mails that I receive I can't
even view. On my modern
computer, I still use Mutt, which is all text based.
And so for those e-mails where I only receive, where it only has a
text/html part, Mutt is configured to actually
pipe that through Lynx, which is a text-only web browser.
And so it just basically dumps all of the text that it extracts
out of there in a slightly formatted way so that I
can at least read what the text is and reply to it. So I'm
hoping I don't have to go that route of writing an HTML parser that can
extract all the text just to show you something. But I might have to
anyway, so the big thing that I have since
the last, since I last worked on this is
that I can reply and I can type a reply
and I can send it and it sends it over SMTP,
this is not drawing right.
So as I was saying before, as I'm typing in
that window that's in MacRoman,
but if there were UTF-8 characters in there, or that if there were
high-ASCII characters in there, like accented characters,
when it sends it through SM-- oh cool. I guess I have a
leak somewhere.
It would need to then encode it in--
or convert it from MacRoman to UTF-8 and do like quoted
printable encoding on it or something.
So that's cool.
So to get all that working, I had to do like-- I had to
write base64 encoding and decoding functions.
I don't know why this is not resetting.
I had to do quoted-printable encoding and
decoding. And then the other thing is that in
a header field like the from address, if the person's name
has an accented character in it, or if the subject has an accented
character in it, there's a way to encode words,
individual words, as quoted printable,
and I can show you what they look like. But that's another thing that
you have to then decode from the server and run
it through like a decoder, because you can
encode-- like in your subject line, you could have
two words, three words. The first one is encoded in UTF-8
in quoted printable, the second word
can be base64
in like ISO-8559 or whatever.
And then the third one can just be plain text. And if you look at
that in a modern e-mail client, it just looks like three regular words that are
next to each other. You don't know anything about it. But a program like
Carl has to do all that decoding and show
them in that same plain text
way. I don't know if there are any in here.
Teach text doesn't have find, that's annoying.
I don't know why that SMTP send crashed. I've been
using this to send and receive e-mails.
Anyway, so I had to write those decoders and
encoders and then I had to do I
added the settings window that I showed, and then I imported my
userland thread library from Subtext so
that all the IMAP operations are now running on a background thread,
and then yielding to the main
GUI thread whenever it's not doing anything.
And just like in Subtext how I needed that is so I could write
top down code and just say, send something to the server,
wait for the reply, read it, and then
parse it. And then that wait for
it is when it does the userland thread-- like the thread
yield to the other thread so that I could just write
those three things in a row, I don't need to send it to the server,
then exit out of the function, go back to the main loop,
do whatever, reenter into the IMAP thread,
into the IMAP routine, try and figure out where I was,
and then go back in there trying to wait for the
server response. Because the way that I had it shown before,
I was doing this like having to maintain state. And then there was like one
big loop that said which state are you in? Go to this function.
And as I've been writing more and more IMAP code
to do lots of individual
functions, it got very tedious to try and
do that state. So I could basically say send
a command, sit in a loop,
expect a line, that is the response from the server with
a LIST response. And then imap_expect line is
what reads from the server. And then if there's no data it just does--
yield so that it
doesn't need to maintain any state. It can just do this and this isn't actually
blocking anything. So then it can read
the response and then do whatever it
needs to do. And it's just been a
lot easier to write code that way. The other annoying thing about
IMAP is, I guess the annoying thing about IMAP is
that when you
have a mailbox open and you have a concurrent,
or when you have a mailbox open and you're
reading messages and you're
just sitting there with an idle connection waiting for new messages to show,
or waiting for the client to,
waiting for the user to want to read an e-mail so that you can
say send to the server. The server can send you updates
whenever it wants, an update being like there's a
new message in this mailbox. Or if you have another IMAP client that
is open somewhere else and it's also reading that
mailbox and doing stuff to it, like marking a message
read or deleting it, the IMAP server
has to send you an update saying this mailbox
that you have opened has just changed and there's a new message in it,
or the last message is now deleted or whatever.
And so as you write all these functions that this one
finds all the mailboxes so it lists, does a LIST and then parses
the whole response. So you could do all that and
then you get a line back from the server
and it has nothing to do with your response. And it's one of these
status updates that you didn't ask for but
that you need to know about because you can't really let the user
click on a message to view it, if that message has just been deleted from
the mailbox. You need to respond to it as
an IMAP client, you need to do something about it quickly and then
update the GUI to take that message out of the mailbox or whatever.
So it's very annoying because you can't just write code that
says send a message to the server and then get the reply and parse it.
You have to get the reply from the server and then you have to make
sure in every function, "is this an update that I care about?"
And if it's not what I'm trying to parse
right now, stick it in a queue and then do
it when I'm done with this function or something like that because you
can't go off on a side tangent trying to respond
to that status update if you're in the middle of trying to parse
something. So that's basically what I had to implement was like a queue.
So every time when you do expect_line,
it tries to figure out if the
response that you just got is not what you need
and if it is, it sticks it in a queue and then tries to read
another line. So that this
thing here, so this kind of works.
But the other problem is that some of
the commands that you send to the server, the response
looks identical to one of these unsolicited status
updates. And so you can't always just read
this response and know that it is an unsolicited
status update. So sometimes you have to send it back to the function and
let the function parse it and see if it's something that it cares about.
So that's kind of annoying.
And the way that I'm kind of doing it is that before every
command that I send to the server I am
going into IMAP IDLE. And IMAP IDLE is useful
because you're in a state where the server can send you--
So like these status updates, they don't come just arbitrarily
over the connection. The IMAP server waits for
you to send it a command. The server will then respond
to that command and then at the end of it it may send you
these unsolicited updates or it may send them
before the server response or something like that.
So it's annoying. With IMAP IDLE you
basically send the IMAP server an IDLE command and
then the server knows that you are idle and you're waiting for
status updates. So the server will just send you these
status updates as soon as it gets them, which is useful
because you can be in that state and most of the updates that
you're going to get are they're
unsolicited but you're in an idle state so you know that
they're not responses to commands that you're sending. But because
there's a remote chance that server does send you a status
update in response to a command that you send in the future,
you can't always rely on that. And if you miss one of those status updates
and you don't respond to it correctly,
like if you have ten messages in the mailbox and another
client deletes the first one, every message
in that mailbox, they all have to get renumbered.
Like subtract one from each id. So if you
don't do that and you send it the server a request
to download message five and
it's actually been renumbered to number four, the server is then going to
send you the wrong message in response. So that's
bad. Anyway, so that's
how I'm doing it. So I'm basically like the server or the client is always
in an IMAP IDLE state and then, except when it needs to
send a command. So it exits the idle state, sends the command
and then reenters idle state all the time.
So as I showed, it can send now it can compose
a new message, it can reply and then when you reply it
has to pull out from address of
the original e-mail that you're viewing. But you
also have to have, if there's "to"
or "CC" addresses that are in there,
you're supposed to include those in your reply. So it has to be able
to parse all those out. And then another
thing that is helpful that the server can do like the mind parsing is that
if you have an e-mail header and it says like CC and there
are two addresses in there, the format of those addresses
can be very different. Like you can have
a name in quotes and then in brackets the
e-mail address and then comma and then you can just have an e-mail
address. And so that's two different formats, but you need to
know that that's not like three separate addresses because one of them is in quotes.
So that has to be the name. But you can also have a name that's
not in quotes if it doesn't have a space character in it. So basically it's
a very loose specification and you need to be able to parse all that.
So another thing that is helpful in an IMAP server is that you can
ask the server to parse that for you and it will give you all of
those broken out in a particular format
that is easy to parse from
code. So there's no ambiguity as far as
like well my parser is different than yours, you can just ask the server to
do it and you know that it's going to be correct
or at least consistent, I guess.
So with all of those things there's a lot
of code that I had to write that does a lot of parsing.
So this is like, this is that thing
that,
so like the thing I was talking about where you could have like three different
words in a subject that are all encoded different. This is
the RFC for it. This is a parser.
So one word will be encoded like this, and then
this says that it's UTF-8 encoded in quoted printable.
And then the value here is a quoted printable
word which can be multiple words. So you need
to look for this terminator and then you need to d
quote print the value in there and then it's
in UTF-8 encoding. But this can also be a
b for base 64 and this can be
US-ASCII or like ISO-8859.
And then you need to implement parsers for
all those. And then of course you have to do it the reverse
way when you're sending out e-mail.
So that was a lot of parsing and I wanted
to make sure that my parsers were correct. So I wrote
this test suite, which I thought was kind of neat because
the way that it works is you can just say you
want to define test suite and
now when you rerun the program it's going to recompile tests
and main.
And in main I have it so that if test
suite is defined it just runs, tests run and then exits
and it doesn't initialize the GUI or do anything else.
So this made it easy to iterate on something where
I needed to. I have the output that I need. So I
guess this is test driven development or whatever. I have the output that the
IMap server is sending me and I know what the parsing function
needs to spit out. And so I could
write a test and then run
this, the test would fail and then I could write a function and
iterate on that function until it got the parsing right.
And then as I would encounter a new thing, I could just add another
test case to each of these.
And then it could just blast through all these tests and then tell
me that everything is okay. And I know that I wouldn't be breaking anything
that I had already parsed correctly. So this was
pretty useful. I don't know
how useful this framework or whatever is
outside of this program, but I
feel like everything takes longer when I'm sitting here in front of the camera,
when I'm just developing this recompilation
doesn't feel like it takes that long.
So now when this runs, it's just going to go through all of these tests
and then tell me that it passed... hopefully.
Yeah. So that's cool.
As I would encounter something new, I would just do
like this and add another test case and then for the
more advanced ones.
So as I was saying about the parsing out the e-mail addresses,
that works when you get an e-mail that's on the IMAP server
and you want the IMAP server to parse it and tell you to break
them up properly, but you also need to implement
your own, because when you are composing an e-mail,
all those things that were broken out are just stuck into a text field,
like the to field or the CC field, and then the user might
go into that field and then just put a comma and then an e-mail address.
And so I guess I did need to end up
implementing my own version of that. I forgot about this.
So it needs to be able to parse this and know that it's
two separate addresses. But then also this, because these are
in quotes, these are two separate addresses,
and then these ones are bogus. And then this one
has quote inside of the quote that's escaped.
And so you need to be able to parse those anyway. So you can see
where all this parsing can get kind of tricky, and you need to
be able to be assured that you're not--
so these are what those quoted words look like,
and then every space that's supposed to be an underscore for
quoted printable, but you can just do like base64
encoding for each word,
and then each word can be encoded
differently, which I don't think any legitimate e-mail client
would do. It's probably a good indicator
that it's spam or something like trying to get around spam
rules. So these are the where
I was talking about having to parse a mime message from the
server where it spits out the structure
of it. Some of these can get really complicated. Like this one I was
actually just working on before the video, because I need it to view
an e-mail. So this one doesn't pass yet. And so maybe I
can work on
that on video, which I don't seem to
do anymore in these videos.
Then the other thing was like, when you reply to an e-mail, you want
to be able to quote the text in it, but you need to deal
with all the different new lines, because on Mac OS all
your new lines are just carriage
return. But e-mail, like the standard is supposed
to be carriage return line feed.
And you kind of need to be able to deal with e-mails that
are only line feed.
So I had to write a test for that.
That's the test suite. The other thing that is
kind of neat that I just came up with recently is
that I actually needed for subtext. Subtext had a memory leak in
it that I could not pin down and
I needed to be able to find it.
So I wrote a new malloc
logging tracing system.
The way that I had been trying to do it in some of my programs
before was I made. So I'm using
a wrapper called xmalloc in this util
c, and it tries to do the
equivalent on Mac OS of a malloc. And then
in an interactive program like Carl, if the malloc fails,
it would present a warning dialog saying memory allocation
failed. Then like panic or whatever.
In Subtext, because it's not interactive,
I needed to basically just return without showing a warning
dialog. And then every function that called
xmalloc and it returned null, I have to handle
that failure. And then I
have a wrapper called xfree, which is a wrapper
around the equivalent of free. And instead of passing
it a pointer, you pass it a pointer to the pointer. So I'll
just show you what that looks like.
So you xmalloc a length. And then when
you return it-- or when you free it,
you call xfree with a pointer to
the pointer.
And what xfree does is it frees scratch,
and then it nullifies the scratch pointer.
So when this returns, scratch is
null. And then that way you can more easily catch
like a use after free, because instead of freeing
something and then later trying to use scratch, scratch would point
to null, and it becomes very obvious that there's a
bug there because you're trying to do something on null rather than some random address
that scratch used to point to. And so
the way that I had been trying to do malloc tracing and logging before
was that I made a second argument to xmalloc,
and you would have to put in a string. And then when
you turned on malloc debugging,
when you call xmalloc, it would keep all
of those in a big array and you would know--
so later when you go to dump all the allocations, there would be that string
there that would say like, oh, this was from mailutil,
in whatever name, whatever function this is. So I could go
in and narrow it down, but it was very laborious because in every
xmalloc call you
have to include this string, and all those strings then have to accumulate
in the program text and in memory,
and I had to type them in each time. And sometimes I would have to
build a custom string to know something
very specific that wasn't helpful anyway.
So anyway, the way that I did that now is,
I don't know if this will work, but I'm going to.
So this is MacsBug. This is
the debugger that a lot of Mac users used back
in the day. The normal one that just shows you
a small window in the middle is called MicroBug,
and that's the one that is included with Mac OS.
And you get very limited functionality as far
as being able to dump memory or set arbitrary
memory. I use it a lot. If something crashes, you can break
into the debugger and then do SM 0 A9F4
which puts A9F4 in the memory
location zero. And then you do G 0 to jump to
that address. And A9F4 is the trap, the system
trap for ExitToShell. So it basically tells the program that
you're in to try and exit. And sometimes it works and sometimes it doesn't.
But anyway, if you install MacsBug,
when you hit the interrupt switch on the side or
something crashes, you now get this full screen thing.
So one of the commands you can do is SC,
which is a stack crawl, which is
not helpful in this case because there's no symbols.
But if you run like some other software
like bottom mine that has all those symbols in it,
you get something very useful. So if I just break into this now
and I do a stack crawl,
hopefully. Oh, because it's in block move.
Hold on just a second. It from,
so a compiler can put in symbols that
max bug can read, and it's like a,
it was a common convention that
a lot of compilers supported.
And so if you leave those symbols in there, when you
do a stack crawl from code that has those symbols, you get this nice
backtrace where you can see my mouse
doesn't work in MacsBug. It is in xmalloc
and it was called from repo_parse_amendment, from repo_load_amendments,
repo_init, repo_open, and main. So that's
basically what I wanted in my malloc tracing
function. I want to
be able to capture like this stack trace and then save it.
That way I don't have to put in a string that says repo_parse_amendment
and I get the full stack trace. So I know how it got there.
I couldn't find any code that implemented this and
I couldn't really find much talk on the old Usenet
groups as far as how this was implemented.
I did get a tip and I basically had to write the code myself.
But again, my mouse
doesn't work so I can't point to things. But the...
A6, that is an address register in the cpu,
and that says the
layout of each stack frame as code calls another function.
And then that calls another function is they all put these stack frames
on the stack and then that basically lets the,
it's like a format that it knows how
to get back to when you call return, where to go
back, and then like how it puts all your arguments on the stack and stuff.
So if you do...
I had some notes on this. So if you dump
the memory at the
first address in this memory, after the,
after this right here is.
And so A6
is currently pointing at-- it has a return address
of 00244018
And then that address.
If you dump the memory at that address, which I just did,
it's showing you the next address up in the stack where
it's going to go. And I
guess it kind of gives it away. So this address 0024404E
is right here. And if I dump that address, it's going to give me
this one and then that one and that one and that one. So if
you dump
the code at 24404E
that doesn't look right.
Right, so that's
the return address and this is the code that it's jumping
back to. So you can see at the top it's
in repo_parse_amendment. And then this is all the code for that
function, which is a lot. So to
find the symbol of what this function is from
in a programmatic way, you're supposed to dump all of this code
and then just look for the return. And after the
return,
this rts. So if we dump the memory,
you can see on the right hand side over here,
there's repo parse what ends up being amendment.
And so if you dump,
I think it's yeah,
so you get repo, parse, amend, and then you can keep reading until
you hit a null byte. So anyway, so think C.
You can tell it that every function at the return
to put that
string in memory at the end of each function
in the code segment.
And so if you just walk memory,
you can look for that RTS instruction like
the opcode for it in the actual bytes in memory,
and then walk to the end of that and you'll have a string that
is the name of the function that you were just in.
Anyway, I probably got some of that wrong,
but if you implement that
in code,
what you get is a way to be able to walk
all of those,
to walk the stack from
a function which in my case I'm doing it from xmalloc.
And you can get a list of each symbol along
the way, and then you stop at like main or whatever. So this
is a code to do it in xmalloc.
If malloc debugging is enabled, I am storing
the value of the A6 register in a variable
called _a6. And then
in a loop here I wrote this all
out, which I could have just read, I guess.
So it sits in a loop trying to walk back the stack until
it gets to main. So you have the return
address is A6 plus four.
Then the code is at that address that it's returning to.
And then from that point forward you walk forward in memory by two bytes
looking for an RTS instruction. Once you
find the RTS instruction, jump over
it, and then there will be the
name of the symbol right after it and it'll be null
terminated. If you have the
if you have MacBugs Names
in long format it's null terminated. And I think if you have it
in short format it's like hard coded to eight characters or something
like that.
And then here, like if the symbol is main or uthread begin,
just stop trying to walk back. And so that lets you get
a list of all these symbols, and then whatever you do with them is up
to you. And in my case I'm storing them in a
big symbol table. And so that avoids the duplication of having
all these separate strings,
So you have like a small set of strings, because there's only
so many functions in my code. And then the
way that it stores them all is it stores like a big array of
the stack as just pointers to elements
in that array of symbols, and then the address
and the size. And so in addition to being able
to get a stack trace of every allocation. You also
get something in xfree. So you're freeing
a pointer. You can then go look up all those allocations
and know that if you're freeing an address
that wasn't in that list of allocations, you're doing like a
double free, or you're trying to free some junk address.
And so that's helpful as well. But I
can only have that on if malloc debug is enabled, which kind of slows
things down. And then once you have this list of all
of the allocations, like in Subtext,
I made it so that they just dump all to the logger window. And now
because Subtext has syslog support, they all
get logged to my server, which makes it easier to
run through them all. And then
in Carl I have it dumping to
a text file that I can just view. So like this
is all running with malloc tracing on.
And if I view a message and
then dump all of the current allocations,
they are now in a file which I can look
at. And if I see stuff
in there that probably shouldn't be in there, I know that I am leaking something.
So if I open that file that I just dumped,
should get a list of every allocation. So you get the
size, and then this is the call chain of how it got there. So through
main browser in it, imap init. So these are probably
like the TCP buffers that it has
to allocate at the setup of the connection.
These are all small allocations in
list mailboxes. So these are probably allocating like
a mailbox struct.
And then in fetch envelopes, it's having to in
get envelope list. It is doing a
malloc allocation for every e-mail address that it comes across.
So obviously in Carl you're going to have tons of these tiny little
allocations. And you get a total.
So that's pretty cool. It slows down the program a little
bit. At least it's noticeable on my slow computer. But in
a production environment
you would just undefine that define and it would remove the
menu and all of the tracing that
has to walk the stack and all that. So I think that's pretty cool.
This is probably also useful for something else,
not just debugging malloc allocations,
but anyway, so all this
code is in util.c if you need it.
The code is on AmendHub if you need something to
walk a stack on
a 68K Mac. I'm sure it's different on PowerPC and whatever
else, but yeah.
So yeah, that's Carl, that's where I'm at. Like I said,
I can compose new e-mail, I can reply to existing
e-mails. Sometimes I guess it crashes, but it wasn't before.
And then it does all the little things. So as you view a message in
Carl, it has to then tell the server to mark it as
seen so that on another e-mail client
it's no longer new. And in Carl, if another e-mail
client does that, Carl would receive that out
of order unsolicited status update and then
it would update the message list to no
longer show that e-mail in bold. Oh yeah,
I don't think I showed that.
So I can show you with concurrent access
what it looks like. It doesn't update very smoothly
right now, but it should.
So I have on my laptop I'm
using mutt and I'm in that same folder, and if I mark a
message as unread,
um,
this one should turn bold as I sync it and it just did.
And then I can mark it as read and sync
it and it disappeared. So like this is what that status,
unsolicited status looks like. I'm just dumping
in here for debugging reasons, but you just get a nine fetch
flags, whatever.
But there's like a fetch command. So when you send an IMAP server a FETCH
command, it will respond with something identical to
this. But you need to know that
if you sent it a fetch
command for message 8 and you get back 9 FETCH,
you're supposed to know that that was an
unsolicited update for a different message and not what
you needed. But you could also theoretically get
this response on the message that
you're working with. The other thing is, so if you mark a message as new,
it should update this as bold too, because it
now has an unread message
in there. But it won't until it refreshes all of the mailboxes,
which it does periodically because it needs
to mark them as bold and whatever.
Something else that I implemented was this kind of truncation.
So like this actual mailbox name is openbsd-hackers. This one
is openbsd-ports, this is openbsd-ports-changes
and something that
an old Usenet client that I used to use, I think it was tin
did was when it wants to truncate--
in that case it was usenet groups. So like comp.unix.os.openbsd
or something like that,
when it didn't have enough space to show that it would truncate it as like
c.u.o.openbsd.
And I always liked the way that it did that and Mutt does not do
that with the sidebar code.
But I implemented it in Carl. So when it can't
display the
unread count next to the mailbox and it's like
if it would overflow this, it goes into this truncation
mode where it tries to truncate mailboxes
at a dash because I
think the right part of the mailbox is more helpful.
Like is it? Because otherwise it would just say like openbsd-po
and this would also say openbsd-po.
But being able to differentiate between ports and ports-changes is more.
Anyway, when this updated so that it's showing this in.
So that's Carl as it is right now.
You can compose new e-mail. I thought that this was funny.
It's the default signature if you used to have.
I don't think they still do it, but the default signature on the iPhone
was Ssent from my iPhone and everybody hated
it because it was like a way of showing off that you had
an iPhone back in the day when not everybody had an iPhone or an Android
phone. So I thought it would be funny to make that the default signature
in Carl. And the other thing is
it includes a User-Agent
header so that other e-mail clients can
see the user agent that
I sent this with.
And I don't have any examples
of it, but it says like Carl and then the version and then
it actually looks up what machine
you're on. So like to..
I can't remember what the name of it is.
It doesn't show it here.
Anyway. Mac OS knows what kind of machine you're on and it
would show like I thought it did show there. Maybe it was in later versions
it would say like that you're running on a Mac Plus or a Mac SE
or whatever. Mac Classic.
Maybe it's X-Mailer.
I think I'm doing that directly in the SMTP code.
Yeah, so it gestalt
I guess is what Apple calls it.
And so you get this code and then
you can return, I made it return a string. So anyway, so when you're,
it would be like the default signature would be like sent from my
Macintosh IIcx or something.
Anyway, I thought that was funny. Nobody's going to use this program except me anyway.
But I thought that was kind of funny. So if you send me an e-mail
and I reply and it says like Sent from my Macintosh Plus, I am actually
sending it from a Macintosh Plus.
So I'm going to continue working on Carl, fixing bugs and
making it faster and making
it so you can download attachments and maybe send
attachments, but that seems far
fetched. I mean, what am I going to send from my Mac
Plus that anybody would find useful.
So yeah, I'll keep working on that. I might do another video.
This one's already an hour long, so I can't show-- I was thinking
of maybe doing live coding or whatever.
Somebody asked me why my live streaming thing is no longer active on
my website and this is the reason why. Because when
I work on Carl, a lot of times I'm like going into my inbox and
viewing actual e-mails that are not in my staged
Carl mailbox and I don't want to
accidentally show something on the live streaming video
that is sensitive or an
e-mail that somebody sent me that I don't want displayed.
Or before I had the password masking stuff,
my password was showing up on the screen. So I had to
disable the live streaming. And I've been working in Carl for a couple of months
now, so I just took it down from my website. So anyway, that's the reason why.
I'll keep working on this and
maybe post another update sooner than later
and maybe write code, if anyone cares to even
see that anymore or if these updates are more useful because I
already have everything written and you don't have to watch me compile things.
Okay, so that's it. Thanks for watching. Goodbye.