2024-03-26T23:31:12-05:00https://jcs.org/rssjoshua steinhttps://jcs.org/2024/02/20/carlVideo: C Programming on System 6 - Carl Update, Test Suite, Malloc Tracing2024-02-20T00:00:00-05:00
<p>I've been working on Carl, my IMAP e-mail client, for the past few months.</p>
<video controls="1" playsinline="" preload="metadata" poster="/images/2024-02-20-thumbnail-980x551.jpg">
<source src="//jcsorg.b-cdn.net/videos/2024-02-20.mp4" type="video/mp4" />
<track default="" kind="captions" label="English" srclang="en" src="/2024/02/20/captions.vtt" />
<p>
Your browser doesn't seem to support HTML video.
You can download the video in
<a href="//jcsorg.b-cdn.net/videos/2024-02-20.mp4" download="">H.264/AAC</a>
format instead.
</p>
</video>
<p>I need to work on not saying "like" so much.</p>
<p>Video notes:</p>
<ul>
<li><a href="https://www.tindie.com/products/jcs/powerbook-1xx-battery/">Remaining PowerBook 1xx Batteries</a></li>
<li><a href="/2024/02/19/subtext4">Subtext 4</a></li>
<li><a href="https://social.jcs.org/@kludge">Kludge BBS on Mastodon</a></li>
<li><a href="https://datatracker.ietf.org/doc/rfc9051/">IMAP4rev2 rev2 RFC</a></li>
<li><a href="https://datatracker.ietf.org/doc/rfc2047/">Encoded-word RFC</a></li>
<li><a href="http://files.mac-attic.com/downloads/reference/rom-debugger/tn1136-rom-debugger.pdf">MicroBug</a>
and <code>SM 0 A9F4</code> and <code>G 0</code></li>
<li><a href="https://macgui.com/news/article.php?t=485">MacsBug</a></li>
<li><a href="https://amendhub.com/jcs/subtext/files/util.c#L185">My stack walking code</a></li>
</ul>
<p>Stack layout showing A6 links:</p>
<figure><a href="/images/2024-02-20-a6-1027x940.png"><img src="/images/2024-02-20-a6-335x306.png" srcset="/images/2024-02-20-a6-335x306.png 1x, /images/2024-02-20-a6-670x612@2x.png 2x" width="335" height="306" class="1/2" /></a></figure>
<p>Please
<a href="/contact">contact me</a>
with any feedback or questions,
<a href="/system6c">view past videos in this series</a>,
and
<a href="https://jcs.org/rss">subscribe to my RSS feed</a>
to be notified about future videos and other posts.</p>
<p>Join me and others on
<a href="//libera.chat/">Libera Chat</a>
in the
<a href="irc://irc.libera.chat/cyberpals"><code>#cyberpals</code></a>
channel if you are interested in following along with this series and have
questions or would like to help others.</p>
https://jcs.org/2024/02/19/subtext4Subtext 4.0 Released2024-02-19T00:00:00-05:00
<p><a href="/subtext">Subtext</a>
4.0 has been released:</p>
<ul>
<li><a href="/subtext/subtext-4.0.sit">subtext-4.0.sit</a>
(StuffIt 3 archive, includes
<a href="//amendhub.com/jcs/subtext">source code</a>
and THINK C 5 project file)<br />
SHA256: <code>c33a6abe15e7f071<wbr />59a18d936dce7c43<wbr />5d9619f80d6c53d1<wbr />cdd26fabf06712f7</code><br />SHA1: <code>0f4bdcd08b57b568<wbr />125d8ddcab5ad97f<wbr />1f5dd586</code></li>
<li><a href="/subtext/ipdb-2023-11.db">ipdb-2023-11.db</a>
(Free IP Geolocation from
<a href="https://db-ip.com/db/download/ip-to-city-lite">DB-IP</a>,
converted to Subtext IPdb format)</li>
</ul>
<!-- SNIP -->
<p>Changes in this version:</p>
<ul>
<li>Move views out of database to flat files in a "views" directory, allowing them
to be edited by other text editors and backed up; views are cached at startup
and can be reloaded through the sysop menu</li>
<li>Speed up telnet IAC negotiation to avoid delay after connect, avoid doing
<code>NEWENV</code> unless talking to the trusted proxy</li>
<li>Add BSD Syslog (RFC3164) support, sending all logged events to a configured
syslog server</li>
<li>When opening the last-opened database at startup, if the Command key is held
down, do not open it and prompt for a different one.</li>
<li>Add support for <code>{{center_XX}}</code> template variable</li>
<li>Remove <code>ATQ0</code> from default modem init string to be able to handle when the
modem responds to init with an error</li>
<li>When tossing FTN packets, log failures and move files to the "bad" directory,
then send local mail to sysop</li>
<li>Add page number to prompt in boards menus</li>
<li>Fix <code>new</code>/<code>signup</code> logins not going directly to the signup menu</li>
<li>Fix bug in modem handling where a session wasn't properly disassociated when
the session was forcefully closed</li>
<li>Fix location lookup when connecting through the trusted proxy</li>
</ul>
https://jcs.org/2024/01/29/wifi_da11BlueSCSI Wi-Fi Desk Accessory 1.1 Released2024-01-29T00:00:00-05:00
<p><a href="/bluescsi">BlueSCSI Wi-Fi Desk Accessory</a>
1.1 has been released:</p>
<ul>
<li><a href="/bluescsi/wifi_da-1.1.sit">wifi_da-1.1.sit</a>
(StuffIt 3 archive)<br />
SHA256: <code>db06d941d7322089<wbr />d8a98ebabf1f9a50<wbr />b8f75ade80edbb35<wbr />5c127acc9a7fa629</code><br />SHA1: <code>c9d6d3d165626ce5<wbr />92a65fd9e2011f36<wbr />1a0f78e4</code></li>
</ul>
<!-- SNIP -->
<p>Changes in this version:</p>
<ul>
<li>Fix masked password entry on Mac OS 8+</li>
<li>Truncate display of long Wi-Fi network names</li>
</ul>
https://jcs.org/2023/11/21/subtext31Subtext 3.1 Released2023-11-21T00:00:00-05:00
<p><a href="/subtext">Subtext</a>
3.1 has been released:</p>
<ul>
<li><a href="/subtext/subtext-3.1.sit">subtext-3.1.sit</a>
(StuffIt 3 archive, includes
<a href="//amendhub.com/jcs/subtext">source code</a>
and THINK C 5 project file)<br />
SHA256: <code>ef127a83e68918ff<wbr />7f57e34c40a539f0<wbr />6bbe026943ac5f55<wbr />be972c0e4518d0d0</code><br />SHA1: <code>962397af3fd8e472<wbr />2a5384ae2634ca86<wbr />e2b447f1</code></li>
<li><a href="/subtext/ipdb-2023-11.db">ipdb-2023-11.db</a>
(Free IP Geolocation from
<a href="https://db-ip.com/db/download/ip-to-city-lite">DB-IP</a>,
converted to Subtext IPdb format)</li>
</ul>
<!-- SNIP -->
<p>Changes in this version:</p>
<ul>
<li>In "who" command, show which area of the BBS each user is currently in</li>
<li>Add a "logged_in_time" template variable, helpful for signoff template
showing how long the user was logged in for</li>
<li>Find proper per-message originating and destination net/zone numbers
from FTN messages when different than FTN packet header</li>
<li>Properly handle skip commands received from binkp servers to end the
session and retry later</li>
<li>Abort idle binkp connections in case of server issues</li>
<li>Set the software major version in outgoing FTN packets and use the
proper code for software that has not been assigned an id such as Subtext</li>
<li>Include the actual timestamp of FTN packet files when sending through
binkp</li>
<li>Fix a bug encoding FTN packets with the wrong termination</li>
<li>Fix a bug where tossing FTN packets was adding messages for areas
not subscribed to into the previously used board</li>
<li>Fix a bug in the console where an assertion was erroneously triggered
when the cursor was at the far corner but in bounds</li>
</ul>
https://jcs.org/2023/11/11/subtext3Subtext 3.0 Released2023-11-11T00:00:00-05:00
<p><a href="/subtext">Subtext</a>
3.0 has been released:</p>
<ul>
<li><a href="/subtext/subtext-3.0.sit">subtext-3.0.sit</a>
(StuffIt 3 archive, includes
<a href="//amendhub.com/jcs/subtext">source code</a>
and THINK C 5 project file)<br />
SHA256: <code>713f9b465281f894<wbr />05522f7aef23d4fa<wbr />2bf035c7cd070f69<wbr />0395302a4a77045a</code><br />SHA1: <code>47ba8aa40e2abcd3<wbr />811c3f26b349a15a<wbr />78bcd956</code></li>
<li><a href="/subtext/ipdb-2023-11.db">ipdb-2023-11.db</a>
(Free IP Geolocation from
<a href="https://db-ip.com/db/download/ip-to-city-lite">DB-IP</a>,
converted to Subtext IPdb format)</li>
</ul>
<!-- SNIP -->
<p>Changes in this version:</p>
<ul>
<li>Add support for using an IP geolocation database to lookup connecting
telnet IP addresses and log their location, storing it in the session
log and making it available in 'who' and 'last' menus</li>
<li>Dynamically apply config changes made from the sysop menu without
requiring a restart</li>
<li>Add a config setting to change the number of rings to wait before
answering the modem, possibly to allow for caller ID information to
come through after the first ring which will get logged</li>
<li>Add '&D2' (hangup on DTR flash) to default modem initialization string
to make it clear we need it for proper hangup</li>
<li>Do faster sorting in many places</li>
<li>Allow "guest", "new", and "signup" logins to be made in mixed-case</li>
<li>Add 'l' key in post view in boards to list posts, and 's' to show the post
again</li>
<li>In sysop settings menu, make 'q' stop pagination at the 'more' prompt</li>
<li>Fix a bug in sysop settings editor that could have caused config
corruption if the sesion was disconnected before exiting/saving the
editor</li>
<li>Do more TCP connection status checks during binkp connections to detect
dropped connections</li>
<li>Fix a bug in pagination that caused extra newlines to be shown in certain
terminals, and a bug that caused incorrect wrapping when the first
character of a post was a newline</li>
<li>Fix a bug in boards that was causing '>' not to work in some cases</li>
<li>Fix text selection and copying in the logger window</li>
</ul>
https://jcs.org/2023/11/06/wikipediaWikipedia Reader 1.1 Released2023-11-06T00:00:00-05:00
<p><a href="/wikipedia">Wikipedia Reader</a>
1.1 has been released:</p>
<ul>
<li><a href="/wikipedia/wikipedia-1.1.sit">wikipedia-1.1.sit</a>
(StuffIt 3 archive, includes
<a href="//amendhub.com/jcs/wikipedia">source code</a>
and THINK C 5 project file)<br />
SHA256: <code>14761cefe2552886<wbr />476cdf018445d445<wbr />4f355fca31636af5<wbr />d55c52e7b28908da</code><br />SHA1: <code>acd1b9433113237d<wbr />0fd5e15fba6a4c4b<wbr />992be783</code></li>
</ul>
<p>Changes in this version:</p>
<ul>
<li>Add support for opening multiple windows</li>
<li>When clicking a hyperlink with the Command key down, open the link in
a new window</li>
<li>Try allocating more memory to resize internal buffers when parsing
rather than just giving up right away</li>
</ul>
https://jcs.org/2023/10/25/wifi_daBlueSCSI Wi-Fi Desk Accessory 1.0 Released2023-10-25T00:00:00-05:00
<p><a href="/bluescsi">BlueSCSI Wi-Fi Desk Accessory</a>
1.0 has been released:</p>
<ul>
<li><a href="/bluescsi/wifi_da-1.0.sit">wifi_da-1.0.sit</a>
(StuffIt 3 archive)<br />
SHA256: <code>ccfc9d27dd5da741<wbr />2d10cef73b81119a<wbr />1fec3848e4d1d88f<wbr />f652a07ffdc6a69a</code><br />SHA1: <code>ff124972f202ceda<wbr />6d7fa4788110a67c<wbr />cda6a13a</code></li>
</ul>
<p>This is the initial public release of my BlueSCSI Wi-Fi Desk Accessory for
classic MacOS.</p>
https://jcs.org/2023/10/17/wikipediaWikipedia Reader 1.0 Released2023-10-17T00:00:00-05:00
<p><a href="/wikipedia">Wikipedia Reader</a>
1.0 has been released:</p>
<ul>
<li><a href="/wikipedia/wikipedia-1.0.sit">wikipedia-1.0.sit</a>
(StuffIt 3 archive, includes
<a href="//amendhub.com/jcs/wikipedia">source code</a>
and THINK C 5 project file)<br />
SHA256: <code>360e12d064f65796<wbr />95f1e627ce34cb2f<wbr />09347a5f86a5fc9f<wbr />553df7763829265</code><br />SHA1: <code>b7761c15c591fcf3<wbr />47edb7026918b794<wbr />313ed2bc</code></li>
</ul>
<p>This is the initial release of my Wikipedia reader for classic MacOS.</p>
https://jcs.org/2023/09/20/vcfmwVideo: C Programming on System 6 - VCF Midwest 18, Subtext FidoNet Support, Wi-Fi DA, PowerBook Batteries2023-09-20T00:00:00-05:00
<p>I attended the
<a href="http://vcfmw.org/">Vintage Computer Festival Midwest 18</a>
and made some things.</p>
<video controls="1" playsinline="" preload="metadata" poster="/images/2023-09-20-thumbnail-980x551.jpg">
<source src="//jcsorg.b-cdn.net/videos/2023-09-20.mp4" type="video/mp4" />
<track default="" kind="captions" label="English" srclang="en" src="/2023/09/20/captions.vtt" />
<p>
Your browser doesn't seem to support HTML video.
You can download the video in
<a href="//jcsorg.b-cdn.net/videos/2023-09-20.mp4" download="">H.264/AAC</a>
format instead.
</p>
</video>
<p>Video notes:</p>
<ul>
<li><a href="http://vcfmw.org/">Vintage Computer Festival Midwest 18</a></li>
<li><a href="/notes/2023/09/08/111031249117616588">My table</a>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Macintosh_Portable">Macintosh Portable</a></li>
<li><a href="https://en.wikipedia.org/wiki/PowerBook_100">PowerBook 100</a></li>
<li><a href="https://en.wikipedia.org/wiki/Outbound_Systems">Outbound Notebook 2030</a></li>
<li><a href="https://en.wikipedia.org/wiki/PowerBook_180">PowerBook 180</a></li>
<li><a href="https://en.wikipedia.org/wiki/StyleWriter#StyleWriter_II">StyleWriter II</a></li>
</ul>
</li>
<li><a href="https://bluescsi.com/">BlueSCSI</a>
<ul>
<li><a href="https://bluescsi.com/docs/WiFi-DaynaPORT">Wi-Fi Beta</a></li>
<li><a href="/2023/08/23/scsiwifi">Adding Wi-Fi to the Macintosh Portable</a></li>
</ul>
</li>
<li><a href="/2022/11/22/subtext">Subtext 1.0</a></li>
<li><a href="/2023/05/01/subtext2">Subtext 2.0 with FidoNet Support</a></li>
<li><a href="/wallops">Wallops</a></li>
<li><a href="https://www.tindie.com/products/jcs/powerbook-1xx-battery/">My new PowerBook 1xx Battery</a></li>
<li><a href="https://www.ebay.com/itm/313567626594">New PowerBook 100 Battery</a> and
<a href="https://www.ebay.com/itm/314777505347">New Mac Portable Battery</a> both by
alaska360</li>
</ul>
<p>Please
<a href="/contact">contact me</a>
with any feedback or questions,
<a href="/system6c">view past videos in this series</a>,
and
<a href="https://jcs.org/rss">subscribe to my RSS feed</a>
to be notified about future videos and other posts.</p>
<p>Join me and others on
<a href="//libera.chat/">Libera Chat</a>
in the
<a href="irc://irc.libera.chat/cyberpals"><code>#cyberpals</code></a>
channel if you are interested in following along with this series and have
questions or would like to help others.</p>
https://jcs.org/2023/08/23/scsiwifiAdding Wi-Fi to the Macintosh Portable2023-08-23T00:00:00-05:00
<p>Over the past year or so, I've been working with other
<a href="https://scsi.blue/">BlueSCSI</a>
developers to add Wi-Fi functionality to their open-hardware SCSI device,
enabling Wi-Fi support for old Macs and other vintage computers going back some
36 years.</p>
<figure><img src="/images/2023-08-23-portable-680x432.jpg" srcset="/images/2023-08-23-portable-680x432.jpg 1x, /images/2023-08-23-portable-1360x864@2x.jpg 2x" width="680" height="432" class="wide" /></figure>
<!-- SNIP -->
<div class="toc_wrapper">
<h2 class="no_toc" id="table-of-contents">Table of Contents</h2>
<ol start="0" class="top_link"><li><a href="#">Top</a></li></ol>
<ol id="markdown-toc">
<li><a href="#scsi2sd-rascsi-piscsi-zuluscsi-and-bluescsi" id="markdown-toc-scsi2sd-rascsi-piscsi-zuluscsi-and-bluescsi">SCSI2SD, RaSCSI, PiSCSI, ZuluSCSI, and BlueSCSI</a></li>
<li><a href="#daynaport-ethernet" id="markdown-toc-daynaport-ethernet">DaynaPORT Ethernet</a></li>
<li><a href="#vcfmw" id="markdown-toc-vcfmw">VCFMW</a></li>
<li><a href="#picoprobe" id="markdown-toc-picoprobe">picoprobe</a></li>
<li><a href="#pico-pio-and-spi" id="markdown-toc-pico-pio-and-spi">Pico PIO and SPI</a></li>
<li><a href="#scsi-debugging" id="markdown-toc-scsi-debugging">SCSI Debugging</a></li>
<li><a href="#wi-fi-debugging" id="markdown-toc-wi-fi-debugging">Wi-Fi Debugging</a></li>
<li><a href="#mac-plus-and-portable" id="markdown-toc-mac-plus-and-portable">Mac Plus and Portable</a></li>
<li><a href="#wi-fi-da" id="markdown-toc-wi-fi-da">Wi-Fi DA</a></li>
<li><a href="#ethertalk" id="markdown-toc-ethertalk">EtherTalk</a></li>
<li><a href="#performance" id="markdown-toc-performance">Performance</a></li>
<li><a href="#release" id="markdown-toc-release">Release</a></li>
</ol>
</div>
<p>This is my Macintosh Portable M5126.
It's very Macintosh and barely portable.
For some reason I'm using it on my lawn reading the
<a href="https://en.wikipedia.org/wiki/Wi-Fi">Wi-Fi Wikipedia article</a>
over Wi-Fi through my
<a href="/2022/09/29/vcfmw">Wikipedia application</a>
for System 6, with my Wi-Fi Desk Accessory showing it connected to my "!"
network with meager signal strength.</p>
<p>This article was originally supposed to be titled
"<em>Adding Wi-Fi to a
<a href="https://en.wikipedia.org/wiki/PowerBook_100">PowerBook 100</a></em>"
since that laptop is much more reasonable to lug around, but its logic board
died while working on this and I'm in the process of bringing it back to life.</p>
<h2 id="scsi2sd-rascsi-piscsi-zuluscsi-and-bluescsi">SCSI2SD, RaSCSI, PiSCSI, ZuluSCSI, and BlueSCSI</h2>
<p>For some background, there's a handful of hardware devices that simulate SCSI
disks for vintage Macintosh, Amiga, and Sun computers (and
<a href="https://www.reddit.com/r/Samplers/comments/llzkjw/adding_scsi2sd_to_an_akai_cd3000xl/">synthesizers/samplers</a>)
and can replace old, loud, power-hungry, unreliable,
<a href="https://en.wikipedia.org/wiki/Hard_Disk_20SC">low-capacity</a>
spinning hard drives with solid-state SD card media similar to
<a href="https://amzn.to/3OvEykT">CompactFlash IDE adapters</a>
for old PCs.</p>
<p>There's a bit of history surrounding these projects, but the first one created
in 2013,
<a href="http://www.codesrc.com/mediawiki/index.php?title=SCSI2SD">SCSI2SD</a>,
is a commercial product coming in various form factors such as
<a href="https://store.rabbitholecomputing.com/SCSI2SD-V5-2-p/scsi2sd-v5.2.htm">internal
50-pin</a>,
<a href="https://store.rabbitholecomputing.com/ZuluSCSI-V1-1-2-5-inch-edition-laptops-powerbook-p/zuluscsi-v1.1-2.5-inch.htm">PowerBook</a>,
and
<a href="https://samplerzone.com/products/scsi2sd-v5-5-plug-in-drive">external DB25</a>.
SCSI2SD requires proprietary software to configure the disks it presents, and
then requires a low-level tool like <code>dd</code> to write a disk image at a particular
offset on the SD card.</p>
<p>In 2017, RaSCSI, since renamed to
<a href="https://github.com/PiSCSI/piscsi">PiSCSI</a>,
did this with a
<a href="https://www.tindie.com/products/landogriffin/piscsi-full-size-assembled/">board</a>
attached to a Raspberry Pi or
<a href="https://www.tindie.com/products/landogriffin/piscsi-zero/">Pi Zero</a>,
but it requires a full Linux OS to serve up everything which makes it kind of
overkill in my opinion, as it takes a while to boot all of that to serve a
computer that wants its hard drive available within a few seconds.</p>
<p>In 2018,
<a href="https://github.com/ztto/ArdSCSino-stm32">ArdSCSino-stm32</a>,
later forked to
<a href="https://scsi.blue/">BlueSCSI</a>,
did this using a
<a href="https://stm32-base.org/boards/STM32F103C8T6-Blue-Pill.html">Blue Pill STM32</a>
Arduino, and improved on SCSI2SD by using a regular FAT filesystem on its SD
card so disk images could be configured just by copying them to the card and
giving them particular filenames (much like the
<a href="http://isostick.com/">isostick</a>
did back in 2012) such as <code>HD4-macplus.hda</code> to present that file as a hard disk
on SCSI ID 4.</p>
<p>In response to the global chip shortage in 2022, the company that created
SCSI2SD created a new product called
<a href="https://68kmla.org/bb/index.php?threads/announcing-zuluscsi-a-file-based-scsi-device-emulator.40749/">ZuluSCSI</a>
which forked BlueSCSI's firmware but was then entirely rewritten.
Then in 2023 in a round-about fashion, BlueSCSI forked the rewritten ZuluSCSI
firmware and created a new
<a href="https://bluescsi.com/v2">BlueSCSI v2</a>
board which uses the
<a href="https://www.raspberrypi.com/products/raspberry-pi-pico/">$4 Raspberry Pi Pico</a>
RP2040 microcontroller.</p>
<h2 id="daynaport-ethernet">DaynaPORT Ethernet</h2>
<p>In addition to SCSI hard disks and Zip drives, on my Macintosh Plus workstation
and
<a href="/2022/07/15/kludge">BBS</a>
computers I use DaynaPORT SCSI/Link-3 SCSI ethernet devices which add 10baseT
networking over SCSI.
I also use a Dayna Pocket SCSI/Link for my PowerBook 100/180 which works the
same way and is powered through the ADB port.</p>
<p>TCP/IP applications on System 6 on the Mac are usually written to use the
<a href="http://macintoshgarden.org/apps/mactcp-206">MacTCP</a>
driver.
MacTCP then talks to another driver for the particular network hardware to
actually exchange packets, such as
<a href="/2020/08/14/macsnap">MacPPP</a>
which talks to serial modems, or in this case, the
<a href="http://macintoshgarden.org/apps/daynaport-drivers">DaynaPORT SCSI driver</a>.</p>
<figure><a href="/images/2023-08-23-daynas-1500x1125.jpg"><img src="/images/2023-08-23-daynas-335x251.jpg" srcset="/images/2023-08-23-daynas-335x251.jpg 1x, /images/2023-08-23-daynas-670x502@2x.jpg 2x" width="335" height="251" class="1/2" /></a><figcaption>Two DaynaPORT SCSI/Link-3 devices</figcaption></figure>
<figure><a href="/images/2023-08-23-pocket-1500x1125.jpg"><img src="/images/2023-08-23-pocket-335x251.jpg" srcset="/images/2023-08-23-pocket-335x251.jpg 1x, /images/2023-08-23-pocket-670x502@2x.jpg 2x" width="335" height="251" class="1/2" /></a><figcaption>DaynaPORT SCSI/Link Pocket</figcaption></figure>
<p>In 2021, the RaSCSI/PiSCSI project gained
<a href="https://github.com/PiSCSI/piscsi/commit/788b2a3b46e7795c94fbd2f6dd980caa8ecddf0b">emulation</a>
of these Dayna ethernet devices which use a
<a href="https://github.com/PiSCSI/piscsi/wiki/Dayna-Port-Command-Set">simple</a>
SCSI command set for exchanging packets.
Since PiSCSI runs on a full Raspberry Pi, it's easy to use its native ethernet
or Wi-Fi interface to deal with network traffic and then send it over SCSI
through the PiSCSI software (or at least as easy as anything is on Linux).</p>
<p>Last year I
<a href="https://github.com/jcs/pce/commit/e73c215ccf60354f2cc353111140901a691e1f2c">implemented</a>
similar emulation in the
<a href="http://www.hampa.ch/pce/">PCE emulator</a>.
I don't usually use classic Mac OS in an emulator, but I found it useful to have
a System 6 environment on my laptop when I was away from my Mac Plus
workstation and wanted proper networking.</p>
<h2 id="vcfmw">VCFMW</h2>
<p>Around this time last year, I started talking to
<a href="https://mastodon.social/@nulleric">Eric Helgeson</a>
of the BlueSCSI project who was looking to make a
<a href="https://github.com/erichelgeson/BlueSCSI-Toolbox">classic Mac app</a>
that could transfer files from a classic Mac to the BlueSCSI itself which would
then transfer to the FAT filesystem on the SD card, rather than into the virtual
disk contained on the SD card.
This way a new hard disk or CD-ROM image could be uploaded from a Mac which
would then show up as a new drive to Mac OS without having to potentially
disassemble the computer the BlueSCSI is in to remove its SD card.</p>
<p>After helping Eric with some
<a href="/system6c">classic Mac OS programming</a>,
we met at the
<a href="http://vcfmw.org/">Vintage Computer Festival Midwest</a>
last September and I learned about BlueSCSI's upcoming v2 design which was set
to use the Raspberry Pi Pico.
I told him that if it could use the newly released
<a href="https://www.raspberrypi.com/news/raspberry-pi-pico-w-your-6-iot-platform/">Pico
W</a>,
I would be interested in writing the BlueSCSI firmware code to emulate a
DaynaPORT ethernet device via its Wi-Fi.</p>
<p>To be clear, this wouldn't be like many
<a href="/wifistation">ESP8266 projects</a>
that handle all of the TCP/IP and present some simplified interface to their
host computers, but would be more of a media converter just connecting to Wi-Fi
and presenting an ethernet interface to the host computer which would handle
TCP/IP itself.</p>
<p>Once the BlueSCSI v2 design was finalized, Eric and
<a href="https://hachyderm.io/@TechByAndroda">Androda</a>
sent me a prototype board and I got to work.</p>
<h2 id="picoprobe">picoprobe</h2>
<p>After fumbling around with the
<a href="https://platformio.org/">PlatformIO</a>
environment in Visual Studio Code (two things new to me) needed for BlueSCSI, I
was able to build and test the firmware on my test device but was limited to
<code>printf</code>-style debugging.
I then learned about
<a href="https://github.com/raspberrypi/picoprobe">picoprobe</a>
which is special firmware running on another Pico that communicates with the
target Pico over its debug pins, which then allows stepping through code and
inspecting variables inside of the PlatformIO interface.
(I later learned there is a
<a href="https://www.adafruit.com/product/5544">Pico WH</a>
model that breaks out these debug pins in a nicer way.)</p>
<figure><a href="/images/2023-08-23-picoprobe-1280x687.jpg"><img src="/images/2023-08-23-picoprobe-680x364.jpg" srcset="/images/2023-08-23-picoprobe-680x364.jpg 1x, /images/2023-08-23-picoprobe-1360x728@2x.jpg 2x" width="680" height="364" class="wide" /></a><figcaption>picoprobe (left) connected to BlueSCSI</figcaption></figure>
<h2 id="pico-pio-and-spi">Pico PIO and SPI</h2>
<p>To perform SDIO traffic as fast as possible, the BlueSCSI firmware offloads SPI
traffic to the RP2040's
<a href="https://blues.io/blog/raspberry-pi-pico-pio/">Programmable I/O (PIO)</a>.
Unfortunately the
<a href="https://github.com/raspberrypi/pico-sdk/tree/master/src/rp2_common/pico_cyw43_driver">Wi-Fi chip</a>
of the Pico W also uses the same PIO instance so once Wi-Fi was activated, SDIO
broke.
Getting everything cooperating together required Androda to do some extensive
debugging and while waiting for that, I started on the SCSI code in the firmware
with the SD card support disabled so at least Wi-Fi and SCSI could work
together.</p>
<p>Since I had already written
<a href="https://github.com/jcs/pce/commits/master">code in PCE</a>
for the DaynaPORT, it was just a matter of adapting it to BlueSCSI's code and
making it configurable like hard disks are by creating a file on the SD card
like <code>NE3.txt</code> to add a network interface on SCSI id 3.
By May of this year, I had a virtual device presenting to the Mac and recognized
by Danya's System 6 driver and utilities:</p>
<figure><a href="/images/2023-08-23-diagnostics-903x296.jpg"><img src="/images/2023-08-23-diagnostics-680x222.jpg" srcset="/images/2023-08-23-diagnostics-680x222.jpg 1x, /images/2023-08-23-diagnostics-1360x444@2x.jpg 2x" width="680" height="222" class="wide" /></a><figcaption>DaynaPORT Diagnostics Utility showing a BlueSCSI</figcaption></figure>
<p>One of the commands that the Dayna driver sends to the DaynaPORT early on is to
ask what its MAC address is, and this has to have Dayna's vendor prefix of
<code>00:80:19</code>.
Since the Pico W would just be converting layer 2 traffic from Wi-Fi to SCSI,
this meant either changing the Pico's MAC address to match the virtual Dayna
device, or rewrite every outbound Wi-Fi packet to have the Pico's MAC and
rewrite inbound packets to replace it with the Dayna MAC.
Since I wasn't sure where that Dayna MAC might show up inside of any packets, I
chose the former.
Unfortunately being able to change the Pico's MAC address required recompiling
the upstream driver library, which meant BlueSCSI now has to maintain our own
Arduino-Pico and Pico-SDK trees with this pre-compiled library change.</p>
<h2 id="scsi-debugging">SCSI Debugging</h2>
<p>Once packets started flowing in from Wi-Fi and in from the Mac, I ran into an
issue where packets going out to the Mac through SCSI were getting dropped,
either by the DaynaPORT driver or MacTCP.
This took a while to track down because everything I was seeing was showing the
bytes being sent over SCSI were identical to what I received over the air, but
the Mac was just ignoring them.
I used PCE to show that the packets being sent to an emulated system was
identical as well, and it was being processed by the virtual Mac just fine.</p>
<p>I eventually purchased this old Ancot DSC-202F SCSI analyzer on eBay which has a
command-line interface over a serial port.</p>
<figure><a href="/images/2023-08-23-analyzer-1500x836.jpg"><img src="/images/2023-08-23-analyzer-680x379.jpg" srcset="/images/2023-08-23-analyzer-680x379.jpg 1x, /images/2023-08-23-analyzer-1360x758@2x.jpg 2x" width="680" height="379" class="wide" /></a><figcaption>Ancot DSC-202F SCSI Analyzer hooked up to a BlueSCSI</figcaption></figure>
<p>The Ancot allowed me to watch the SCSI bus traffic and dump the commands and
data to verify that things were leaving the BlueSCSI properly.
I compared traffic from the BlueSCSI to what my real DaynaPORT device sent and
it looked identical, which was helpful but also frustrating.</p>
<pre><code>00B47: Bus free
00B48: Arbitration /80 (7)
00B4B: Select /88 (3,7)
00B4F: Command /08 (Read/Receive)00 00 05 F4 C0
00B57: Data-In /00 66 00 00 00 00 00 80 19 10 0B E9 00 E0 67 1F
00B67: CB 67 08 00 45 00 00 54 DF D9 00 00 FF 01 58 75
00B77: C0 A8 01 01 C0 A8 01 08 08 00 DC FD 80 46 00 01
00B87: A6 27 78 A7 EC 66 29 B3 39 EE 62 17 68 24 21 CC
00B97: 0C 02 FD 06 E7 9B DC B3 18 19 1A 1B 1C 1D 1E 1F
00BA7: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F
00BB7: 30 31 32 33 34 35 36 37 84 32 6C 6C
00BC4: Status /00 (Good)
00BC6: Message-In /00 (Cmd Cmplt)
00BC8: Bus free
</code></pre>
<p>Eventually Androda figured out
<a href="https://github.com/BlueSCSI/pico-sdk-internal/compare/a84619a9fc827f13dadf011630f5ca496cf0af65..6fd26c4d87c2bbbc85b9d5972c4c51dba60499f5">what was required</a>
to get the PIO for SCSI, SDIO, and Wi-Fi all working at the same time and just
when I was growing weary of debugging the SCSI issue, Eric pointed out that the
PiSCSI code had something my new BlueSCSI code didn't:</p>
<blockquote class="code">
[10:29] erichelgeson: i wonder if it's sending the reponse too fast, piscsi has a send delay<br />
[10:30] erichelgeson: <a href="https://github.com/PiSCSI/piscsi/blob/048444ce017de6b3c8e03eb80b023d47a62e6d5a/src/raspberrypi/devices/scsi_daynaport.cpp#L599">[...]/scsi_daynaport.cpp#L599</a><br />
[10:30] Androda: Looks like that's set to 2 + 4 (6)<br />
[10:33] jcs: i'll give it a shot<br />
[10:41] jcs: dammit<br />
[10:41] jcs: that was it
</blockquote>
<p>Apparently the BlueSCSI was sending traffic to the Mac too fast and the driver
needed time to process the incoming packet after receiving its size, but before
receiving the actual packet data.
My code for PCE didn't have any delay and seemed to work fine, which was
strange.</p>
<p>With all of that in place, I was finally able to join my <code>#cyberpals</code> IRC
channel from my Macintosh Plus with its native TCP/IP stack, over Wi-Fi, through
my
<a href="/wallops">Wallops</a>
IRC client.
Though of note, this was booted with its SCSI hard disk on a SCSI2SD, not (yet)
on the BlueSCSI which was providing Wi-Fi.</p>
<figure><a href="/images/2023-08-23-irc-1500x813.jpg"><img src="/images/2023-08-23-irc-680x368.jpg" srcset="/images/2023-08-23-irc-680x368.jpg 1x, /images/2023-08-23-irc-1360x736@2x.jpg 2x" width="680" height="368" class="wide" /></a><figcaption>Wallops IRC client on a Macintosh Plus</figcaption></figure>
<p>Something I noticed while briefly testing it was that during large
<a href="https://macintoshgarden.org/apps/fetch-212">Fetch</a>
downloads, it would pause once in a while but then catch up.
I didn't give it much thought, but it ended up being another show-stopper.</p>
<h2 id="wi-fi-debugging">Wi-Fi Debugging</h2>
<p>By this point in June, I needed a break and the project got "paged out" of my
desk since I generally only have
<a href="/uses">room</a>
for one big project at a time.
The other BlueSCSI developers were testing it on newer, faster Macs and seeing
the same pauses in traffic but it was much more frequent there.</p>
<p>In August, Eric wanted to release a beta BlueSCSI firmware with the new Wi-Fi
code, in hopes that someone could devote more time to finding the cause of the
traffic issues.
This was enough motivation for me to "page the project back in" and try to
figure out the cause.</p>
<blockquote class="code">
[02:00] jcs: i spent some more time debugging the wifi situation and found something, are you still set on releasing it on friday @erichelgeson?<br />
[08:00] erichelgeson: I can hold, what did you find?<br />
[08:06] jcs: packets are dropping during wifi processing<br />
[08:07] jcs: not the scsi injection<br />
[08:53] Androda: Is it a raspberry pi framework issue?<br />
[15:59] jcs: ugh, found it<br />
[15:59] jcs: and now i get sustained 60kb/sec via fetch
</blockquote>
<p>While using the excellent
<a href="https://hpd.gasmi.net/">Hex Packet Decoder</a>
on packets printed out from the BlueSCSI's console, I noticed that packets were
coming in through Wi-Fi properly, getting added to the ring buffer, but then
getting corrupted before they were being sent out through SCSI.
This turned out to be a dumb mistake I made when cleaning up the code to use
<code>#define</code> constants which ended up reversing the dimensions of the ring buffer
so it became an array of <code>packets[packet_size][queue_size]</code> instead of
<code>packets[queue_size][packet_size]</code>.</p>
<p>When storing packets, instead of properly writing to <code>packet[1]</code> which should
have been <code>packet_size</code> (1520) bytes offset from <code>packet[0]</code>, it was writing at
offset <code>queue_size</code> (30), in the middle of <code>packet[0]</code>.
This didn't matter if <code>packet[0]</code> had already been transmitted to the Mac, but
on larger transfers where the queue has multiple incoming packets to transfer,
the current packet has the next packet written to the middle of it.
The delay and restart seen was the TCP stream not getting an expected packet
because it was corrupted, the sending side not getting an ACK for it, and then
having to timeout and re-transmitting it.</p>
<h2 id="mac-plus-and-portable">Mac Plus and Portable</h2>
<p>Once transfers were working at proper speeds, Eric
<a href="https://tinkerdifferent.com/threads/bluescsi-v2-daynaport-wifi-beta.2857/">announced</a>
a beta build of the BlueSCSI firmware and we got some initial feedback on it.
Notably it broke booting hard disks on the Macintosh Plus, which I hadn't been
testing with because I was using mine for other things and testing meant
constant reboots.
The Macintosh Plus and Macintosh Portable have notoriously "special" SCSI
booting code in their ROM due to them being released before the SCSI spec was
formalized and the
SCSI2SD/ZuluSCSI/BlueSCSI already need
<a href="https://ini.bluescsi.com/">workarounds</a>
to handle booting on these machines.</p>
<p>Since I couldn't leave my favorite classic Mac out in the cold, I dug into what
broke in the BlueSCSI code between the last release and adding my new Wi-Fi
code.
While my firmware changes didn't directly change any of the disk handling code,
it did bring in newer Arduino, Pico, and cyw43 driver code that could have
broken something.</p>
<p>After wading through a recursive maze of dozens of Git submodules and many hours
of trying to pick apart commits in each one, I finally figured out what broke
the Mac Plus.</p>
<blockquote class="code">
[16:37] jcs: ok, the "PlatformIO Setup" commit builds and boots, which moved to a different raspberrypi tree<br />
[16:38] jcs: hard to pick apart the pico-sdk and later changes<br />
[21:17] jcs: i can compile and boot with "Missed a few steps", "build: change board and core to upstream", "don't hard-code watchdog timer id", "uart_init to 115200"<br />
[21:19] jcs: this leaves out "Booting from SD, and comms with CYW43 working"<br />
[22:06] jcs: getting there...<br />
[22:39] jcs: my mac plus is booted with wifi<br />
[22:40] erichelgeson: Wow! What was the culprit?<br />
[22:43] jcs: this change broke it:<br />
<pre>
-#define LED_ON() sio_hw->gpio_set = 1 << LED_PIN
-#define LED_OFF() sio_hw->gpio_clr = 1 << LED_PIN
+#define LED_ON() cyw43_gpio_set(&cyw43_state, 0, true) //sio_hw->gpio_set = 1 << LED_PIN
+#define LED_OFF() cyw43_gpio_set(&cyw43_state, 0, false) //sio_hw->gpio_clr = 1 << LED_PIN
</pre>
[22:45] erichelgeson: Led logic. The first place I'd look when debugging boot issues<br />
[22:46] jcs: right?<br />
</blockquote>
<h2 id="wi-fi-da">Wi-Fi DA</h2>
<p>With the Mac Plus now booting, I started writing a Wi-Fi
<a href="https://en.wikipedia.org/wiki/Desk_accessory">Desk Accessory</a>
which lives in the Apple menu once installed.
Just like a modern computer with its own Wi-Fi card, I wanted to be able to scan
for networks, change to a different one, and see the current signal strength.</p>
<p>I created the Desk Accessory in
<a href="/system6c">THINK C 5</a>
and when launched, it scans the SCSI bus to find the BlueSCSI, then gets the
current network information such as SSID and RSSI.
It then launches a background Wi-Fi AP scan and when the results are ready, it
populates a menu with the AP list.
When a different network is selected, it prompts for the password and then
issues a connection command to the BlueSCSI.</p>
<figure><a href="/images/2023-08-23-menu-1500x722.jpg"><img src="/images/2023-08-23-menu-221x106.jpg" srcset="/images/2023-08-23-menu-221x106.jpg 1x, /images/2023-08-23-menu-442x212@2x.jpg 2x" width="221" height="106" class="1/3" /></a><figcaption>DA Menu</figcaption></figure>
<figure><a href="/images/2023-08-23-networks-1500x722.jpg"><img src="/images/2023-08-23-networks-221x106.jpg" srcset="/images/2023-08-23-networks-221x106.jpg 1x, /images/2023-08-23-networks-442x212@2x.jpg 2x" width="221" height="106" class="1/3" /></a><figcaption>List of nearby networks</figcaption></figure>
<figure><a href="/images/2023-08-23-password-1500x722.jpg"><img src="/images/2023-08-23-password-221x106.jpg" srcset="/images/2023-08-23-password-221x106.jpg 1x, /images/2023-08-23-password-442x212@2x.jpg 2x" width="221" height="106" class="1/3" /></a><figcaption>Prompting for a password</figcaption></figure>
<h2 id="ethertalk">EtherTalk</h2>
<p><a href="https://en.wikipedia.org/wiki/AppleTalk">AppleTalk</a>
is the common protocol that Macs use for file and printer sharing.
I use it on my local network with
<a href="https://github.com/rdmark/netatalk-2.x">Netatalk 2.x</a>
to host a file server on a Linux VM that my Mac Plus running System 6 can
access, while also supporting my M2 MacBook Air at the same time which makes it
very easy to shuffle files between the two.
I also use
<a href="http://macintoshgarden.org/apps/timbuktu">Timbuktu</a>
over AppleTalk to be able to remotely control my
<a href="/2022/07/15/kludge">BBS Server</a> Mac Plus
from my workstation Mac Plus, just like using VNC.</p>
<p>Historically AppleTalk ran over LocalTalk, which used cables daisy chained
between Macs and printers.
AppleTalk eventually gained the ability to send packets over Ethernet, which
became
<a href="https://en.wikipedia.org/wiki/AppleTalk#AppleTalk_over_Ethernet">EtherTalk</a>.
There are also
<a href="/notes/2023/05/04/110312888271884691">devices</a>
that help bridge LocalTalk to EtherTalk.</p>
<p>I had tried getting EtherTalk working in PCE but was never able to receive
anything over my laptop's Wi-Fi connection, so I just assumed it was not going
to work well over Wi-Fi for BlueSCSI.</p>
<p>I read about an informal
<a href="https://windswept.home.blog/2019/12/10/localtalk-over-udp/">LocalTalk over UDP</a>
specification and thought about possibly integrating it into the
BlueSCSI firmware where it could convert EtherTalk packets it receives from the
Mac into UDP packets, enabling EtherTalk over Wi-Fi.
After looking at the code of
<a href="https://github.com/sfiera/multitalk">multitalk</a>, I realized that EtherTalk was
just multicast traffic and was likely being received over Wi-Fi just fine, but
that the CYW43439 Wi-Fi chip on the Pico W was filtering it out.</p>
<p>After
<a href="https://github.com/BlueSCSI/pico-sdk-internal/commit/cfc14491896557d6fab2a71a8d875a8aff56c808">updating</a>
the cyw43 driver we are using to include newer multicast configuration
functionality, I was able to configure the Wi-Fi chip to allow the <code>08:00:07</code>
prefix used by AppleTalk.
Once that filter was modified, I started to receive EtherTalk traffic over the
Wi-Fi interface and could connect to my Netatalk share.</p>
<figure><a href="/images/2023-08-23-chooser-1500x1125.jpg"><img src="/images/2023-08-23-chooser-335x251.jpg" srcset="/images/2023-08-23-chooser-335x251.jpg 1x, /images/2023-08-23-chooser-670x502@2x.jpg 2x" width="335" height="251" class="1/2" /></a><figcaption>File server appearing over EtherTalk over Wi-Fi</figcaption></figure>
<figure><a href="/images/2023-08-23-appletalk-1500x1125.jpg"><img src="/images/2023-08-23-appletalk-335x251.jpg" srcset="/images/2023-08-23-appletalk-335x251.jpg 1x, /images/2023-08-23-appletalk-670x502@2x.jpg 2x" width="335" height="251" class="1/2" /></a><figcaption>File server share</figcaption></figure>
<h2 id="performance">Performance</h2>
<p>The Pico W is not a terribly fast device, but neither are the vintage computers
it's talking to so it doesn't really matter.
On a faster Mac, it can do a few hundred kilobits/second which is about on par
with the real DaynaPORT ethernet devices.
Since the packets are all going over SCSI, doing an upload or download of a file
that has to be read or written to the SCSI disk can put quite a bit of
contention on the SCSI bus.</p>
<p>The biggest issue with this project is probably the tiny integrated antenna on
the Pico W which limits Wi-Fi reception and throughput.
Especially once it's been put into a laptop or other metal cage, its signal
strength can suffer quite a bit.
Unfortunately the Pico W has no provision for an external antenna and I'm not
sure how easy it is to hack one in, but this would certainly be helpful for
portable machines like the Macintosh Portable or 68k PowerBooks.</p>
<h2 id="release">Release</h2>
<p>The
<a href="https://github.com/BlueSCSI/BlueSCSI-v2/pull/59">pull request</a>
for the BlueSCSI firmware implementation is still pending as more people test
out the
<a href="https://github.com/BlueSCSI/BlueSCSI-v2/pull/59#issuecomment-1666548418">beta firmware</a>
and report bugs.
The next major firmware release will have full Wi-Fi support and
<a href="https://scsi.blue/">BlueSCSI distributors</a>
are already shipping v2 units with Pico W boards as an option.</p>
<figure><a href="/images/2023-08-23-portable_installed-1500x1125.jpg"><img src="/images/2023-08-23-portable_installed-335x251.jpg" srcset="/images/2023-08-23-portable_installed-335x251.jpg 1x, /images/2023-08-23-portable_installed-670x502@2x.jpg 2x" width="335" height="251" class="1/2" /></a><figcaption>BlueSCSI v2 in a Macintosh Portable</figcaption></figure>
<figure><a href="/images/2023-08-23-powerbook_180-1500x1125.jpg"><img src="/images/2023-08-23-powerbook_180-335x251.jpg" srcset="/images/2023-08-23-powerbook_180-335x251.jpg 1x, /images/2023-08-23-powerbook_180-670x502@2x.jpg 2x" width="335" height="251" class="1/2" /></a><figcaption>BlueSCSI v2 in a PowerBook 180</figcaption></figure>
<p>The Wi-Fi Desk Accessory for System 6 and 7 will be released shortly as well
once I do a bit more testing.
Without it, Wi-Fi support still works with the SSID and password
<a href="https://github.com/BlueSCSI/BlueSCSI-v2/wiki/WiFi-DaynaPORT">configured</a>
in the <code>bluescsi.ini</code> file on the SD card.</p>
<p>By the way, I'll have a table with some of this Macintosh stuff at the
<a href="https://vcfmw.org/">Vintage Computer Festival Midwest</a>
September 9th and 10th so if you see me there, say hello.</p>
https://jcs.org/2023/07/12/apiAdvice for Operating a Public-Facing API2023-07-12T00:00:00-05:00
<p>I've been operating
<a href="https://pushover.net/">Pushover</a>'s
public-facing
<a href="https://pushover.net/api">API</a>
for
<a href="https://blog.pushover.net/posts/2022/3/ten">over a decade</a>
now and I thought I'd pass on some advice for those creating a new API.</p>
<p>Pushover's API might be unusual in that it is used by a wide range of devices
(embedded IoT things, legacy servers, security cameras, etc.) and HTTP
libraries, rather than mostly being accessed from JavaScript in the latest web
browsers.
It also doesn't process sensitive financial information, so the advice given
here may not be applicable to something operating like Stripe's API.</p>
<!-- SNIP -->
<div class="toc_wrapper">
<h2 class="no_toc" id="table-of-contents">Table of Contents</h2>
<ol start="0" class="top_link"><li><a href="#">Top</a></li></ol>
<ol id="markdown-toc">
<li><a href="#host-the-api-on-its-own-hostname" id="markdown-toc-host-the-api-on-its-own-hostname">Host the API on its own hostname</a></li>
<li><a href="#dont-be-too-liberal-in-what-you-accept" id="markdown-toc-dont-be-too-liberal-in-what-you-accept">Don't be too liberal in what you accept</a></li>
<li><a href="#avoid-oauth-if-you-can" id="markdown-toc-avoid-oauth-if-you-can">Avoid OAuth if you can</a></li>
<li><a href="#log-a-unique-id-with-every-request" id="markdown-toc-log-a-unique-id-with-every-request">Log a unique id with every request</a></li>
<li><a href="#be-descriptive-in-your-error-responses" id="markdown-toc-be-descriptive-in-your-error-responses">Be descriptive in your error responses</a></li>
<li><a href="#use-prefixed-tokens" id="markdown-toc-use-prefixed-tokens">Use prefixed tokens</a></li>
<li><a href="#stay-on-top-of-failures" id="markdown-toc-stay-on-top-of-failures">Stay on top of failures</a></li>
</ol>
</div>
<h2 id="host-the-api-on-its-own-hostname">Host the API on its own hostname</h2>
<p>Serve your API at <code>api.example.com</code>, never at <code>example.com/api</code>.
As your API's usage grows, it will expand beyond your website/dashboard server
and need to move to a separate server or many separate servers.
You'll want to be able to move things around by just pointing its IP somewhere
else rather than trying to proxy things from your dashboard server.</p>
<p>Your API may also have more relaxed security restrictions in terms of TLS
versions and ciphers accepted that you don't want to relax on your dashboard
website that handles sensitive information.
Having your API at its own hostname means it can have its own TLS certificate
and TLS restrictions.</p>
<p>Also, when it comes to
<a href="#stay-on-top-of-failures">blocking bots</a>
and poorly-written clients, a user should still be able to reach your main
website for support even if their IP is blocked from reaching your API server.</p>
<h2 id="dont-be-too-liberal-in-what-you-accept">Don't be too liberal in what you accept</h2>
<p>Accepting a slightly non-conforming API request today from someone's
ESP8266 buried in a forest might mean you'll have to keep accepting those same
non-conforming requests being made years later, often at the cost of having to
<a href="https://arstechnica.com/gadgets/2022/10/windows-95-went-the-extra-mile-to-ensure-compatibility-of-simcity-other-games/">implement workarounds</a>
in your web framework or server code as it gets upgraded.</p>
<p>A lot of users will hack something together until it works with your API and
when it suddenly stops working months or years down the road, you're going to
have to deal with the fallout.
Rather than bending over backwards
<a href="https://en.wikipedia.org/wiki/Robustness_principle">trying to support poorly written code</a>,
don't let their bad code function properly in the first place so it doesn't get
deployed.</p>
<p>Though you don't have to be pedantic about it.
Pushover's API has a message size limitation of 1,024 characters.
If the message parameter is larger than that, I could reject the request because
it's not correct, but then the user's message is lost and they may not have any
error handling.
In this case, I truncate the message to 1,024 characters and process it anyway
(assuming it wasn't so large that it hit the web server's request message size
limit).
The user still receives <em>something</em>, and if they care that it's truncated, they
can properly implement continuation or smarter truncation.</p>
<h2 id="avoid-oauth-if-you-can">Avoid OAuth if you can</h2>
<p>It's a confusing protocol that brings its
<a href="https://portswigger.net/web-security/oauth">own security problems</a>
and it introduces a lot of overhead for your users to get up and running.
With OAuth your API can't be used from a
<a href="https://pushover.net/splash#integration">simple curl request</a>
but has to be a custom multi-step process pulling in a whole OAuth library.</p>
<p>Use static API tokens if you can, but make it easy to rotate them.
If possible, avoid using an authentication mechanism that requires custom HTTP
headers (including basic auth) because some esoteric devices and plugins don't
support them.
Some don't even support HTTP <code>POST</code> properly and will only be able to put form
parameters in the URL query string (though you still need to insist on requiring
a proper <code>POST</code> method, don't be an animal).</p>
<p>Remember, not everyone is going to be sending HTTP requests from the ideal code
you would have written, they're using what they have available.</p>
<h2 id="log-a-unique-id-with-every-request">Log a unique id with every request</h2>
<p>This is probably done by your web framework, but if not, generate a unique ID or
UUID with every request, return it to the user in the message body somewhere,
log it, and ask for it on your support form.</p>
<p>This will make your life much easier in the future when you need to track down
requests in log files and correlate them to a user's support request.
Often times the user won't know the IP they're sending API requests from or
their parameters aren't getting parsed properly so you can't search your logs
for their API token.</p>
<p>Side note: you probably don't need a fancy centralized logging setup to capture
every detail of every HTTP request.
Filter out sensitive <code>POST</code> parameters (Pushover redacts <code>title</code> and <code>message</code>),
log
<a href="https://github.com/roidrage/lograge">one line per request</a> including the
request's UUID to flat files, then rotate and compress the log daily and delete
old ones according to your retention policy.
In 99.9% of Pushover support requests where I needed to consult server logs,
having the date/time, sending IP, method, timing information, and sanitized
<code>POST</code> parameters logged was enough to resolve the issue for the user.</p>
<pre><code>[2023-07-12 16:30:24] [62dd8009-3174-4b86-8078-18330a1e7b0e] [1.2.3.4] method=POST \
path=/1/messages.json format=json controller=Api::One::MessagesController \
action=create status=200 duration=66.14 view=0.1 db=14.73 \
params={"token"=>"apn9u395cgrxxxxxxxxxxxxxxxxxxx", \
"user"=>"uc65prcvfxxxxxxxxxxxxxxxxxxxxx", "device"=>"iphone", "sound"=>"carl", \
"message"=>"[FILTERED]", "monospace"=>"1"} cf_ray=7e5c6320196e0252-ORD \
time0=0.05 time1=3.15 time2=4.78 time3=4.81 timebuild=16.65 timesave=59.43 \
message_ids=m749182415778658654 queued=1 server=hippocampus
</code></pre>
<h2 id="be-descriptive-in-your-error-responses">Be descriptive in your error responses</h2>
<p>Assume a human will read them, even if it's unlikely.
You might be surprised how error messages
<a href="https://old.reddit.com/r/PBSOD/top/?sort=top&t=all">propagate</a>
but at least give the error a fighting chance to be seen.</p>
<p>When applicable, have your API errors include URLs to documentation to avoid
support requests rather than just responding with a minimal "invalid xyz
parameter" that the user has to decode.</p>
<h2 id="use-prefixed-tokens">Use prefixed tokens</h2>
<p>This took me years to stumble upon, but use a short prefix for each type of
random ID you create.
Instead of generating an API token of <code>Mk7vuCg9eptiV8qid4mn</code>, make it
<code>appMk7vuCg9eptiV8qid4mn</code>.
Instead of a user key of <code>zo2iD3x3J9</code>, use <code>userzo2iD3x3J9</code>.
Pushover uses <code>a</code> for API tokens, <code>u</code> for user keys, <code>g</code> for group keys, <code>s</code> for
subscribed user keys, etc.
This makes it easier for users to keep multiple keys/tokens straight when they
all look like gibberish and it makes it possible to automate helpful API error
responses like "your token parameter has a user key instead of an API token".</p>
<h2 id="stay-on-top-of-failures">Stay on top of failures</h2>
<p>Reportedly <a href="https://www.statista.com/statistics/420400/spam-email-traffic-share-annual/">half of all e-mail processed is
spam</a>.
That means a lot of money, servers, and administrative overhead is wasted just
scaling up resources to deal with junk that no one wants.
If you don't stay on top of error responses from your API, they will
accumulate and you'll end up wasting a lot of your own money constantly serving
bad requests.</p>
<p>Obviously you'll want instant alerts of <code>5xx</code> errors from your API when your
database falls over, but here I'm talking about the normal <code>4xx</code> client errors
generated in response to missing required parameters, expired tokens, and other
conditions that you don't normally need to worry about.</p>
<p>Pushover's API generates about 1.5 million <code>4xx</code> errors every day and each one
increments an expiring counter in its database for the sending IP (for IPv6,
rounded off to the <code>/64</code>).
When that count reaches a certain limit in a short amount of time, the IP is
banned for one hour and subsequent requests short-circuit all of the API logic
and are responded to with a <code>429</code> status and a descriptive error message.</p>
<p>When an IP block can be mapped back to a user by way of an API token in the
request, that user is sent an automated e-mail explaining why they were blocked
and that they need to shut down whatever is continuing to send the failed
requests.</p>
<p>If that error count keeps increasing past another threshold, the IP is
blocked harder by adding it to a
<a href="https://www.openbsd.org/faq/pf/">pf</a>
table, blocking it on the IP level.
After 1 hour, the block is automatically removed.
If the IP gets blocked again in a short amount of time, it is blocked for 2
hours, then 4, then 8, etc.
However, avoid doing this type of IP-level block too early or you'll get
inundated with generic "I can't connect" complaints.
Let the requests fail on the HTTP level with proper error responses for as long
as possible so they
<a href="#be-descriptive-in-your-error-responses">get seen</a>.</p>
<p>Something I learned long ago is that
<a href="/2010/04/11/properly_stopping_a_sip_flood">automated things can have a hard time dealing with failure</a>.
Many people are also bad programmers and their method for dealing with "<em>my
HTTP query didn't get a 200 response</em>" is to immediately hammer the API server
again until it does (or it crashes, causing it to be restarted over and over).
So a user's script sends Pushover normal API traffic all month until it gets to
the final week, their API token reaches its monthly limit, and now the script is
in a loop hitting the API with dozens of requests per second.</p>
<p>After dealing with this numerous times, my solution for Pushover's API was to
add a step after the soft block (where requests are dropped with a <code>429</code>
status), but before the hard block (where all traffic from the IP is dropped).
After sending enough <code>429</code> responses that the IP's failure count is reaching the
hard-block limit, Pushover's API will temporarily respond to rejected requests
with a <code>200</code> status code but with a descriptive message in the body explaining
why the message was rejected.
The user's messages weren't going to go through anyway because they were already
sending bad requests (their API token reached its usage limit or the input was
bad), so responding with a <code>200</code> shouldn't cause too many problems and the
descriptive error might possibly be seen by someone once they check why their
messages aren't being processed.
If these temporary <code>200</code> responses fail to slow down the client, they'll
eventually run into the hard block limit and get cut off, but it was worth a
try.</p>
https://jcs.org/2023/05/01/subtext2Subtext 2.0 Released with FidoNet Support2023-05-01T00:00:00-05:002023-05-02T10:30:16-05:00
<p><a href="/subtext">Subtext</a>
2.0.1 has been released:</p>
<ul>
<li><a href="/subtext/subtext-2.0.1.sit">subtext-2.0.1.sit</a>
(StuffIt 3 archive, includes
<a href="//amendhub.com/jcs/subtext">source code</a>
and THINK C 5 project file)<br />
SHA256: <code>9db9c5f4090fd72e<wbr />b353e52607ee6a48<wbr />d3ec31e45adc29d5<wbr />09a28ef8f12fd5e8</code><br />SHA1: <code>241dd8c9e50fe4a4<wbr />e4a8e6fcb36db05c<wbr />53f54385</code></li>
</ul>
<p>This is a major update to Subtext that represents many months of development,
over
<a href="https://amendhub.com/jcs/subtext/amendments">200 amendments</a>,
and many nights of testing and debugging.
The largest change is the addition of
<a href="https://en.wikipedia.org/wiki/FidoNet">FidoNet (FTN)</a>
support for distributed mail (Netmail) and message boards (Echomail) by way of a
<a href="https://en.wikipedia.org/wiki/Binkp">binkp hub</a>.</p>
<!-- SNIP -->
<p>Many thanks to Andy Diller for testing and debugging many release candidates.</p>
<p>All changes in this version:</p>
<ul>
<li>Add support for FTN (FidoNet) EchoMail and NetMail, with support for fetching
from and sending to a Binkp hub, including PKZIP archives of packets</li>
<li>Move mail to a separate database file for easier backup</li>
<li>Implement sysop paging/answering, displaying the user's message on the
screen with a beep</li>
<li>Implement custom "NoModem" protocol for file transfers when a host is
connecting through the trusted host assuming this is a web gateway</li>
<li>List more items per screen in boards, files, and sysop</li>
<li>List newest mail first and a global setting to prune mail after a configurable
amount of days</li>
<li>Add options to sysop menu to forcefully hangup/reset the modem, and to reboot
the system</li>
<li>Add timezone UTC offset setting, use to convert timestamps in FTN messages to
local timezone, and add a "timezone" template variable</li>
<li>Add setting for separate maximum idle time for sysops (which can be set to 0
to disable)</li>
<li>Mask password fields in sysop settings menu</li>
<li>Add <code>BOARD_LIST_BOARDS</code> command for main menu, which lists all local boards</li>
<li>Faster session log pruning, fix a bug where failed logins were still being
written to the session log</li>
<li>Fix SHA1 checksum calculation for uploaded files and show progress
during calculation</li>
<li>Implement folder file description/notes editing and add an option for sysop to
recalculate SHA1 checksum before saving</li>
<li>Try to handle malloc failures gracefully everywhere rather than exiting</li>
</ul>
<p>Note: version 2.0.1 was released shortly after 2.0 to fix a corrupted default
menu list resource.</p>
https://jcs.org/2023/04/18/amendAmend 3.7.1 Released2023-04-18T00:00:00-05:00
<p><a href="/amend">Amend</a>
3.7.1 has been released:</p>
<ul>
<li><a href="/amend/amend-3.7.1.sit">amend-3.7.1.sit</a>
(StuffIt 3 archive, includes
<a href="//amendhub.com/jcs/amend">source code</a>
and THINK C 5 project file)<br />
SHA256: <code>2ea2718c890d0c50<wbr />ff90364f89baccba<wbr />f184314036370dcd<wbr />9c873d21a13d4470</code><br />SHA1: <code>d1f097d0b144b97e<wbr />c45c43144896803e<wbr />970ca767</code></li>
</ul>
<p>Changes in this version:</p>
<ul>
<li>Fix a bug that sometimes caused a crash after the diff/commit window closed
after producing a large diff and using a lot of memory</li>
<li>Fix bug when exporting an amendment as a patch where the suggested filename
was bogus</li>
<li>Fix bug that caused a system lockup on some machines with slower disks; thanks
to Valtteri Koskivuori for helping locate and fix this bug</li>
<li>Show progress window when opening repositories since the process may take a
while</li>
<li>Add resource to allow MultiFinder to correctly open a repo from
double-clicking a .repo file when Amend is already running (thanks to Steve
Crutchfield)</li>
<li>Properly update scrollbars when cutting or pasting in edit and commit window
text fields</li>
</ul>
https://jcs.org/2023/03/04/crtTaking a Better Photo of a CRT Screen with a Phone2023-03-04T00:00:00-05:00
<p>As a frequent reader of the
<a href="https://www.reddit.com/r/retrobattlestations/">retrobattlestations</a>
and
<a href="https://www.reddit.com/r/VintageApple/">VintageApple</a>
subreddits, I see a lot of photos of CRT screens that show significant scanlines
resulting in images like the one on the left.</p>
<p>With a simple post-processing tip on the iPhone (though there is probably a
similar technique for Android phones), it's easy to fix this photo after it's
been taken so it looks like the one on the right:</p>
<figure><img src="/images/2023-03-04-scanlines-335x188.jpeg" srcset="/images/2023-03-04-scanlines-335x188.jpeg 1x, /images/2023-03-04-scanlines-670x376@2x.jpeg 2x" width="335" height="188" class="1/2" /></figure>
<figure><img src="/images/2023-03-04-fixed_photo-335x188.jpeg" srcset="/images/2023-03-04-fixed_photo-335x188.jpeg 1x, /images/2023-03-04-fixed_photo-670x376@2x.jpeg 2x" width="335" height="188" class="1/2" /></figure>
<!-- SNIP -->
<div class="toc_wrapper">
<h2 class="no_toc" id="table-of-contents">Table of Contents</h2>
<ol start="0" class="top_link"><li><a href="#">Top</a></li></ol>
<ol id="markdown-toc">
<li><a href="#live-photos" id="markdown-toc-live-photos">Live Photos</a></li>
<li><a href="#changing-the-key-photo" id="markdown-toc-changing-the-key-photo">Changing the Key Photo</a></li>
<li><a href="#video" id="markdown-toc-video">Video</a></li>
</ol>
</div>
<h2 id="live-photos">Live Photos</h2>
<p>I'm a big fan of
<a href="https://support.apple.com/guide/iphone/take-live-photos-iph17471d6de/ios">Live Photos</a>
on the iPhone.
While they can capture sound and movement that help give context to a photo, I
often use it to salvage a blurry photo by changing the Key Photo, or in this
case, by using Long Exposure to remove scanlines from photos of
<a href="/system6c">my Macintosh</a>.</p>
<p><a href="https://en.wikipedia.org/wiki/Cathode-ray_tube">CRT screens</a>
produce an image by scanning an electron beam across and then down the screen at
a high rate of speed, often at 60 times per second.
When a still photo is taken, it often captures this scanning mid-process,
resulting in big dark bars covering some or nearly all of the screen.</p>
<p>When you look at the CRT through the phone's camera, it will look something like
this:</p>
<figure><a href="/images/2023-03-04-viewfinder-1500x692.jpeg"><img src="/images/2023-03-04-viewfinder-680x314.jpeg" srcset="/images/2023-03-04-viewfinder-680x314.jpeg 1x, /images/2023-03-04-viewfinder-1360x628@2x.jpeg 2x" width="680" height="314" class="wide" /></a></figure>
<p>But then you take a picture and it ends up like this:</p>
<figure><a href="/images/2023-03-04-scanlines-1500x843.jpeg"><img src="/images/2023-03-04-scanlines-680x382.jpeg" srcset="/images/2023-03-04-scanlines-680x382.jpeg 1x, /images/2023-03-04-scanlines-1360x764@2x.jpeg 2x" width="680" height="382" class="wide" /></a></figure>
<p>The reason is that the on-screen viewfinder showing the CRT was showing video
captured at many times per second, so the scanlines are not visible (or are
visible as a single bright line moving slowly down the screen).</p>
<p>The trick to fixing this photo requires that Live Photo was enabled when the
photo was taken.
In the camera app, there are 3 rings in the upper left corner showing it's
enabled, and in Photos, the photo will show "LIVE" in the upper left corner.</p>
<figure><a href="/images/2023-03-04-photos-1500x692.png"><img src="/images/2023-03-04-photos-680x314.png" srcset="/images/2023-03-04-photos-680x314.png 1x, /images/2023-03-04-photos-1360x628@2x.png 2x" width="680" height="314" class="wide" /></a></figure>
<p>Tap on that, and it will reveal a menu.
Tap Long Exposure.</p>
<figure><a href="/images/2023-03-04-menu-1500x692.png"><img src="/images/2023-03-04-menu-680x314.png" srcset="/images/2023-03-04-menu-680x314.png 1x, /images/2023-03-04-menu-1360x628@2x.png 2x" width="680" height="314" class="wide" /></a></figure>
<p>And now the scanlines are gone.</p>
<figure><a href="/images/2023-03-04-fixed-1500x692.png"><img src="/images/2023-03-04-fixed-680x314.png" srcset="/images/2023-03-04-fixed-680x314.png 1x, /images/2023-03-04-fixed-1360x628@2x.png 2x" width="680" height="314" class="wide" /></a></figure>
<p>This does require that the phone was held relatively stable while taking the
photo since it overlays a bunch of still photos over one another.
If you zoom in, you can see that the quality is not as high as it used to be,
but it's often good enough.</p>
<figure><a href="/images/2023-03-04-zoomed-1500x692.png"><img src="/images/2023-03-04-zoomed-680x314.png" srcset="/images/2023-03-04-zoomed-680x314.png 1x, /images/2023-03-04-zoomed-1360x628@2x.png 2x" width="680" height="314" class="wide" /></a></figure>
<p>It's also worth noting that the Photos app will slightly zoom the picture in
when changing it to Long Exposure, so sometimes it's necessary to have the phone
a little farther back than normal when taking the photo.</p>
<h2 id="changing-the-key-photo">Changing the Key Photo</h2>
<p>Rather than overlaying a bunch of still images on top of each other, Live Photo
also allows one to change the Key Photo.
This is often what I use to turn a blurry action shot of my son into a better
still frame.</p>
<p>If the photo was changed to Long Exposure, tap the Live/Long Exposure menu again
and bring it back to Live.
Tap Edit at the top menu and then in the photo editor, you can adjust the
scrubber along the bottom to pick a single Key Photo frame that does not show
scanlines.</p>
<figure><a href="/images/2023-03-04-keyphoto-1500x692.png"><img src="/images/2023-03-04-keyphoto-680x314.png" srcset="/images/2023-03-04-keyphoto-680x314.png 1x, /images/2023-03-04-keyphoto-1360x628@2x.png 2x" width="680" height="314" class="wide" /></a></figure>
<p>This is taking a single frame from the ~2 second video that is the Live Photo,
so it's still not as high quality as the original still photo, but it can be
less blurry than what Long Exposure produced.</p>
<figure><a href="/images/2023-03-04-fixed_photo-1500x843.jpeg"><img src="/images/2023-03-04-fixed_photo-680x382.jpeg" srcset="/images/2023-03-04-fixed_photo-680x382.jpeg 1x, /images/2023-03-04-fixed_photo-1360x764@2x.jpeg 2x" width="680" height="382" class="wide" /></a></figure>
<h2 id="video">Video</h2>
<p>If you need to take video of a CRT, you'll need to use an app that supports
controlling the frame rate to match the CRT's refresh rate (or an even
divisor/multiplier of it).</p>
<p>On my iPad, I use the
<a href="https://www.moviepro.app/">MoviePro</a>
app to take flicker- and scanline-free video for my
<a href="/system6c">C Programming on System 6</a>
series since it supports capturing from both front and back cameras at once and
also supports choosing a frame rate of 30 or 60 fps to mostly align with the
Macintosh's 60.15 fps CRT.</p>
https://jcs.org/2022/12/01/subtextSubtext 1.1 Released2022-12-01T00:00:00-05:00
<p><a href="/subtext">Subtext BBS Server</a>
1.1 has been released:</p>
<ul>
<li><a href="/subtext/subtext-1.1.sit">subtext-1.1.sit</a>
(StuffIt 3 archive, includes
<a href="//amendhub.com/jcs/subtext">source code</a>
and THINK C 5 project file)<br />
SHA256: <code>8b81cc451e8564fc<wbr />b1516c0d3e9183c9<wbr />aaed3658c62f5e83<wbr />84c288af1978171d</code><br />SHA1: <code>fdefd8f8a774b692<wbr />94e6ceee8b3340fd<wbr />c593e094</code></li>
</ul>
<p>Changes in this version:</p>
<ul>
<li>Fix crash that could occur when more than one user is in chat, one user quits
chat, and then remaining users continue to chat</li>
<li>Fix problem when running under non-MultiFinder where subsequent runs would
fail with errors opening board and folder databases because they were not
properly closed on the first run</li>
</ul>