Post by Brian CandlerPost by James Craig BurleyPost by Brian CandlerIf the majority of mail ends up in this state, then under your proposal the
sender of the mail will need to keep polling, just to prove that they are
'still interested'.
No. That might *indicate* such interest. But if the sender isn't
genuinely interested in the email to *that* degree, there's no need to
track the delivery, is there? A quick answer to someone's question on
an email list is really more of interest to the *recipient* than the
sender, so the sender doesn't really care if it reaches the
destination.
That I disagree with strongly. If I didn't care whether the recipient got
the response, I wouldn't have spent the time to compose it in the first
place.
Even a message like "thanks" or "ok" or "the dog needs a walk"?
Anyway, to the extent you care, *you* have to deal with it
accordingly. That is true regardless of whether you're sending via
SMTP, IM2000, or some other system. (I care enough about getting my
email to reach AOL users that I patched qmail-remote to handle that
sort of thing gracefully.)
Post by Brian CandlerAnyway, I think the point is this: if I send a mail, I *do* want it
delivered. It's up to the mail system to do whatever is necessary to get it
delivered, or let me know if it cannot.
No, it's up to *you* to do whatever is necessary to get it delivered,
because *you* have a much wider range of methods by which to transmit
your message to the *person* on the other side.
Email is just *one* way. So "the mail system" should make whatever
reasonable best effort it can, *and* give you reasonably honest
feedback as to its progress (including "I don't know, the recipient's
system doesn't seem to want to tell me", if that's appropriate), so
*you* can decide how to deal with it.
Of course, you can tell *your* end of the mail system that an outgoing
message is crucial, and, under my proposal, it could make that clear
to all the downstream entities. But they'd have more freedom and
flexibility to respond saying "um, no, it doesn't seem that
interesting to *us*" than under SMTP (which has little more than a
black-or-white ability to respond that way).
Post by Brian CandlerBut I think the fundamental problem with your proposal - if I understand it
properly, which I may not as it seems to be quite complex - is that you
would like to see a new set of hoops to jump through to ensure successful
delivery of mail.
Absolutely the opposite: I propose to eliminate *all* built-in hoops,
and, where there's a necessity of choosing whom to burden, to place
the burden on the *sender*, giving the *recipient* the most freedoms
and the most control over the terms of any transaction (including
*willingly* making things easier for a particular sender, or all
senders, compared to other senders or other recipients).
What you call "hoops" are merely various measures recipients and
senders could employ on top of the base system in order to assure that
message transmission and delivery were successful, that the sender is
truly interested in sending *that* email (not just one out of a
bazillion), etc.
And, in fact, my system is actually quite *simple*: a sender sends a
message by transmitting, to a recipient's MX, a notification
consisting of an envelope recipient and the message contents.
If the sender wants to subsequently track the message, the sender
either provides its own tracking ID (probably a unique message ID) or
waits for a response (that it somehow requests in its transmission)
from the recipient containing such an ID or, potentially, a final
delivery status.
If the sender doesn't want to be any more complicated than that, it
doesn't have to. It can simply notice that its delivery might not
have been immediately accepted and report that to its user.
Or, it can use the agreed-upon tracking ID (sender's message ID or the
recipient's replacement ID) when it sends subsequent tracking requests
to the recipient's MX.
A recipient's life can be extremely simple as well: accept incoming
message and report success to the sender as soon as the message is
written to disk in a user's mailbox. That can all happen within a
single TCP session, involving just one handshake: accept message
transmission, respond with indication of success (responsibility for
message taken by recipient) or lack of success (responsibility not
taken).
Increasingly levels of complexity would be allowed by the
specification, but would be optional for both participants.
Post by Brian CandlerThese hoops may involve being given a tracking number, and
having to poll status using that tracking number - potentially for a long
period of time after sending, if the recipient chooses not to confirm that
the message has been successfully delivered.
That's one of the hoops a *recipient* might choose to require the
sender to jump through. If the sender offers its own unique ID, and
the recipient doesn't insist on replacing that with its own, the
recipient can simply "promise" (a la an SMTP 2xy response to the DATA
phase) to take responsibility for the delivery.
I imagine most exchanges would work that way, because they'd involve
sender/recipient combinations that had been previously used, so each
would "whitelist" the other. (The recipient would either accept
responsibility for an incoming message from that sender right away, or
the sender would just assume the recipient would make every effort to
deliver the message and not bother sending lots of tracking requests,
or both.)
And since these conversations can naturally occur between MUAs (though
the recipient's MUA might have to be "reached" via an MX that knows
where to find it, assuming its online -- a sort of smarthost), these
whitelists and blacklists are much smaller and exist on a
per-recipient and per-sender basis -- directly managed by each, though
potentially "backed up" by the usual sorts of upstream lists that
exist today (ISP-wide lists, RBLs, DULs, and the like), for when the
local list provides insufficiently definitive or up-to-date
information.
Post by Brian CandlerNow, the trouble is, everyone will have to write their mail sending system
to be able to jump through those hoops, because when I press "send" on my
mail client, I expect the mail system to do the *utmost* to get it
delivered.
That's your problem right there: "The mail system" seems to include
both the *sender* and the *recipient* subsystems in your view. But
you can legitimately expect only *your* side of things to "do the
utmost" to get it delivered. So you can't burden "the mail system"
without burdening *others*, unless you magically exclude all senders
of UBE from "the mail system" in order to convince all potential
recipients of your email to voluntarily submit to a system that
restricts their use of it so it always deliveries your email.
But, if by "do the utmost" you mean you're willing for your MUA to
send (and potentially resend) your message to *all* listed MXes, and
track them fairly constantly, until it's sure the recipient has seen
the message, without having to worry about duplication of the message
in the recipient's in-box, you can do that with my system...but not
with SMTP or IM2000.
Yes, your MUA might "annoy" the admin of those MXes, but if you do
this only for messages you *really* care about, and if you're not
sending UBE, you won't likely show up on the admin's radar at all, and
if you do, you'll just incentivize her, once she confirms you're not
sending UE, to improve your "rating" so you get a more positive
response from her system more quickly, maybe the first time.
Post by Brian CandlerThere's no point having another button which says "send but don't
try very hard". Why would I ever push that button when I could push "send"
to have a better chance of my message being delivered?
If the default is like SMTP, the default "Send" button would probably
do little more than transmit the message and then send one tracking
request about a minute later, another maybe 10 minutes later, a third
maybe 24 hours later, and a fourth about a week or two later, as
necessary.
That reflects the reasonable correspondence between likelihood of
successful delivery versus getting a bounce back. (Of course, there's
no reason this proposed system *can't* do something akin to a
"bounce", if the recipient system chooses to cooperate.)
So, that is, at *most*, four tracking requests per typical outgoing
email. Hardly a big load for a message *your* MUA already knows
about.
And assuming your IP address isn't blacklisted by the recipient, that
first or second tracking request should be enough to ensure that your
message, which probably hit the recipient's incoming mailbox when it
first arrived, doesn't get prioritized *downward* as likely UBE.
But if you're sending and tracking *many* outgoing emails to a
recipient's entire system (their MX), and that system notices that
you're not whitelisted on many of them (based on MUA activities, such
as content analysis, users hitting "this is spam" buttons, etc.),
they'll be able to deprioritize or even drop *all* your pending
messages and just ignore, or respond without committing to deliveries
to, further tracking requests from you (which you might have to keep
sending to not risk being treated as an uncaring sender in the future,
though doing so for a few messages out of many that you send would
probably not have that effect).
A la IM2000, until your sending MUA gets a confirmation of
responsibility being accepted (and assuming you don't otherwise
archive all your outgoing messages), *your* MUA bears the cost of
storing the message locally until the recipient agrees to relieve it
of that burden. (Remember, though it may have accepted the content
previously, it can always subsequently claim "dog ate my homework", in
essence, asking you to resend the entire message.)
*Unlike* IM2000, your *local* MUA does that, not some third-party
message store that, in reality, does nothing more than to conceptually
shift the anti-UBE filtering problem to a third party. (Submission to
message stores could be done via SMTP, and probably would be for
nearly 10 years, after widespread use of IM2000 by ISPs, as far as I
can tell, from the various discussions of how IM2000 would actually
work, in practice.)
So the burden is where it belongs: on you, as the sender, to get the
message through, via email or some other means.
Now, if your MUA doesn't get a sufficiently positive response to its
initial submission or to a subsequent tracking request -- say, the
recipient's system has strong policies regarding opacity of its users'
activities -- it *might* see something even more helpful: a *reply* to
your outgoing email, in the form of an incoming email.
That is, suppose your MUA sees an incoming email with the equivalent
of this in the headers (though my design would probably include the
ability to encode this sort of thing in the envelope, since it does
pertain to message transmission, not just content):
References: <***@fdsafdsa.fdsfdsa>
If it's smart enough to notice that your *outgoing* email just got
referenced via that ID, then it can decide to not bother tracking it
anymore! Sure, maybe no entity accepted "responsibility" in a strict
sense; but clearly it was seen and acted upon (by a user, by
list-distribution software, whatever), so what's the difference? Your
sending MUA can stop worrying about tracking it, and maybe just delete
your local copy of the outgoing message.
Now, suppose that 1m/10m/1d/7d tracking schedule doesn't work for you
for a *given* piece of email. Well then, adjust it for that
particular email, or just hit the "Track" button when you went to
throw in another tracking request. The recipient's system might not
respond, of course, but that's life.
Do you think you, as a sending user, would prefer being able to track
progress of delivery like this, via your MUA and a well-designed
tracking facility, or would you prefer today's bounce/DSN mechanism,
with all its warts?
(Speaking as an end user, I'd much prefer tracking. Even if bounces
and DSNs were simply notifications based on message IDs and thus fully
and cleanly implemented and cheap, I'd rather know my message was
*accepted* and even *read*, say, 10 minutes after I sent it, than to
wait for not getting a bounce. The closest thing to tracking would be
an across-the-board implementation of DSNs on successful delivery or
on end-user reading of email; in many cases, that's preferable to
tracking, since it's pro-active and thus potentially immediate, but
it's not *quite* as reliable, since it assumes DNS lookups work in the
reverse direction, and the Internet isn't The Phone Company.)
Post by Brian CandlerNow, because this new mail architecture provides more hoops to jump through
which are met more often, then people who write mail sending software
(including spammers) will be forced to make their code jump through those
hoops.
They won't be *forced* to do anything. A sender *can* do nothing more
than throw a submission transmission at a recipient. It needn't wait
for any response whatsoever, or send any tracking messages.
Why might it do this? Well, if it knows, from experience, that the
recipient's system has already whitelisted it and is reliable, why
should it bother doing anything else?
(In particular, embedded controllers would *love* this new protocol.
They'd be designed to assume they're always "whitelisted", and
wouldn't have to implement anything as complicated as SMTP. Any new
email protocol that wants to be adopted widely will, IMO, have to be
easily and naturally implementable in embedded controllers and
processors.)
So, the first three or four times a sender's MUA transmits a message
to a recipient's MTA/MUA ("MA"), it might get less-definitive
responses than desired for awhile, until both sender and recipient
MA's learn, via their owners' actions, that the email relationship is
"confirmed".
After that, both the sender and recipient MA's trust more in each
other. The recipient is more likely to quickly accept responsibility
(a la a 2xy SMTP response) to any email sent by that sender; the
sender is less likely to send frequent tracking requests to the
recipient MA until it reaches the desired level of assurance.
(Since exchanges are MA<->MA without needing large, trusted, third
parties such as smarthosts, relays, and mail stores, the problem of
"mixed blessings" -- opaque sources of what appears to be a mix of
legit email and UBE -- is potentially greatly reduced. That is, a
given MUA acting on behalf of a particular person is unlikely to be
considered a "mixed blessing" by a given recipient to which that
person sends email! The problem of another person behind that same IP
address sending forged email remains, but that becomes more of a
*local* problem that an admin, if not the two mutually trusting users
themselves, can address, while the problem of another person at
another IP address is often detectable as potential forgery by the
recipient MUA, which can easily notice this, if it isn't more
definitively handled by something like SPF or SES.)
Spammers are left much more in limbo in this situation. They aren't
immediately shut out, except by recipient MA's who choose to shut them
out (and thus expose that they are doing so), and they might never see
any definitive indication any of their messages have been *rejected*.
But their messages also won't tend to be widely *accepted* enough for
them to ever acquire "whitelist" status on more than a small % of
their recipients' MAs.
So, spammers will have to find other ways to convince those MAs that
each *new* message they send is even *more* important than the
previous one.
But, each of those MAs, having already heard from spammers with
consistent identification (incoming IP address, for example), will be
even *more* suspicious of such payloads.
(And each recipient's MA will be more "tailored" to that user's needs,
meaning the upstream ISP that acts on behalf of many users will be
less likely to employ some draconian anti-UBE filter that a spammer
can reverse-engineer and then target his spam to "get around".)
Spammers will naturally keep trying to gin up new identities, but the
new system is more easily able to "punish" that, without necessarily
triggering a lot of false positives, by expecting any *legitimate*
messages that come from unknown sources to be reasonably and
frequently tracked by their sender.
Reasonably tracking outgoing messages will end up being an effort
senders of *legitimate* email can easily do, when it comes to their
sending messages to "newish" recipients. (The key insight here is
that *true* creation is a rare event, compared to distributing
advertisements, aka UBE, or to self-replication, aka viruses and
worms. Senders of legit email are more in the camp of "true
creators", so they'll naturally have plenty of resources and desire to
publish their rare creations. If www.timecube.com doesn't prove that,
*nothing* will. Uniqueness and rarity do not, by themselves, imply
desireability; they do discourage involvement in the UBE business,
however. ;-)
Spammers won't be able to afford track their millions or billions of
outgoing email so easily, and they'll have less and less success at
it, over time, as recipient MA's become more sophisticated (based on
learning where, on the Internet, spammers tend to congregate, what
messages they tend to send, and so on).
On the other hand, there are those recipients who are *required* to
accept all incoming email *content*, for various reasons (legal and
otherwise).
My proposal makes that more efficient and easier to deal with, because
accepting *content* needn't imply accepting *responsibility*, and it's
unlikely there'd be legal requirements that recipients accept
*responsibility* immediately, as long as *content* was accepted and
reasonable best efforts made to deliver it downstream.
IM2000 makes life harder for recipients under such constraints, and in
fact they might legally be *unable* to accept *any* incoming IM2000
message notifications, because, upon acceptance, their potential
inability to retrieve actual message *contents* (due to DNS or
mailstore outage) could be viewed as a serious failure on their end.
(I don't agree with those requirements, but certainly it's clear that
IM2000 is a less reliable mechanism for receiving, as well as sending,
email, since it assumes a reliable and trusted third party -- a
mailstore -- will be available 24x7.)
Post by Brian CandlerThat is, greylisting will become *less* effective, because the new
mail system puts a stronger requirement on senders to retry when the mail is
not completely delivered, and so spammers will do the same.
No, "the new mail system" puts no such requirement on senders. It
merely allows *recipients* to raise, or lower, the bar as they see
fit, on more of an ad-hoc, per-message, *per-recipient* basis.
("Per-recipient" probably only with a new protocol or SMTP++, given
the RCPT TO problem.)
"The new mail system" is simply more *flexible* than the current, yet
subsumes all of the current systems' pertinent capabilities. Of
course, senders will tend to *want* to send tracking requests on some
sort of reasonable schedule; recipients will tend to *want* to take
advantage of that tendency to help them decide whether incoming email
is legit or UBE when it is otherwise difficult to characterize, unless
senders of UBE always send tracking requests in just the same way (in
which case they're being "punished" for sending in Bulk, even if the
*content* is all the same, something IM2000 doesn't punish them for
nearly so well).
As to making greylisting "less effective": greylisting is a technique
that is highly specific not only to SMTP as a protocol, but to its
particular universe of implementations (including spamware). It'd be
irrelevant, as a specific technique, under the new protocol.
The main thing people don't like about greylisting is that it delays
legitimate (but not whitelisted) email delivery. Ecrulisting "solves"
that problem by delivering "rejected" email right away, though
downstream entities should be able to feed back instructions to the
MTA regarding how to handle subsequent delivery, assuming spammers
increasingly retry deliveries in such cases; SMTP++ and/or my proposal
would address that problem as well.
(Immediate delivery of such email might not seem to make sense, but,
at the level greylisting is deployed, it often can. In many cases, it
is merely a tactic used by an SMTP server to try to avoid depositing
so much spam in a POP3/IMAP/mbox/maildir mailbox for a user. If the
user isn't soon going to read her mailbox, the delay isn't a problem,
so whether greylisting delays it, or ecrulisting sorta-delivers it and
then yanks it back out again, doesn't really matter. If the user is
"live", then ecrulisting lets the user read the email right away,
unless the user is, for the moment, choosing not to read such email.)
Post by Brian CandlerPost by James Craig BurleyPost by Brian Candlerbut spammers can easily match the raised bar too.
Addressed previously. True for all designs; *inherently* least true
for my proposal, since spammers have to send *all* message contents at
least once (despite the vast majority of recipients being unlikely to
ever read them), as with SMTP but not IM2000, and would have to then
track *all* deliveries in a typical fashion to avoid recipient's
detecting lack of interest in incoming email from a previously unknown
source, which IM2000 provides in its own way, but not SMTP.
But that's not a very high bar, is it? I mean, it's the same as what we have
now (i.e. spammers have to send a copy of every mail to every recipient),
plus some subsequent probing to indicate "interest" in the message delivery,
which is easily added.
Right, just as running their own message stores is easily added. As
long as you're going to play *that* game, why bother changing SMTP at
all? Spammers *already* get around SPF, RBLs, DULs, and so on, just
by upping the ante. Any tactic to make things *more* expensive for
spammers is, apparently, going to make it more expensive for everyone
else as well.
The question would be, can spammers generally afford to track the
millions of messages they send every day, just to make them appear as
important as typical legit email? Even if so, their joe jobs aren't
causing bounces to innocent third parties; recipients have other ways
to notice and filter their spam; and they have to be "present" on the
Internet for longer periods of time than the usual SMTP hit-and-run
approach requires of them.
So *my* approach is to make *ordinary* exchange of emails as
*inexpensive* as possible, favoring recipients over senders where
there's any conflict in the design, so resources are freed up by
*legitimate* exchanges in order to better expend them defending
against *illegitimate* ones.
Post by Brian CandlerIf this modified SMTP doesn't provide any better anti-forgery, or any better
tracking of spam to original source, then it's hard to see what benefits it
offers.
Well, as a new protocol, it'd be trivial to provide "better tracking
of spam to original source", since problems like parsing "Received:"
headers would be solved "out of the box".
Meanwhile, "anti-forgery" is orthagonal to stopping UBE. It's also
orthagonal to exchanging messages. Your own posts have done a great
job of explaining how and why; you seem to be forgetting (or ignoring)
the implications!
Accordingly, under my proposal, pretty much all of the present
anti-forgery systems being discussed would have similar applicability,
and similar pitfalls. (Some pitfalls, like having to encode envelope
senders or recipients a la SES or BATV, would be eliminated, of course
-- that's the same sort of advantage *any* new protocol, including
IM2000, would offer.)
One big upside is that any anti-forgery checking could be delayed
until well after message *contents* are received, so such checks could
affect the decision as to whether to subsequently accept
*responsibility*, or perhaps just quietly drop the email before the
end user ever sees it. Ideally, that's done after (cheap) content
analysis decides it can't assume the message is UBE, so the burden
placed on third parties to authenticate the user is kept to a minimum;
after that, more-expensive content analysis could then be done (this
includes end-user reading of the message, of course).
(And, with a properly designed protocol, which I have increasingly in
mind these days, anti-forgery checking could be done *without*
receiving message contents -- just the envelope, as defined under the
new system. Some anti-forgery checking could even be requested of the
*sender* by the *recipient*, though that makes certain tradeoffs.
Generally, a recipient is going to be choosing between incoming
bandwidth use and increased latencies in almost any transaction
anyway. Since recipients have a wide range of needs and potential
bottlenecks, and these are variable even for a given recipient in any
particular situation, my protocol would try to offer maximum
flexibility here.)
Post by Brian CandlerAnd if it's SMTP just with added complexity, then the complexity
itself will become a problem as there are more corner-cases which may cause
failures; that is, if there are 10 different responses to receiving a
message which the recipient may give, and one of those is very rarely used,
then you may well find it doesn't interoperate very well.
Yes, that's a risk. It's a much bigger risk with IM2000, of course,
where not only does there exist *coding* risk, there exists *dynamic*
risk (an email needs at least *three* distinct parties to be up and
running to be successfully exchanged).
Post by Brian CandlerEven SMTP which
just has 2xx (OK), 5xx (fail) and 4xx (tempfail) has this type of problem;
you will find mail systems which do stupid things if you give a 4xx response
to a EHLO or a MAIL FROM, so in the end to maximise interoperability you
have to accept EHLO and MAIL FROM, and give 4xx/5xx responses to each RCPT
TO recipient (even if what you were trying to do was to reject the MAIL FROM
sender)
Right. That (partially) speaks to the problems of having inadequate
specs, ancient code, etc.
One way in which my system improves on this is that it places *no*
requirements on recipients in most cases. (Maybe not at all, though
one might assume that if a recipient responds to a tracking message by
saying "recipient has actually read and understands message", that
would indeed be the case.)
And since anti-UBE measures would be uppermost in implementor's
thoughts, they'd likely take full advantage of making *recipient*
software resistant to UBE and abuse in various forms.
Accordingly, it'd be much harder for buggy *sender* software to be
successfully deployed in the wild. Its inability to cope with a
certain kind of response would be more quickly found during testing,
since recipient MAs, compared to yesteryear's SMTP servers, would
present a fairly wide range of responses.
And I've already got some tricks up my sleeve, in terms of making the
protocol and its implementations inherently resistant to certain
common sorts of bugs, though those aren't pertinent in this discussion
(which is really more about whether IM2000 is a worthwhile direction
in which to head, versus something like the direction I'm *proposing*,
but not yet 100% convinced is preferable or that, if it is, obviates
the need for IM2000).
My approach definitely turns the "be liberal in what you accept,
conservative in what you produce" koan on its head, insofar as "what
you accept" becomes, in light of the end-to-end principle, a matter of
accepting that you, as *sender*, might never know for sure whether
your outgoing email has been truly received and read, and "what you
produce" pertains to the degree the sender produces only legitimate,
i.e. desired, emails and a reasonable pattern of tracking requests.
(In other words, the burden of the koan is entirely on the *sender*.
The recipient is under few, if any, requirements in terms of what to
accept or what to produce.)
Post by Brian CandlerPost by James Craig Burley1. The infrastructure handling *receiving* emails might be unable to
cope with the load, so legit email wouldn't get through.
2. Those reading email have trouble finding the few emails they
really care about among the huge amounts of UBE they (mostly)
don't, so legit email isn't always read.
Focusing on #1, I say let's think in terms of a system that makes
*receiving* email as cheap as possible, at the (possible) expense of
the sender.
Hmm, except (1) is OK at the present - that is, E-mail infrastructure copes,
even with >50% of all mail being spam. (1) is a problem for ISPs, but can be
solved by suitable spending of (their customers') cash. (2) is where the
real problem lies, for end-users at least.
But (1) is often used as a major argument against UBE, and especially
in favor of deploying anti-UBE measures that result in lots of false
positives (or at least more-risky paths for email to take).
In particular, if (1) isn't a problem, then IM2000 isn't a solution,
because IM2000 doesn't inherently help with (2) anymore than does SMTP
or anything else I can think of.
(Well, I suppose it does if you assume you can reliably detect
forgeries, and always assume forgeries are UBE, *and* assume anything
*else* is *not* UBE. That makes whether a message is forged pretty
much the sole determining factor for whether it's spam! The
phenomenon of 0wned machines argues otherwise.)
Post by Brian CandlerSMTP is already cheap to deliver to, and already people have been adding
various hoops to jump through (e.g. validating EHLO domain, validating MAIL
FROM) - some of which are arbitary, since the EHLO domain should not need
"validating" in the first place, and indeed the RFCs tell you that you must
not. Your proposal needs either to be able to convince people to drop their
arbitary hoops, or give them a new set of standardised hoops to jump through
- hoops which minimise the risk of rejecting legitimate mail, which
unfortunately current ones often do.
Right. Though, whatever arbitrary hoops people employ now would work
just fine, where pertinent, under my all-new proposal, and more of
them would be pertinent under an SMTP++ variant of it (and especially
using ecrulisting).
(By "pertinent" I mean that validating EHLO domain would cease to make
any sense in a protocol that was HELO/EHLO-free. Rejecting
presumptuous, or "early talker", clients would cease to make sense in
a protocol that required no handshaking. Using multi-line greetings
to foil stupid spam software would cease to make sense in a protocol
that had no such thing as a greeting. Etc.)
Post by Brian CandlerI see where you're coming from about leaving policy up to individual sites,
but unfortunately that leaves those sites free to choose very bad policies.
Which leaves us no worse off than today. ;-/
I get the impression you don't yet have a *coherent* view of what an
email *system* should do.
That is, when you talk about your priorities as a sender, they seem to
be in serious conflict with what you'd previously been saying (or
implying) about the priorities a recipient (such as yourself) might
have.
This is not unusual. Most everyone tends to want *any* system to
favor *their* immediate needs, and, by dumping their "requirements"
onto that system, they are often able to be blissfully unaware or
unconcerned that they are imposing conflicting requirements on the
system.
(E.g. "I want to feel safe, so if someone seems scary to me, I want to
be able to call the police and have them investigated. I want to feel
safe and free, so the police must never investigate *me* just because
someone else says I seem scary to *them*." The only way "the police"
can certainly meet both requirements is to cater to only a single
individual in their entire jurisdiction. Different societies resolve
the conflict in different ways, but there is *always* going to be an
external conflict like this for people who hold such internally
conflicting views.)
The essence of my proposal derives from recognizing that what you or I
might call "very bad policies" is, nevertheless, the *right* of sites
to implement.
However, it also derives from recognizing that the *worst* policies
involve burdening third parties (especially those not explicitly
willing to so participate) with "helping" a recipient decide whether
incoming email is legitimate.
That is, whether a recipient's policy is "very bad" may be subjective,
but I think we can all agree that policies that inflict collateral
damage (transmissions or blacklisting that might negatively affect
otherwise-uninvolved parties) are *objectively* "very bad", insofar
as, in their presence, the entire Internet becomes less useful.
That metric rules out Challenge/Response, callback implementations of
all sorts, and, generally, bounces/DSNs (in the presence of potential
joe jobs, aka forgeries), at least in their present forms in the SMTP
world.
It also appears to rule out IM2000, or at least a whole class of
potential designs for it.
(To avoid this problem in IM2000, only mail stores would be allowed to
send message notifications, and a notification would be required to
point directly back to the same mail store -- not at some other store
-- for retrieval to work at all. That not only implies that mail
stores *can* be identified in notifications by IP address; it
basically *requires* such identification, rather than use of domain
names, along with the notion that stores are long-lived at a given IP
address, or long-enough lived so that a response to a notification can
*assume* the destination of the response is the IP address from which
the notification originally eminated. In essence, an IM2000 message
notification might as well be in the form of a TCP connection to a
recipient's agent via a protocol that allows that recipient to say "go
ahead, send the message" or "I'll fetch it later by initiating a TCP
connection to this same IP address". The former is basically SMTP;
the latter would be new in IM2000; neither includes the sort of
inherent anti-forgery capability so many, such as yourself, expect
from IM2000.)
Once you accept that recipients should be, by *design*, prevented from
implementing policies that inflict collateral damage on innocents,
you've pretty much exhausted your ability to create a design that also
practically prevents a recipient from making *other* choices,
regarding filtering and prioritization, to which you object.
And since recipients ultimately are the ruling class, in that they'll
drive the decision as to whether to move away from SMTP to some new
system *and* what that system will be, you'll find that, to convince
recipients to choose a system that *inherently* rules out choosing
policies that inflict collateral damage on the Internet, it's best to
give them *maximum* freedom to do whatever else they want, regardless
of whether that includes choosing what you describe as "very bad
policies".
(In general, it's inconceivable to me that anyone will be able to
deploy a widely-used mail system that keeps recipients from filtering
or prioritizing based on RBLs, DULs, and the like. But I know plenty
of intelligent people who would call these "very bad policies"; I'm
not crazy about blocking based on DULs, since I'm one myself!)
Post by Brian CandlerThe trouble is that the mail system purports to give some information about
a message (the sender/origin), when in fact that is NOT information, it's
mere heresay.
Ah, here's a wonderful attribute of my new proposal: there is *no*
requirement that a sender provide any equivalent to "MAIL FROM:".
So, right out of the box, recipients don't have any envelope sender
address to assume they should, or have to, look up. Anonymous emails
are *inherently* supported by it, and recipients can easily choose to
accept or reject them.
Obviously, if a sender wants the equivalent of bounces/DSNs sent to an
email address, it could provide such an address under the new
protocol. It'd be up to the recipient to decide whether to pay any
attention to that, and the recipient could certainly choose to send
*all* tracking responses to that address, including one that means
"could you verify whether you sent msg #dsahj43fds89?". Instead of an
email address, a URL for some other notification mechanism could be
provided by the sender.
The key thing is to *allow* a sender to identify herself via an
envelope sender address without necessarily requesting bounces/DSNs
(or any other change to *exchange* protocols for the message), so a
recipient MA can *choose* to validate that address on behalf of its
user. (As you noted, this address might have no relationship to
whatever is represented, in the content of the message itself, as the
"sender" or "source" of the message. The MA and user have to be smart
enough to deal with that.)
(Conversely, a sender should be able to request the equivalent of
bounces/DSNs via some channel *other* than email. That's not
particularly pertinent to this discussion, except it's consistent with
my view that any *new* email system must be *extremely* flexible and
efficient from the get-go, or it won't be adopted to any useful
degree. Most people contemplating designing such things seem doomed
to adopt a "straitjacket" mentality, as in "recipients must always
..." and "senders cannot ...". I don't think that works or is
necessary anymore, even assuming it did/was during the days when RFCs
821 and 822 were written. My proposal is mainly about specifying a
*language* for communicating about exchanging emails, so any
straitjacketing and other restrictions are typically *communicated*
about *within* that language and acted upon by the actors who are
doing the communicating.)
Post by Brian CandlerSo more important to me is that any "information" which a receiving site
might use to validate or reject a message, is correct.
That problem cannot be solved in general, as you've pointed out,
without a human being who receives the message being involved (and
smart enough to know to, and how to, validate and authenticate a
message).
Not all messages require such authentication; therefore, the email
exchange system shouldn't require it. Any email that *does* require
such authentication on all incoming messages is, IMO, doomed from the
start as a replacement for SMTP; it'll either be horribly inefficient
soon after it sufficiently encompasses the Internet to actually work,
or its anti-forgery mechanisms will quickly be defeated by spammers,
leaving us with a less-efficient email exchange mechanism than SMTP
and, yet, receiving tons of spam anyway.
Post by Brian CandlerNow, even if your proposal removes the idea of a sender address and bounces
altogether, what you have instead is that the whole message headers and body
may be received before you analyse whether to accept or reject the message.
Right. Or, at least, *some* of the message. A recipient's server
could simply decide to close the incoming connection after seeing the
first 500 bytes of a message, for example.
Such a decision can be practically made within the scope of IM2000,
and made even more practical by allowing a recipient to request
message contents starting at a given byte (one beyond the byte
previously read and presumably saved away somewhere), to save
bandwidth.
SMTP servers can do the same thing, but that prevents them from
providing any definitive response to the upstream client, which will
normally retry the delivery later on. If the server is going to
ecrulist or greylist the message anyway, then that's okay, but it'll
still have to allow for the entire message contents to be redelivered
if that's what it wants to allow later on.
(If it doesn't, it probably has to wait to see enough of the message
to be sure it's the same one before it disconnects again...and go
through this again and again...until the sender gives up or it decides
to wait for the entire contents and return a 5xy code.)
But the RFCs discourage the idea that SMTP servers have complete
freedom with regard to how to treat the incoming connection, so
there'd be all sorts of heated arguments back and forth about whether
servers that "prematurely" closed an incoming connection after
receiving enough of a message to make a decision to ecrulist/greylist
it (such as the "Message-ID:" header or "Received:" headers) were
"conformant".
Under my design, recipients would be explicitly allowed to close an
incoming connection at any time, without having to provide any
explanation or response. It allows tracking requests to be sent and
replied to *after*, and out of band with respect to, the transmission
of message contents. (IM2000 is similar helpful in giving recipients
more flexibility. A request for the contents of a message is distinct
from, and thus out of band with respect to, anything else a recipient
agent wants to do with or request concerning the message. So it can
simply close a TCP connection it opens to read a message once it sees
enough of a message to make that decision.)
Post by Brian CandlerThat means, for example, that people will do the same sort of stupid
filtering on the From: header that they used to do on the MAIL FROM
envelope. And of course, forged From: headers have another particular
problem with "phishing"-type attacks.
Under my *new* proposal, message content is completely unspecified by
the transport protocol. That makes it clear to those authoring
recipient-side software that they can't assume "From:" in the content
means *anything*, if it's even present.
Of course, content can be inferred anyway, based on
upstream/downstream knowledge, and inspected by the recipient's MA
during the transmission to make transmission-time decisions, if
desired.
Post by Brian CandlerSo I think it's essential that any proposal include ways to inherently
validate this information.
I think exactly the *opposite*: that any proposal for exchanging
*email* must *not* assume that validating or authenticating sender
identity is inherent for all email exchange.
However, we might be in agreement, if by "include ways to inherently"
you mean "offer standardized methods for recipients to optionally
attempt to".
Post by Brian CandlerIM2000 does because you *have* to use the sender
identity to collect the mail; if it's forged, mail collection doesn't work.
We've covered that; that's some combination of false and impractical.
(If the implementation doesn't inherently prevent delivery of forged
email, then it imposes a potentially-unacceptable degree of collateral
damage on third, or joe-jobbed, parties, without senders of UBE
themselves being necessarily prevented from getting their email read
during such an attack.)
Post by Brian CandlerSomething like DomainKeys on top of SMTP tries to do the same, although it's
heavyweight and take-up of anything relying on public-key cryptography has
historically been poor.
So, same problem. New protocols should at least avoid making
something like DK any *harder* than it is on top of SMTP, however.
Post by Brian CandlerPost by James Craig BurleyAs the expense of ordinary exchange of email drops across the board,
would it drop very much, compared to SMTP? The bandwidth and disk space for
receiving mail are the same, so you just lose the cost of being required to
send bounces after a message has been accepted.
The main thing is that email becomes more feasible on a point-to-point
basis, so use of third-party relays (including smarthosts,
i.e. senders' ISP's relays) is less necessary.
Further, such relays needn't actually *store* email they relay, if
they are reasonably certain, for a given message, that either the
recipient, having seen it once, will reliably not lose it, or the
sender, not being a mobile laptop, will be online and thus able to
resend it if it *is* actually lost.
A "smarthost" in such a system could in fact be, for most messages,
quite "dumb", simply forwarding all incoming transmissions on the fly,
and doing so for all tracking requests, when recipients are able to
directly respond to the upstream senders.
Most small businesses, even those with mobile users, would probably
use this sort of setup, where the main job of a "smarthost", or
Internet-facing server (the MX), would do little more than know where
a given user is currently at, IP-wise, and forward incoming
transmissions pertaining to that user directly to that user's MUA
(which is listening on a particular port). (It would presumably need
a local DB to associate IDs with envelope recipients, unless the
protocol allowed it to tell senders to always "remind" it who the
envelope recipient for each message was -- which it probably should.)
Providing NAT-style opacity is a step up from this simplicity; here,
recipients respond not directly to the sender, but to the smarthost,
which forwards such responses back to the sender.
But I believe that, in general, moving *away* from requiring
smarthosts for most message exchange is best. It leaves whitelisting,
blacklisting, content filtering, and most related decisions directly
in the "hands" of the recipient's MUA (and the recipient herself),
plus, to a greater degree than at present, leaves some control in the
hands of the sender's MUA (and the sender himself).
Still, "large" ISPs, like AOL and Comcast, will undoubtedly stick to
the "huge smarthost" model for quite some time, and my proposal is
designed to work well in that sort of environment as well.
Post by Brian CandlerOf course, most *recipient* E-mail systems avoid this already, by validating
the recipient at RCPT TO time and sending a 5xx response if the mailbox does
not exist or is otherwise unavailable. So the main cost here is at the
sending mail relays, who may be forced to try to return bounces to
(possibly) non-existent senders.
Yes, but the overall cost to the entire infrastructure of supporting
bounces is still way too high, because plenty of sites are *not* able
to avoid sending joe-job bounces all over the place.
Post by Brian CandlerEliminating that may reduce the cost of building outbound mail relays, a
bit.
Some sites have been *drowned* in bounce handling, especially in ways
that a system like mine would not have burdened nearly so much.
I do agree that, on the whole, there appears to be an across-the-board
effort to reduce generation of bounces.
But that effort will run up against some pretty hard boundaries down
the road, because of the inflexibility of SMTP.
(Think of my proposal as SMTP with more flexibility and efficiency.
That's basically what it is: pretty much anything SMTP can do, my
system can do at least as reliably and efficiently, if not moreso.
That's a design goal. And while recipients generally control the
degree to which SMTP-like behavior is implemented by them, senders
have some say in the desireability of that, e.g. whether to provide
the equivalent of an envelope sender address and/or to request
bounces/DSNs.)
Post by Brian CandlerPost by James Craig Burleythat frees up capital for other things, such as improved anti-UBE
measures, improved vetting of new users by ISPs to be sure they
haven't previously engaged in spamming, etc.
Ah, but if *I* save some money at my ISP, that doesn't free up money for
*other people's* ISPs to invest in controlling their users, unfortunately.
No, but it still frees up money to more precisely determine *which*
users at those ISPs you might decide to trust and thus accept email
from more quickly.
Whether you think that's significant, I don't know, but, previously,
you were saying you had concerns about the costs to recipient ISPs
regarding various approaches to handling email.
With IM2000, I have *major* concerns about the costs to *sender* ISPs
(well, mailstores generally), which apparently have to hold onto
outgoing messages for, potentially, weeks, months, or even *years*,
even if recipients have already read and acted on them.
(They might not tell their MUA to "unpin" them. Probably 50% or more
of the IM2000 mail-reading audience would simply never click "unpin".
It'd either be a default -- meaning UBE would be quickly unpinned as
well -- or mailstores would have to have huge storage capacities.)
And with IM2000, it isn't just the cost of storage, it's the cost of
24x7 availability of and near-zero latency to access, that storage
that worries me.
I've tried to find a better way, and I believe leaving *original*
sender MUAs -- not third-party mailstores -- responsible for holding
onto message contents until recipients explicitly accept them is best,
since *most* senders probably want to archive their outgoing email
anyway.
And by assuming that contents are usually sent along with
notifications, so *only* responsibility is typically not immediately
transferred, sender MUAs aren't expected to be available 24x7 nor to
be near-zero latency to access.
But sender MUAs are more directly burdened by this model than under
SMTP (though senders, via mailstores, don't have to have DNS pointing
back at them to send email). Recipient MAs are *less* directly
burdened, in terms of storage and response-time requirements.
I think that strikes a reasonable and simple middle ground between
SMTP and IM2000.
To convince me that IM2000 is still the best bet, you won't get very
far by complaining about how much *more* expensive my system will be
for senders *or* recipients to support, since IM2000 is even *more*
expensive.
However, you might be right that, when taking into account the
deployment costs, the economics for sending legit email, and the
economics for sending UBE, my proposal isn't any better than SMTP, and
yet IM2000 somehow is (because IM2000 will inherently resist UBE much
better than my proposal).
I'm less and less convinced that's the case, the more I think about
it, discuss it, and see others (even IM2000 proponents) discuss it.
But I'm always on the lookout for that one "gotcha!" that might
afflict my proposal, because I sure don't want to waste any more time
designing or implementing it, or even thinking about it, if there's a
"gotcha!" down that path somewhere!
Post by Brian CandlerPost by James Craig BurleySince my proposal obviates the need for bounces as does IM2000, joe
job are eliminated, and each host's capacity for handling incoming and
outgoing email can be speced based on *actual* expectations for
*legitimate* outoing email plus legitimate incoming email plus a
certain amount of incoming UBE.
Joe-jobs are a small subset of the problem, IMO. If that's your main
concern, then you can fix that using SES/SRS/BATV today.
Of those, only BATV promises an improvement without adding a whole new
external infrastructure (in DNS, mainly), and it only helps partially.
All it helps with is avoiding the extra energy to *store* an incoming
bounce; it doesn't help avoid it being *sent* in the first place, at
least as a message notification.
The others require a whole new infrastructure, and they don't solve
the problem of errant bounces being received from sites that don't
implement them.
And *none* of the proposals (except possibly IM2000) solve the problem
of *genuine* downstream failures resulting in:
- expensive (but inconsistently implemented) bounces being delivered
- bounces being lost because of problems sending them back
- senders not being sure messages were received until they can
reasonably conclude they'll no longer receive a bounce (maybe as
much as two or more *weeks*)
So, not only are joe-jobs not my main concern, SES/SRS/BATV don't
really promise to "fix" them until there's an across-the-board
implementation (well, of SES/SRS anyway, at which point BATV is
probably not strictly needed), and the general problem of bounces
being not what people want these days *anyway* remains.
I do think that if we could be sure we would someday get back to the
situation where over 99% of all *sent* email was actually *desired* by
each recipient, bounces would return to their status of being a
reasonable substitute for tracking outgoing email.
I'm not sure there's any way to get from here to there, other than to
assume email will cease to be widely used, and return to being used by
a smallish group of cooperative, like-minded users, as it was in the
1970s and 1980s. (Hey, maybe that'll describe our planet's entire
population someday...? ;-)
Post by Brian CandlerIt would help if
there were a common agreed standardised format though, so that things like
mailing list software could recognise an encoded sender.
Well, there's an inherent set of improvements we would all
theoretically realize by designing a new exchange protocol from
scratch, one that avoids only the agreed-upon bodges in SMTP (RFCs
2821, 2822, etc.), but otherwise introduces, or at leasts requires, no
"new" twists, a la IM2000 or my proposal.
But that costs so much $$ to do, the question is, would the
improvements be worthwhile.
So, my (or any) proposal must be that much *better* than a
theoretically "clean" replacement for SMTP, in order to make it worth
rolling out. That theoretical replacement is something I keep in mind
as my "baseline" for analyzing IM2000 and for architecting and
designing my own protocol(s).
If that baseline can't be substantially improved upon, we might as
well just re-do what we already understand, but do it *right* this
time -- that is, implement the baseline! (Okay, technically, today's
SMTP is the baseline, but since deployment costs are now so huge
thanks to the popularity of SMTP impementations worldwide, it is
easier for me to think in terms of a *fresh* implementation that
replaces SMTP as a baseline for any other protocol that also would
require a worldwide deployment.)
I *think* my proposal is sufficiently close to the baseline to be
reasonably sure it won't be a failure, and, to the extent it differs
from SMTP in conception, it's a big-enough improvement to *possibly*
justify rollout.
Even if not -- and it probably *is* not enough of an improvement -- it
might be worth implementing as a sort of under-the-radar, low-overhead
way for sysadmins and other "gurus" to converse.
In that case, I might as well just design and implement the full-blown
network OS, or GUIXP as I called it earlier, since mere email is not
*that* useful for such communication, as it's too limited, and SMTP
works pretty well for sysadmin<->sysadmin conversation,
notwithstanding the nonparticipation of people like Knuth and JdBP
(who, IMO, *could* just run SMTP servers on unique port numbers and
publish those suitably, so the rest of us can email them with their
being flooded with spam, but hey ;-).
Post by Brian CandlerPost by James Craig Burley(False positives aren't a problem with my proposal compared to SMTP,
since a message is, presumably, never fully accepted and *then*
discarded as spam, unless that's what a recipient *wants* to do. SMTP
servers accept and silently drop messages all the time these days.)
But with tons of spam hitting an inbox, how are you going to stop people
becoming overwhelmed with it - your problem (2) above? Surely people will
*demand* automatic filtering, and that in turn will involve accepting and
silently discarding mail as people do know? In my definition, automatically
receiving a mail into a 'spam' folder counts as 'silently discarding', since
people rarely look at such folders, and valid messages which end up there
are almost certainly lost anyway.
Problem (2) isn't, ultimately, a mail *transport* problem. It's a
problem for the recipient to filter and prioritize her incoming email,
which requires having all sorts of (pertinent) criteria upon which to
do so.
Realize that I'm not advocating tracking as, first and foremost, a
*prioritization* criterion. Its main purpose is to provide what's
functionally *missing* in SMTP -- tracking -- and also *flawed* about
SMTP (bounces, and inflexibilities and inefficiencies in the protocol
itself).
Generally, tracking of some sort is a reasonable way to realize the
end-to-end principle in email exchange, and once one starts designing
a protocol assuming that principle "rules", all sorts of wonderful
things (mainly, simplifications) happen, as I'm discovering as I
design this new protocol.
So, once you *assume* tracking as a basic property of a mail-exchange
protocol, then it so happens you have yet another criterion for
recipients to use to filter and prioritize their incoming email, since
senders of UBE will *tend* to not want to track their bazillions of
pieces of outgoing email the same way senders of legit email will.
Of course, with tracking instead of bounces, the costs of people
"accepting and silently discarding mail", as you acknowledge they do
now (and I assume they'll *always* do), are much lower: no *actual*
false positives ever occur, so there's no need to to generate bounces
to avoid them. This is one of those "wonderful things" that results
from obeying the end-to-end principle in this protocol.
(By "actual false positives", I mean cases where an intermediate
agent, such as an MTA or MUA, has decided to silently drop an email
that it, or some other MTA/MUA speaking on its behalf, promised the
sender it would deliver. SMTP puts a lot of pressure on an SMTP
server to make such a promise -- the costs of *not* doing so are high
for many reasons -- so they often make that promise. Then, if a
downstream filter decides to not deliver the message, or just dumps it
into a rarely-if-ever-read "spam folder", the sender has no clue the
message has been effectively but falsely discarded, unless a bounce is
sent. At *that* point, anti-forgery mechanisms could be used to
determine whether to actually send such a bounce, if they consistently
work that much later than the original delivery to the first SMTP
server -- but many proposed mechanisms don't. Besides, failure to
authenticate a source is *not* a reliable indicator of a forged email
-- it might be due to DNS problems, for example -- which means a false
positive designating *content* as Unsolicited could be followed by a
false positive designating *sender identity* as potentially forged,
leading to a actual false positive overall: legit email dropped with
no bounce saying so. A "nonactual false positive" is where a legit
email is dropped, but without the sender ever being promised it
wouldn't be dropped.)
You see, in a world where over 99% of outgoing emails are reasonably
expected to be desired and read by recipients, yet where
interconnection is neither 100% available nor nearly zero-latency, a
system like SMTP makes a lot of sense, because messages might need a
series of "hops" to get from point A to point B, but once it reaches
point B and gets into a mailbox, the end user will likely want to read
it.
So, there's no real need to track outgoing messages, since the vast
majority of them will be desired and will reach their destination,
thanks to the best efforts of cooperating SMTP relays. All the sender
needs to know is that the message was successfully injected into the
first component (SMTP relay, or local queue that reliably talks to
downstream relays).
What a sender wants, in that world, is a notification when there's an
actual *failure*, ideally including the entire original message, in
case the sender didn't have enough storage to keep it around. (And
why should the sender keep it around, since the recipient will likely
want to read it, so each relay will do its best to assure it is
successfully transported even in the presence of a system crash?)
Hence the bounce/DSN concept and the assumption that responsibility
for message delivery immediately precedes and usually follows
transmission of message content.
That was email circa the 1970s and 1980s.
We now live in another kind of world, where a very small % of outgoing
emails are reasonably expected to be desired and read by recipients,
thanks to the huge amount of UBE being sent.
IM2000 recognizes that reality and tilts the economics against the
senders (of lots of *distinct* messages, anyway), but it assumes
interconnection is nearly 100% available and zero-latency in order to
make recipients happy with the responsiveness of their MUAs when
reading email.
I believe IM2000's assumptions about interconnection are false, and
will be for decades, so I suggest we stick with SMTP's push-oriented,
relaying concept.
But, in light of the fact that most messages aren't *wanted*, we shift
the burden of "discovering" whether a message is wanted -- from the
*recipient* (who presently bears that burden in the form of sending
bounces or, if quickly able to do so, sending 5xy responses during
SMTP conversations) to the *sender* (who would, under my proposal,
discover whether messages are really being accepted downstream by
occasionally sending tracking requests).
To the extent recipient systems are burdened with sending bounces
*today*, they *should* be less burdened receiving tracking requests
under my proposal. Add to that the improvements of quality of service
-- no actual false positives and no misdirected bounces causing
collateral damage -- and it seems like a potential winner.
(You might be right to say that bounces are less of a problem now.
Well, how is that possible? Either SMTP servers are more quickly
rejecting unwanted email at the door or they are accepting
responsibility for messages that are subsequently dropped. Under my
proposal, a recipient MA can also reject unwanted email at the door,
either during the initial TCP session or in response to the first
tracking request that arrives, usually within a minute or so, and with
less worry about duplicate deliveries a la SMTP's DATA phase
response-time issue. And if it does accept a message *without*
accepting responsibility, something SMTP doesn't really do right now,
a recipient MA can, under my proposal, subsequently drop that message
without resulting in an actual false positive, since the sender won't
have been promised that the message was, or would be, delivered.)
Some of what I'm advocating could be viewed as little more than
allowing an SMTP server more time to respond (to, especially, the DATA
phase) with a 5xy code, plus the ability for the server to respond
with a whole new code class, a 6xy code, meaning "connect to me again
later, remind me what message you're talking about, and maybe I'll
have an answer then". The "remind me" function would take a message
or transmission ID, not the entire message contents.
Since a lot of the difficulty with making SMTP resistant to UBE,
including doing content analysis, anti-forgery checking, and so on,
has to do with having to come up with a definite decision within a
short window of opportunity -- a window made *artificially* short by
the SMTP protocol, which places unnecessary burdens on the recipient,
insofar as they could be placed on the sender -- my proposal is really
just the outgrowth of my considering all the ways SMTP could, or
should, be improved or redesigned to give the *recipient* lots more
flexibility in deciding whether, how, and when to respond to an
incoming message.
The rest of my proposal assumes it'd be nice to give recipient *users*
more direct control over the kinds of things that can normally be done
only by upstream SMTP *servers* (such as reject with 5xy, which
recipients cannot, at present, do), and that it'd be nice to give
sending *users* more visibility regarding what's going on and more
control in determing just when and how message status is obtained.
In particular, what the legitimate message-sending community would, I
think, passionately *love* about my system is the fact that, much of
the time, they'd have a fairly quick positive response that any given
message they cared about had indeed been received and read, without
their having to wait for the recipient to finish composing a reply.
(I certainly love that about TXT messaging via my cell. That "message
delivered" notification is wonderful, because, when I don't get it, I
can decide for myself whether to try a direct phone call to the same
cell #, to another number where I know the person might be, etc. But
I don't know whether that's implemented by an outgoing poll, a la
tracking, or an incoming notification, a la DSN, though it "feels"
more like the latter. The latter is much more reasonable in a
monolothic communications system like The Telephone Company than in a
heterogenous environment like the Internet, though, under a totally
new protocol, it can be made much more reliable and inexpensive than
SMTP's DSNs.)
Is it worth trying such a system out? Well, the "easy" way to do so
and gain some real-world experience with it is to try ecrulisting,
which implies some modest (at least) changes to an SMTP server and
some more substantial changes to downstream components (including
MUAs) that want to directly take advantage of ecrulisting.
Is IM2000 worth trying out? Well, the "easy" way to do *that* is to
implement something like it via SMTP, by having all your (site's?)
outgoing email be munged such that the contents are put up on a web
site you control (complete with an "unpin" button on each page) and
the actual outgoing email has little more than a URL to the page (plus
some kind of "X-IM2000:" header to make it easier for experimental
MUAs to handle things automatically), so recipients gain experience
with whatever delays and outages would be inherent in IM2000, along
with not having to find their mailboxes stuffed with tons of UBE
*content* (just, theoretically, with notifications pointing to UBE
*websites*, assuming spammers decide to jump on that experimental
bandwagon).
Both approaches can be tried, and can be restricted so they "engage"
only when talking to certain sites on the other end, though
ecrulisting is less likely to be disruptive (it doesn't require any
real changes to *external* end-user behavior, except to accommodate
broken, but legitimate, SMTP clients).
--
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>