Post by Clemens FischerPost by James Craig BurleyYup. However, spammers and vermin authors will soon learn how to make up
80-character subject lines (assuming your model) that are very
convincing. People (and their MUAs) will ultimately have to base their
decisions on the sender and recipient addresses, plus attributes of
delivery such as injecting system (today's SMTP client), message store
(today's SMTP relay), and very little else.
what should be part of a notification, then? note that real spammers will
continue to use non-existent email addresses, so { subject, from, to, date,
X}, with X beeing a user negotiated set of message headers, might already
be enough to distinguish spam from ham.
Which recurses to the problem we have today of automated content-based
spam detection (e.g. Bayesian filters).
Post by Clemens Fischerbut please, by all means go ahead and propose the headers you need.
Though it seems like ages since I wrote that, I believe at the time I
was actually pointing in a different direction -- towards JdBP's im2k
proposal, which prohibits things like a "Subject:" header in a
notification.
That is: { To: From: Date: }, tightly controlled fields, and nothing
else (aside from the necessary glue, e.g. message store identifier,
not normally shown to end users).
I'm not sure whether that'd be sufficiently practical to work, but
it'd largely eliminate the problem of "covert channels" that JdBP
rightly describes on his web site.
Post by Clemens FischerPost by James Craig BurleyPost by Shae Matijs ErissonA spammer tries to forge an email sender. Result: your client can't pick
up an email from a server that doesn't exist.
That's a big advantage of im2000. Although, I think you mean "tries to
forge an outgoing message store"; im2000 per se doesn't really "care"
about the identity of an email *sender*, or does it?
yes it does. im2k message are supposed to be authenticated by a (possibly
PGP) key.
Really? That's news to me.
But, then, why not just authenticate SMTP messages? In fact, aren't
people doing that *now*? Couldn't we then just slowly migrate to that
model, if it's sufficiently useful?
Post by Clemens FischerPost by James Craig Burley- Adoption requires SMTP compatibility (JdBP's "shims", I think he
calls them), but breaks a crucial SMTP feature, namely, bounces.
IMO if "we" are allowed to "break" SMTP's bounce feature, why
don't we just do that *now* and see how it goes? We'd save $$$$$
on all the joe jobs out there as the breakage is implemented, in
favor of sender-side polling or success-only notification or
whatever.
bounces are no longer a crucial feature: you either download a message or
you don't, period.
Then we can migrate away from *SMTP* bounces, independently of
migrating towards anything like im2k, correct?
The point is, if we can do *that*, and it saves sufficient resources
presently being consumed by spam and vermin (and, for my part,
incoming misdirected bounce messages probably account for at least 75%
of my total email load), why spend all the extra time and money trying
to migrate everyone to im2k?
Post by Clemens FischerPost by James Craig Burley- Given the above, is the huge cost of a rollout of a new email
system that makes *two* major changes to everyone's assumptions
about email works justifiable? (Pull-style delivery is one
assumption; eliminating receipient-side notification of delivery
status, aka DSNs or bounces, in favor of sender-side polling, is
another.)
this is nonsense. there are no bounces and DNS doesn't change.
First, the fact that there are no bounces is important: people will
see two *major* differences between their SMTP and im2k emailing
experiences:
1. Additional delays reading incoming messages.
2. No bounce messages for outgoing messages that are undeliverable.
Second, I was talking about DSNs, "Delivery Status Notifications", not
DNS.
Right now, we can (AFAICT) only speculate about the effects #s 1 and 2
will have on the viability of any new email system.
But we can explore the implications of #2 by migrating away from
bounces in SMTP, which appears to be the inevitable result of the
Internet migrating towards a JdBP-like im2k system (i.e. with SMTP
shims) anyway, except with much less overhead.
(The promised payoff for rolling out im2k is, of course:
3. Permanent, significant reduction in incoming spam and vermin.
Whether that materializes is central to the debate over whether im2k
*should* be rolled out, especially to the issue of whether SMTP email
can or should be improved pending, or in conjunction with, im2k
rollout.)
Post by Clemens FischerPost by James Craig BurleyAt some point in that timeframe, spammers and verminware authors will
target (like never before, if they have played around with it before)
im2000, exploiting its weaknesses *as deployed*, which will include
weaknesses im2000's designers are currently assuming won't exist (just as
SMTP's authors didn't envision today's balkanized SMTP implementation
worldwide). For example, "pink contracts" between owners of outgoing
message stores and spammers will mean that users will have to choose,
just as they do today, how aggressively they'll block potentially
legitimate email in the hopes of blocking well-disguised spam and vermin.
understand that you don't "block" email, you just don't download it.
From an end-user point of view, there's really no difference: their
MUA (or some further entity acting as an agent for the user) decides
whether to make the notification and probably the email contents
themselves known to the end user.
So an automated im2k-based system that helps end users avoid reading
messages that are stored at likely spam/vermin sites will also avoid
notifying those users of such messages. To those users, the effect
should be the same as if the messages were blocked under today's SMTP
regime.
(If the end user sees notifications without message contents and has
to decide whether to "click" on them to "download" the messages, then
the system will fail for many end users who'd get so much spam and
vermin that they'd tire of trying to make those decisions themselves,
but who aren't particularly burdened by having the message contents
already downloaded and thus immediately available on their behalf.
I'm exactly that sort of user; im2k benefits me hardly at all as a
*reader* of email, because I'd rather have all the message contents
available on my own machine for immediate reading when I pull up my
MUA, so it might as well have come in via SMTP and save the extra
overhead. im2k would help users who want to avoid the expense of
downloading message content they can easily deem as "unwanted" based
solely on the message notifications themselves. I don't know whether
there's a large-enough audience for that. But I love im2k's promise
of no more bounces, as an MTA admin!)
Post by Clemens FischerPost by James Craig BurleyThat's assuming early protoypes prove workable. With proper up-front
research, such as simulations using tiny DNS caches on a mini-Internet,
it might be quickly learned that the DNS cache problem and typical
network latencies *will* lead to an unacceptable end-user experience,
such that "real people" will refuse to use im2000 because it's "too slow
reading mail".
this is pure speculation.
Please clarify which of the following you mean by "this":
- DNS depends on caching to work sufficiently well.
- DNS caching relies on locality of reference (that is, repeated
lookups over a smallish set of all potential domain names).
- im2k's pull-based retrieval of email *guarantees* at least one DNS
lookup based on an arbitrary domain name per message retrieved from
a mail store.
- Sufficiently frequent notifications delivered to a site, using
arbitrary domain names for message stores, could lead to that site's
DNS cache losing performance, since the names being looked up will
not exhibit sufficient locality of reference to keep the cache
hit rate high enough to avoid upstream ("miss") lookups.
- Sufficiently frequent upstream ("miss") lookups from a variety
of systems under this form of attack could lead to substantial
latencies for DNS lookups.
- Sufficiently large latencies for DNS lookups would make reading
incoming im2k emails very painful, and (unless distinct caches
were employed) make other sorts of activities, like pulling up
web pages, very painful as well, because each time a user sees a
message notification and requests the content, a DNS lookup is
required to find the message store.
I believe we are already seeing plenty of evidence for this scenario,
although these are *somewhat* different cases (which spammers
naturally attack in appropriately different ways).
Specifically, I'm thinking of the various RBLs people use. They, like
im2k, rely on DNS lookups for each incoming piece of mail. And, like
I think im2k would end up being when attacked, they result in
substantial delays, sometimes deferred email, though I admit some of
those situations involve a large portion of incoming email that does
not require those lookups but they are done anyway as a matter of
course. After all, if us regular joes can lookup addresses in those
RBLs, why can't spammers do so in order to dDOS these centralized data
bases?
For the IPv4 address space, spammers can exploit only a portion of the
< 4 billion potential addresses.
For domain names -- upon which all im2k message retrievals are likely
to depend -- spammers need not exploit even .00001% of the nearly
infinite range of possible addresses in order to force any DNS cache
used by a system they're attacking to thrash, making im2k pretty much
unusable by its end users.
Try as I might, I can't think of any countermeasure that doesn't
simply recurse to being identical to the measures being employed
*today* to combat spam and vermin (under the SMTP regime).
(I see the same potential problems with any system depending on
lookups of arbitrary external data bases to determine trustworthiness:
SPF, DomainKeys, etc. As end-user-invoked means to make those
determinations, i.e. after other forms of filtering are done, they
should be quite useful; as currently advocated, though, they are
invoked automatically for each incoming email, which hands spammers an
attack vector AFAICT.)
Again, the proposed model could, and should, be tested in "miniature":
10K DNS caches instead of 10M, for example, then subject to various
forms of attack to see how the system behaves.
So, I remain of the opinion that im2k would probably be a wonderful
option to have available for emailing people who use it, and I applaud
anyone who devotes time and energy to producing it (especially
well-done free implementations of it)...
...but, as a replacement for SMTP email based on the idea that it'll
substantially reduce spam and vermin and the costs associated
therewith, I remain convinced that it won't justify the expense of
rollout (even given free implementations).
It should have a wonderful initial honeymoon, but once it includes a
sufficiently wide audience of adoptees, it'll become the target of
attacks.
(There have been times, recently, when I considered that im2k, mainly,
pull-style delivery, would be preferable to push-style when it comes
to some mailing lists. So I'm serious about it being a useful option.
I'd prefer we design and roll out a new system offering a wider range
of transport, delivery, and notification options; but I think I've
said that before and it wouldn't *be*, even though it would "subsume",
im2k.)
--
James Craig Burley
Software Craftsperson
<http://www.jcb-sc.com>