Brian Candler
2005-03-24 18:06:49 UTC
OK, a bit more googling around and I found www.im2000.org (it wasn't linked
from DJB's original paper) and from there
http://homepages.tesco.net./~J.deBoynePollard/Proposals/IM2000/
Is this the current 'state of the art' with regard to these proposals, or is
there anything more up-to-date / more detailled / competing?
I would like to make a few observations and raise a few questions re. what
I've seen so far.
(1) There is an existing protocol which allows people to authenticate to a
message store and put messages there; which allows people to retrieve a
message given its message-id; and which lets the user query whether new
messages have arrived using a sequence number. That protocol is NNTP.
The example given at
http://homepages.tesco.net./~J.deBoynePollard/Proposals/IM2000/CaseStudies/public-mailing-list.html
sounds *very* much like running your own NNTP server to host your mailing
lists.
NNTP server software might form the basis of a prototype implementation, if
not the real thing; NNTP servers have the additional benefit of being able
to replicate to other NNTP servers (e.g. if lists.ietf.org wanted to have
some mirrors around the world), and of having been heavily load tested by
carry gigabytes of pr0n and war3z every day :-)
I see on "djb's questions" that this has been ruled out. But it doesn't hurt
to play.
(2) Continuing with mailing lists, there is no mention of RNASP on that
page, so I guess that in the mailing list scenario, the message store is not
expected to send out notifications to all the list members?
Does this mean that there are essentially two different types of message
store account, those intended for personal use and those intended for lists
(perhaps set by some sort of flag?) Or is posting a message to a mailing
list effectively the same as posting a mail with zero recipients?
Is there never any case where a user can post a message via their own local
message store? (e.g. by sending a mail via their own message store to
"upgrade-to-***@lists.ietf.org"; the mailing list server receives a
notification, collects the message, and puts it in its own message store for
public consumption)
If that's not true, then either:
- mailing list message stores are fully open to the public to post.
In that case, what (if anything) stops these message stores becoming
bogged down with spam a la USENET? Especially if public cancels are
disabled?
- you must get an account (username/password) on the mailing list message
store before you're allowed to post, which forces you to go through some
sort of registration procedure, and store the username/password on your
client. That just makes life harder for people who want to join lists,
although an easy hurdle for mailing list spammers to overcome.
(3) Now, to the meat. Looking at a typical transaction at
http://homepages.tesco.net./~J.deBoynePollard/Proposals/IM2000/CaseStudies/lucy-reading.html
I observe that:
- Lucy is manually filtering her mail
- Lucy's client still has to retrieve the headers of the mail from the
spammer's server, in order to decide if it's spam or not
- By doing so, Lucy has immediately confirmed to the spammer that her
account works and she is reading from it
- Lucy can choose to block notifications from a particular message store;
she could also subscribe to a blacklist of known spammer message stores.
This is exactly the same situation as today where we have IP blacklists
of SMTP sources. She can also blacklist a tuple of <message store,
user ID>
So: how, exactly, is life made harder for spammers, and easier for
legitimate users?
Let me put myself in a spammer's shoes for a minute (holding my nose to
avoid the smell). What would I do?
(a) Set up my own message store, and send notifications to every E-mail
address I could find. This has the nice advantage that I only need to keep
*one* copy of the spam on my server! However, people might start building
bulk-detectors like DCC, in which case I might prefer to keep a separate
copy for each recipient, each of which is subtly different (disk space is
cheap, after all; a 250GB hard drive holds a lot of spam). Users may block
notifications based on the user ID I use on the message store, but then I
just create new user IDs. Ultimately I may find my message store is
blacklisted by its IP address.
But for as long as it works, people are forced to find notifications in
their inbox of the spam from me, and at least look at the headers, as they
do now. Or they'll build content filters which behind-the-scenes download
the mail and decide to chuck it or keep it. Same as now.
(b) I use my ISP's message store; they provide me with this service as part
of my Internet access. That would be equivalent in the current world of
relaying SMTP via my ISP's smarthost. Now, the world cannot block
notifications from that mail store, because in doing so they would block
notifications from legitimate users too.
They *could* however block notifications based on the user ID, not just the
IP address of the message. The assumption is that a well-run ISP will not
give out new user IDs repeatedly to the same physical person. (That may or
may not be true, especially if 'hotmail' services continue to exist).
This is the first real advantage that the new architecture would have. It's
more or less the same as if all mail submissions to an ISP's smarthost were
required to use SMTP AUTH, and this information were carried along with the
message. IM2000's advantage here is that messages never go through multiple
hops, and therefore you don't need to worry about authentication information
being forged. But see below.
(c) OK, so my own message store is blacklisted, and I can't use my ISP's
mailstore because my user ID is blacklisted. So, now I go to my network of
0wned zombie machines. I install message stores on all of those, and I send
notifications from them. This is the same in the current world of sending
SMTP from those boxes. Those boxes may send notifications directly, or they
may relay via the ISP's smarthost. Again, people can blacklist by IP if they
receive them directly, or can blacklist the message store user ID.
This is almost the same as the current world, except that it's hard to
blacklist mail which spews via an ISP's smarthost. To fix this, users would
have to use SMTP AUTH to talk to their smarthost, and SMTP would have to
have some way of carring the SMTP AUTH information forwards through the
smarthost.
But actually it does: RFC 2554 defines an extension
MAIL FROM:<***@bibble> AUTH=***@flowerpot1.org
The AUTH information is only as trustworthy as the server you receive it
from, of course, and ISTM this is the actually the fundamental point where
IM2000 wins.
But if SMTP AUTH were widely used, it would be sufficient for our purposes:
- blacklist IP X where it's clear that IP X is a spammer or bot machine
- blacklist IP Y + AUTH ID Z where it's clear that IP Y is an ISP
smarthost which has not been compromised, and AUTH ID Z is the account
being used by the spammer or bot.
Bingo. So the only problem left is to convince the whole world to stop
running smarthost services on port 25 which relay based on source IP only,
and make everyone use SMTP AUTH. And that, at the end of the day, seems to
be the biggest hurdle.
The IM2000 approach can then be considered to be like starting the Internet
from scratch where everyone uses SMTP AUTH. At that point, an incoming
message could be rejected if the MAIL FROM line didn't carry an AUTH string;
and anyone who was running a relay which faked up AUTH strings would be
blacklisted by IP address.
A similar effect could be achieved without the AUTH extension, if submission
MTAs forced the Sender: header to SMTP AUTH id. But then you'd have to wait
until the end of the DATA phase to check it.
So in summary:
- When spammers install IM2000 servers, there are very few reasons why it
would make their lives any harder. They will *save* storage costs, and they
will *save* bandwidth for undeliverable spams (just sending notifications
instead)
- Users will still be forced to download headers or bodies, and to filter
them; and in doing so they will give out useful information about the
actual IP address they are on at that instant, and the fact that they have
read the message
- The one possible advantage I can see is that it requires users to
authenticate to message stores. Untrusted message stores can be blacklisted
by IP (just as now), but trusted message stores can be blacklisted by
<IP,userID> tuple, thus not affecting non-spammers on that store. But this
will only work if users cannot easily signup for free message store
accounts. And if the world continues to provide "hotmail"-type services
which allow sending as well as receiving, then clearly spammers can get as
many userIDs as they wish (until the whole "hotmail" service itself is
blacklisted for not controlling its userbase properly, which is again back
to where we are today)
- We will still rely on the good people who run public blacklist services if
we want our mail to remain relatively spam-free. And those blacklist
services tend to rely on donations, and are always at risk of legal threats,
violence and extortion, and lack of funds.
So to me, the 'brave new world' of IM2000 doesn't look all that much rosier
than we have today. Have I missed something?
(4)
http://homepages.tesco.net./~J.deBoynePollard/Proposals/IM2000/djb-answers.html
"Recipients are identified by mailbox names, just as they are with the
SMTP-based Internet mail system. Message stores use the DNS to determine,
from a mailbox name, where the RNASP service of each receipient's recipient
notification agent is to be found. In particular, they perform a SRV
resource record lookup on the domain name portion of the recipient's mailbox
^^^^^^^^^^^^^^^^^^^
name."
There is a huge opportunity here not to be missed - the ability to make the
whole E-mail address portable, not just the domain part. If you're going to
rebuild the world, why not arrange that mail to ***@pobox.com does a
DNS lookup for b.candler._at.pobox.com ?
This means:
- different users at the same domain could have different receipt
notification agents
- senders could check recipient validity before even sending an RNASP
notification
Wildcard DNS could be used if you want *._at.pobox.com to all hit the same
receipt notification server.
If you wanted this to work, you'd have to register your new address with the
recipient notification service (otherwise incoming notifications for
***@pobox.com would be refused with "I don't know that address"). But
essentially, forwarding services like pobox.com would become nothing more
than DNS hosting services.
(5) Has any thought been given as to how often notifications would be sent,
and for how long? If sending to someone who only collects their mail once a
week, how much bandwidth will be eaten up by useless notifications?
(Presumably, recipient notification agents are not allowed to 'silence' the
sending of these notifications on behalf of the receiver, because they are
non-reliable devices who don't guarantee to preserve state).
(6)
http://homepages.tesco.net./~J.deBoynePollard/contacting-the-author.html#IM2000
I'd like to send a mail to the author using IM2000, but in order to do so I
need to have a copy of the technical specs. Do any exist yet?
Well, that's enough for now...
Cheers,
Brian.
from DJB's original paper) and from there
http://homepages.tesco.net./~J.deBoynePollard/Proposals/IM2000/
Is this the current 'state of the art' with regard to these proposals, or is
there anything more up-to-date / more detailled / competing?
I would like to make a few observations and raise a few questions re. what
I've seen so far.
(1) There is an existing protocol which allows people to authenticate to a
message store and put messages there; which allows people to retrieve a
message given its message-id; and which lets the user query whether new
messages have arrived using a sequence number. That protocol is NNTP.
The example given at
http://homepages.tesco.net./~J.deBoynePollard/Proposals/IM2000/CaseStudies/public-mailing-list.html
sounds *very* much like running your own NNTP server to host your mailing
lists.
NNTP server software might form the basis of a prototype implementation, if
not the real thing; NNTP servers have the additional benefit of being able
to replicate to other NNTP servers (e.g. if lists.ietf.org wanted to have
some mirrors around the world), and of having been heavily load tested by
carry gigabytes of pr0n and war3z every day :-)
I see on "djb's questions" that this has been ruled out. But it doesn't hurt
to play.
(2) Continuing with mailing lists, there is no mention of RNASP on that
page, so I guess that in the mailing list scenario, the message store is not
expected to send out notifications to all the list members?
Does this mean that there are essentially two different types of message
store account, those intended for personal use and those intended for lists
(perhaps set by some sort of flag?) Or is posting a message to a mailing
list effectively the same as posting a mail with zero recipients?
Is there never any case where a user can post a message via their own local
message store? (e.g. by sending a mail via their own message store to
"upgrade-to-***@lists.ietf.org"; the mailing list server receives a
notification, collects the message, and puts it in its own message store for
public consumption)
If that's not true, then either:
- mailing list message stores are fully open to the public to post.
In that case, what (if anything) stops these message stores becoming
bogged down with spam a la USENET? Especially if public cancels are
disabled?
- you must get an account (username/password) on the mailing list message
store before you're allowed to post, which forces you to go through some
sort of registration procedure, and store the username/password on your
client. That just makes life harder for people who want to join lists,
although an easy hurdle for mailing list spammers to overcome.
(3) Now, to the meat. Looking at a typical transaction at
http://homepages.tesco.net./~J.deBoynePollard/Proposals/IM2000/CaseStudies/lucy-reading.html
I observe that:
- Lucy is manually filtering her mail
- Lucy's client still has to retrieve the headers of the mail from the
spammer's server, in order to decide if it's spam or not
- By doing so, Lucy has immediately confirmed to the spammer that her
account works and she is reading from it
- Lucy can choose to block notifications from a particular message store;
she could also subscribe to a blacklist of known spammer message stores.
This is exactly the same situation as today where we have IP blacklists
of SMTP sources. She can also blacklist a tuple of <message store,
user ID>
So: how, exactly, is life made harder for spammers, and easier for
legitimate users?
Let me put myself in a spammer's shoes for a minute (holding my nose to
avoid the smell). What would I do?
(a) Set up my own message store, and send notifications to every E-mail
address I could find. This has the nice advantage that I only need to keep
*one* copy of the spam on my server! However, people might start building
bulk-detectors like DCC, in which case I might prefer to keep a separate
copy for each recipient, each of which is subtly different (disk space is
cheap, after all; a 250GB hard drive holds a lot of spam). Users may block
notifications based on the user ID I use on the message store, but then I
just create new user IDs. Ultimately I may find my message store is
blacklisted by its IP address.
But for as long as it works, people are forced to find notifications in
their inbox of the spam from me, and at least look at the headers, as they
do now. Or they'll build content filters which behind-the-scenes download
the mail and decide to chuck it or keep it. Same as now.
(b) I use my ISP's message store; they provide me with this service as part
of my Internet access. That would be equivalent in the current world of
relaying SMTP via my ISP's smarthost. Now, the world cannot block
notifications from that mail store, because in doing so they would block
notifications from legitimate users too.
They *could* however block notifications based on the user ID, not just the
IP address of the message. The assumption is that a well-run ISP will not
give out new user IDs repeatedly to the same physical person. (That may or
may not be true, especially if 'hotmail' services continue to exist).
This is the first real advantage that the new architecture would have. It's
more or less the same as if all mail submissions to an ISP's smarthost were
required to use SMTP AUTH, and this information were carried along with the
message. IM2000's advantage here is that messages never go through multiple
hops, and therefore you don't need to worry about authentication information
being forged. But see below.
(c) OK, so my own message store is blacklisted, and I can't use my ISP's
mailstore because my user ID is blacklisted. So, now I go to my network of
0wned zombie machines. I install message stores on all of those, and I send
notifications from them. This is the same in the current world of sending
SMTP from those boxes. Those boxes may send notifications directly, or they
may relay via the ISP's smarthost. Again, people can blacklist by IP if they
receive them directly, or can blacklist the message store user ID.
This is almost the same as the current world, except that it's hard to
blacklist mail which spews via an ISP's smarthost. To fix this, users would
have to use SMTP AUTH to talk to their smarthost, and SMTP would have to
have some way of carring the SMTP AUTH information forwards through the
smarthost.
But actually it does: RFC 2554 defines an extension
MAIL FROM:<***@bibble> AUTH=***@flowerpot1.org
The AUTH information is only as trustworthy as the server you receive it
from, of course, and ISTM this is the actually the fundamental point where
IM2000 wins.
But if SMTP AUTH were widely used, it would be sufficient for our purposes:
- blacklist IP X where it's clear that IP X is a spammer or bot machine
- blacklist IP Y + AUTH ID Z where it's clear that IP Y is an ISP
smarthost which has not been compromised, and AUTH ID Z is the account
being used by the spammer or bot.
Bingo. So the only problem left is to convince the whole world to stop
running smarthost services on port 25 which relay based on source IP only,
and make everyone use SMTP AUTH. And that, at the end of the day, seems to
be the biggest hurdle.
The IM2000 approach can then be considered to be like starting the Internet
from scratch where everyone uses SMTP AUTH. At that point, an incoming
message could be rejected if the MAIL FROM line didn't carry an AUTH string;
and anyone who was running a relay which faked up AUTH strings would be
blacklisted by IP address.
A similar effect could be achieved without the AUTH extension, if submission
MTAs forced the Sender: header to SMTP AUTH id. But then you'd have to wait
until the end of the DATA phase to check it.
So in summary:
- When spammers install IM2000 servers, there are very few reasons why it
would make their lives any harder. They will *save* storage costs, and they
will *save* bandwidth for undeliverable spams (just sending notifications
instead)
- Users will still be forced to download headers or bodies, and to filter
them; and in doing so they will give out useful information about the
actual IP address they are on at that instant, and the fact that they have
read the message
- The one possible advantage I can see is that it requires users to
authenticate to message stores. Untrusted message stores can be blacklisted
by IP (just as now), but trusted message stores can be blacklisted by
<IP,userID> tuple, thus not affecting non-spammers on that store. But this
will only work if users cannot easily signup for free message store
accounts. And if the world continues to provide "hotmail"-type services
which allow sending as well as receiving, then clearly spammers can get as
many userIDs as they wish (until the whole "hotmail" service itself is
blacklisted for not controlling its userbase properly, which is again back
to where we are today)
- We will still rely on the good people who run public blacklist services if
we want our mail to remain relatively spam-free. And those blacklist
services tend to rely on donations, and are always at risk of legal threats,
violence and extortion, and lack of funds.
So to me, the 'brave new world' of IM2000 doesn't look all that much rosier
than we have today. Have I missed something?
(4)
http://homepages.tesco.net./~J.deBoynePollard/Proposals/IM2000/djb-answers.html
"Recipients are identified by mailbox names, just as they are with the
SMTP-based Internet mail system. Message stores use the DNS to determine,
from a mailbox name, where the RNASP service of each receipient's recipient
notification agent is to be found. In particular, they perform a SRV
resource record lookup on the domain name portion of the recipient's mailbox
^^^^^^^^^^^^^^^^^^^
name."
There is a huge opportunity here not to be missed - the ability to make the
whole E-mail address portable, not just the domain part. If you're going to
rebuild the world, why not arrange that mail to ***@pobox.com does a
DNS lookup for b.candler._at.pobox.com ?
This means:
- different users at the same domain could have different receipt
notification agents
- senders could check recipient validity before even sending an RNASP
notification
Wildcard DNS could be used if you want *._at.pobox.com to all hit the same
receipt notification server.
If you wanted this to work, you'd have to register your new address with the
recipient notification service (otherwise incoming notifications for
***@pobox.com would be refused with "I don't know that address"). But
essentially, forwarding services like pobox.com would become nothing more
than DNS hosting services.
(5) Has any thought been given as to how often notifications would be sent,
and for how long? If sending to someone who only collects their mail once a
week, how much bandwidth will be eaten up by useless notifications?
(Presumably, recipient notification agents are not allowed to 'silence' the
sending of these notifications on behalf of the receiver, because they are
non-reliable devices who don't guarantee to preserve state).
(6)
http://homepages.tesco.net./~J.deBoynePollard/contacting-the-author.html#IM2000
I'd like to send a mail to the author using IM2000, but in order to do so I
need to have a copy of the technical specs. Do any exist yet?
Well, that's enough for now...
Cheers,
Brian.