IMAP Spam Begone (isbg)

You should check your spamd configuration: that kind of broken pipe occur when spamd (or a firewall) close connection due to a lack of rights.

You should

1. try by hand to submit mail with spamc
2. check your firewall rules

Pourtant, isbg fonctionne correctement sur le même compte mail, mais dans un autre dossier (INBOX.ITHACA au lieu de INBOX):

$ isbg.py --imaphost bat710.univ-lyon1.fr --ssl --imapuser dclauzel --verbose --spaminbox INBOX.Junk --imapinbox INBOX.ITHACA --flag --spamc --learnhambox INBOX.ITHACA
Trackfile is /home/ltp/.isbg-trackXXXXX
SpamFlags are (\Flagged)
Password file is /home/ltp/.isbg-XXXXX
Successfully read password file
('login', 'dclauzel', 'xxxxxxxx') = ('OK', ['LOGIN Ok.'])
('select', 'INBOX.Junk', 1) = ('OK', ['10'])
('select', 'INBOX.ITHACA', 1) = ('OK', ['24'])
('fetch', '1:24', 'UID') = ('OK', ['1 (UID 67)', '2 (UID 68)', '3 (UID 69)', '4 (UID 70)', '5 (UID 71)', '6 (UID 72)', '7 (UID 73)', '8 (UID 74)', '9 (UID 75)', '10 (UID 76)', '11 (UID 77)', '12 (UID 78)', '13 (UID 79)', '14 (UID 80)', '15 (UID 81)', '16 (UID 82)', '17 (UID 83)', '18 (UID 84)', '19 (UID 85)', '20 (UID 86)', '21 (UID 87)', '22 (UID 88)', '23 (UID 89)', '24 (UID 90)'])
('uid fetch', '67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90', '(UID RFC822.SIZE)') = ('OK', ['1 (UID 67 RFC822.SIZE 1512)', '2 (UID 68 RFC822.SIZE 1696)', '3 (UID 69 RFC822.SIZE 3588)', '4 (UID 70 RFC822.SIZE 3023)', '5 (UID 71 RFC822.SIZE 4538)', '6 (UID 72 RFC822.SIZE 3447)', '7 (UID 73 RFC822.SIZE 8958)', '8 (UID 74 RFC822.SIZE 4988)', '9 (UID 75 RFC822.SIZE 7052)', '10 (UID 76 RFC822.SIZE 8514)', '11 (UID 77 RFC822.SIZE 7184)', '12 (UID 78 RFC822.SIZE 8946)', '13 (UID 79 RFC822.SIZE 3646)', '14 (UID 80 RFC822.SIZE 3980)', '15 (UID 81 RFC822.SIZE 4877)', '16 (UID 82 RFC822.SIZE 3680)', '17 (UID 83 RFC822.SIZE 3982)', '18 (UID 84 RFC822.SIZE 8049)', '19 (UID 85 RFC822.SIZE 5115)', '20 (UID 86 RFC822.SIZE 4534)', '21 (UID 87 RFC822.SIZE 4653)', '22 (UID 88 RFC822.SIZE 4918)', '23 (UID 89 RFC822.SIZE 9715)', '24 (UID 90 RFC822.SIZE 2872)'])
67 score: 0.0/3.1

68 score: 0.0/3.1

69 score: 0.0/3.1

70 score: 1.4/3.1

71 score: 0.0/3.1

72 score: 1.2/3.1

73 score: -0.2/3.1

74 score: 0.0/3.1

75 score: 0.0/3.1

76 score: 0.6/3.1

77 score: 0.7/3.1

78 score: 0.6/3.1

79 score: -0.2/3.1

80 score: -0.2/3.1

81 score: -0.2/3.1

82 score: 0.0/3.1

83 score: 0.0/3.1

84 score: -0.2/3.1

85 score: 0.0/3.1

86 score: -0.2/3.1

87 score: 0.0/3.1

88 score: 0.8/3.1

89 score: 0.1/3.1

90 score: 0.0/3.1

Teach HAM to SA from: INBOX.ITHACA
('select', 'INBOX.ITHACA') = ('OK', ['24'])
('fetch', '1:24', 'UID') = ('OK', ['1 (UID 67)', '2 (UID 68)', '3 (UID 69)', '4 (UID 70)', '5 (UID 71)', '6 (UID 72)', '7 (UID 73)', '8 (UID 74)', '9 (UID 75)', '10 (UID 76)', '11 (UID 77)', '12 (UID 78)', '13 (UID 79)', '14 (UID 80)', '15 (UID 81)', '16 (UID 82)', '17 (UID 83)', '18 (UID 84)', '19 (UID 85)', '20 (UID 86)', '21 (UID 87)', '22 (UID 88)', '23 (UID 89)', '24 (UID 90)'])
67 
68 
69 
70 
71 
72 
73 
74 
75 
76 
77 
78 
79 
80 
81 
82 
83 
84 
85 
86 
87 
88 
89 
90 
24/24 hams learnt
0 spams found in 24 messages

Doubt it'll solve your problem, but please try latest version of the 0.99 branch: http://github.com/ook/isbg/tree/0.99
Thank you to precise which version of python you're using.

Thomas Lecavelier wrote:

Doubt it'll solve your problem, but please try latest version of the 0.99 branch: http://github.com/ook/isbg/tree/0.99 Thank you to precise which version of python you're using.

$ python --version
Python 2.5.4

$ uname -a
Linux serveur.LT-P.net 2.6.30-2-686 #1 SMP Sat Sep 26 01:16:22 UTC 2009 i686 GNU/Linux

$ cat /etc/debian_version 
squeeze/sid

Les branche 0.99 me semble mieux fonctionner, dans le sens où, je crois, je rencontre moins souvent le problème.
Aussi, j'ai l'impression que peut-être le problème serait causé par mails précis, ou un grand nombre de mails dans un répertoire. C'est difficile à dire.

Mais le problème est parfaitement reproductible : si isbg fonctionne sur un dossier, alors il fonctionnera tout le temps dessus; et s'il crash, alors il crashera tout le temps aussi sur ce dossier.

Big news! Bug reproduced with a big bad inline encoded attachement. I'll investigate that.

Content-Type: application/vnd.ms-powerpoint
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="joyeux noel.pps" 

Guilty file spotted. Can you check that your guilty mail has an attachment, please?

Thomas Lecavelier wrote:

[...]

Guilty file spotted. Can you check that your guilty mail has an attachment, please?
I don't think it has.

Sender: xxx yyy <xxx.yyy@gmail.com>
Cc: =?ISO-8859-1?Q?aaa_bbb=E9?= <aaa.bbb@liris.cnrs.fr>
Message-Id: <83D3CF94-C9C8-4984-B3CE-000000000000@liris.cnrs.fr>
From: xxx yyy <xxx.yyy@liris.cnrs.fr>
To: Damien Clauzel <Damien.Clauzel@liris.CNRS.fr>
In-Reply-To: <42EC039E-9F75-4C5A-B7AB-111111111111@liris.CNRS.fr>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: =?ISO-8859-1?Q?Re:_[zzzzzzzzz]_Mod=E8les?=
Date: Thu, 2 Jul 2009 16:00:18 +0200
References: <53B7A82A-11B4-4FEF-9FEB-22222222222@liris.CNRS.fr> <6D4DB187-5F5E-4897-828B-333333333333@liris.cnrs.fr> <42EC039E-9F75-4C5A-B7AB-4444444444444A@liris.CNRS.fr>
X-Mailer: Apple Mail (2.930.3)

This issue is also affecting me, ill try and find the offending email tomorrow and attach some of it here.

Anders Jenbo wrote:

This issue is also affecting me, ill try and find the offending email tomorrow and attach some of it here.

I'd be very glad: if I can't identify a common denominator, I can't event propose a workaround.

I identified this one as being the issue, i still have to go threw about 150 more messages so i might find some more.

http://www.arms-gallery.dk/isbg-killer.eml

It's 3MB and has a 2.3MB large .xls file attached to it. It is defiantly the largest messages that i have scanned.

the options i use
--learnthendestroy --learnspambox INBOX.spam.learn --expung --teachonly

when i run with --verbos it dumbs the full content of the message (and attachement) to the screen just before dying.

I haven't mail with problem currently in my inbox, so I can't test, but heavy googling suggest say we should launch the script with python's -u switch (for unbuffered stdin/stdout). Could you try for me, please?

Found one more:
http://www.jagtogfiskerimagasinet.dk/isbg-killer2.eml

This one is also a very larg email with .xls attachments.

You should be able to upload the files i posted to your inbox (works with evolution and windows live mail, simply by draging the file to your "inbox".

I'll try launching the script with the switch once i figure out how :)

didn't seam to help

python -u /domains/isbg.py --imapuser mail@jagtogfiskerimagasinet.dk --imaphost imap.jagtogfiskerimagasinet.dk --teachonly --learnspambox INBOX.Junk.er-spam --spamc --delete --expunge
Traceback (most recent call last):
File "/domains/isbg.py", line 633, in <module>
out = p.communicate(body)[0]
File "/usr/lib/python2.6/subprocess.py", line 689, in communicate
return self._communicate(input)
File "/usr/lib/python2.6/subprocess.py", line 1197, in _communicate
bytes_written = os.write(self.stdin.fileno(), chunk)
OSError: [Errno 32] Broken pipe

this is the one with my eta code so the lines numbers woun't be the same as yours probably.

One more
this time it is 0.5Mb and has a pdf attachment

http://www.huntershouse.dk/isbg-killer3.eml

The previous one was 10.3 Mb

This one is a 12MB mail contaning two larg JPEGs
http://www.jagtogfiskerimagasinet.dk/isbg-killer4.eml

Got one more, this time it was a virus attachment so i'm not going to post it.

With the last update, I am now getting this error:

…
ICCYGEQEIg\r\nIXA9I1DORK68trbGxsfLSsfHx0rz+bxVVVX5R97i40Tveq5k0j0hkBC4gRCAvZmcnLDKykrL5XJ+\r\nLmfr1rhEcG
PiWe+718YhNm6Od/m47yhTbIH22qhfqkVCICGQEEgIJAQSAgmBhEBCICFwIyNQ\r\nPjs728+xF6JqcmJyYo7JT8nY2JjpkyZAN3LXSH
y/CIzz/pqYmHAiR+8yEVjxmJmZ9jjFx3dd\r\njLvOz+LosKmd21laVpafm53V/jIFG7PrvG5J/YRAQiAhkBBICCQEEgIJgYRAQuAGR+
qrYMEx\r\nCEbWAAAAAElFTkSuQmCC\r\n--001485f78fd65d255b047a4dabd9--\r\n'), ')'])
Traceback (most recent call last):
  File "/home/ltp/Travail/isbg/isbg.py", line 458, in <module>
    out = p.communicate(body)[0]
  File "/usr/lib/python2.5/subprocess.py", line 670, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.5/subprocess.py", line 1223, in _communicate
    bytes_written = self._write_no_intr(self.stdin.fileno(), buffer(input, input_offset, 512))
  File "/usr/lib/python2.5/subprocess.py", line 1003, in _write_no_intr
    return os.write(fd, s)
OSError: [Errno 32] Broken pipe

I put in a try: to prevent isbg from crashing, it's not a prefect solution since that message still will not be learned or scanned.