[Solved] Mail Not Being Sent.

[r0nmlt]

Is it just me or is mail not being sent out???

[Krydos]

Please post the following information:

• Your cPanel username
  
• Your main domain
  
• The server that you are on
  

[HosterSlice]

Just saying, in my experince on Stevie, mail is queued, and not sent right away. Should send at some point.

[Krydos]

I tested both servers sending to a gmail address and received both emails within seconds. Yes, it's a queue but it's usually the difference between 20ms and 600ms.

[HosterSlice]

I can confirm that those times are accurate for the few times I have sent mail.

[Shinryuu]

If you're getting an error return to sender message about your IP being blocked or something like that, it's on your end, go into terminal or cmd and release your IP to request a new one that -hopefully- works.

[r0nmlt]

My cpanel username is fcsltd, my domain is fcs.com.mt and the server i am on is stevie.

 

I mainly use this services as a mailhosting, therefore any email being sent to @fcs.com.mt will be forwarded to 2 accounts. One being hotmail (as backup) and the other my local ISP.

 

This forwarding stopped working last Saturday.

 

I tried sending emails from heliohost webmail. Nothing is getting to destination. I tried sending to hotmail and to my local ISP (vodafone.com.mt) as sometimes it occurred that vodafone blocks emails from stevie, but hotmail never did. This time, nothing is arriving at destination and nothing is being returned with error.

 

The wait is more than 24 hours .. so its not a matter of a normal queue. To be honest I haven't got an email since Saturday!!!

 

Ron.

[r0nmlt]

ok i found the culprit to the problem

 

http://cbl.abuseat.org/lookup.cgi?ip=65.19.143.2

 

IP Address 65.19.143.2 is listed in the CBL. It appears to be infected with a spam sending trojan or proxy.

It was last detected at 2012-09-29 12:00 GMT (+/- 30 minutes), approximately 2 days, 18 hours, 30 minutes ago.

It has been relisted following a previous removal at 2011-10-21 01:53 GMT (347 days, 4 hours, 27 minutes ago)

This IP is infected with the DarkMailer/YellSOFT DirectMailer or other similar trojan. This involves perl or PHP scripts being uploaded to web servers resulting in the sending of large quantities of spam email (usually pharmacy pill spams).

 

PAY VERY CLOSE ATTENTION TO THE FOLLOWING

Darkmailer infects web hosting environments. ONLY the hosting company can fix these infections properly.

If you are not the administrator of this hosting environment, there is probably nothing you can do to fix this infection, you MUST refer this listing to them. The hosting administrator has to do the fix.

This is a checklist of the four things the administrator needs to do before delisting. There is more detailed information about each of these later.

NEW A lot of the darkmailer spam we see involves spoofing/phishing/forging linkedin, facebook and other similar sites. Several correspondents noted:

It creates a folder in /tmp called .state, owned by apache. The folder is a self contained holy terror of spam til you catch it. It hogs resources so you cant ssh in to fix the issue, but doesn't compromise anything else. Once you kill the process and delete the folder,it goes away. I think it stems from a joomla vulnerability.

• Check your FTP logs to find uploads of Darkmailer scripts. Forward to us a copy of the FTP log records that you find. These logs will often be in /var/log/messages, but this depends on your system configuration.
  
• Identify, kill and remove all Darkmailer scripts currently on the web server. NOTE: Many Darkmailer operators cause the Darkmailer scripts to be removed either after they're used, or even during their use. If you cannot find the scripts, this does NOT mean that the CBL is in error in this listing NOR does it mean that you are not presently vulnerable to anotherDarkmailer infection.
  
• Change the passwords of every userid identified as performing FTP uploads, and warn these users that their passwords had been compromised by a keylogger infection. They need to run anti-virus software on their computers.
  
• NEW WARNING: we're getting indications that once initially compromised by FTP, the attackers are uploading alternate file transfer programs that do not rely on the user's password. See below under "r57shell"
  
• Implement port 25 blocking so that only your mail server software userid can make outbound port 25 connections from this machine.
  

 

Darkmailer/DirectMailer Background

 

This detection is that of a spammer who has broken into your web server (usually) via cracked or keylogged FTP credentials. Once they've logged in via FTP, they install perl scripts that do the spamming. These perl scripts are usually installed in a cgi-bin directory, and present (usually) a Russian language spam control panel that the spammer can use to blast out large quantities of spam (most often illegal Pharmaceutical drugs or replica watches).

See, for example, Darkmailer in Wikipedia and this thread in the CPanel Forum.

CPanel and Plesk installations on Linux are the usual targets of this attack, but the reality is that ANY web server on ANY operating system capable of running Perl CGIs and permitting uploads is susceptible to this problem. We've seen it on Solaris or FreeBSD, we've seen it on Windows, we've seen it uploaded with FrontPage, we've seen it under many different web server packages.

You can often identify this (on UNIX/Linux systems) by doing "ps" (process status) and finding many (often 10 or more) long-running processes named ".cgi", ".php" or ".pl" that are owned by the same user as your web server instance. As an example, one infectee saw 25 copies of a "dm.cgi" program running under his Apache server's userid. But this will only help if the script is currently running.

 

Another approach is running the command "netstat -nap" as root. Lines like this (with random program names rather than your MTA software) shows the Darkmailer software in operation:

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 1 192.168.2.2:58246 212.69.102.240:25 SYN_SENT 12614/b.pl

tcp 0 0 192.168.2.2:35843 209.85.201.27:25 ESTABLISHED 7996/ciwhcnsb.pl

tcp 0 0 192.168.2.2:53051 81.13.48.2:25 TIME_WAIT -

tcp 0 0 192.168.2.2:53623 77.243.121.126:25 TIME_WAIT -

tcp 0 0 192.168.2.2:57816 217.13.210.81:25 TIME_WAIT -

tcp 0 1 192.168.2.2:50531 217.16.16.81:25 SYN_SENT 12270/nxhbo.pl

tcp 0 0 192.168.2.2:52437 217 begin_of_the_skype_highlighting FREE 52437 217 end_of_the_skype_highlighting.198.11.26:25 TIME_WAIT -

tcp 0 1 192.168.2.2:50140 195.64.222.2:25 SYN_SENT 9273/yzezihd.pl

 

Foreign addresses that end with ":25" indicate _outbound_ email connections. TIME_WAIT means the connection has been shutdown, but other states indicate active outbound connections. You may not be able to find the program names (eg: b.pl, ciwhcnsb.pl, nxhbo.pl etc) on your file system, because they deleted themselves immediately after starting. But you will be able to find the process via "ps" based on process id (PID).

Again, this ONLY works if you catch it while it is running. See the next paragraph:

The spammers run this spamware in several different ways. The first way is that the spammer simply uploads the software and runs it at will. You will sometimes be able to find these in the cgi-bin directory. The second way is that they upload the software, run it, and then delete it (perhaps when it's STILL running). You won't be able to find the files in the cgi-bin. Either way if you don't secure your system, the spammer can just do it again at any time.

Checking FTP logs/Securing Users

 

Normally, web hosting environments log FTP uploads, often in /var/logs/syslog, /var/log/messages or some similar file.

For example, this is some logs an administrator found from a Darkmailer infection - notice how it ws deleted after every upload:

Mar 4 04:02:11 enam pure-ftpd: (example@117.41.229.131) [NOTICE]

/home2/example//public_html/rocker/dark.cgi uploaded (74627 bytes,

126.66KB/sec)

Mar 4 05:03:54 enam pure-ftpd: (example@117.41.229.131) [NOTICE] Deleted

dark.cgi

Mar 4 07:04:42 enam pure-ftpd: (example@117.41.229.131) [NOTICE]

/home2/example//rocker/dark.cgi uploaded (74627 bytes, 122.25KB/sec)

Mar 4 07:11:43 enam pure-ftpd: (example@117.41.229.131) [NOTICE] Deleted

dark.cgi

 

Assuming you're running some flavor of UNIX, simply grep the log file for "ftp", "cgi", "\.pl" or "php" and see if you can identify such log records. The CGI scripts are usually around 74K bytes in size. There is frequently also an upload of small test script (around 1-2K) called "test.pl" that the spammer uses to test whether the big script will work. Sometimes the spammer uploads the files under different names and renames them by FTP to an executable suffix (like ".cgi" or ".php" etc). So your grep may only find the rename commands.

We are providing Law Enforcement with whatever intelligence we can find about where these come from, so, please forward such log records to us - the timestamps (tell us what timezone you are in) and IP addresses are the most important.

It's not always FTP that is used to upload these scripts. Eg: scftp, rsync, rcp, server-side FrontPage extensions etc. We have reports of the scripts being uploaded via a Joomla vulnerability. Check their logs, and disable any access that you/your customers do not really need.

The account used ("example" in the above example) will tell you which users' passwords have been compromised. Reset their passwords and warn them that their computer is probably compromised with a spam trojan and they should run anti-virus software to find it.

Unfortunately, anti-virus software has fallen FAR behind in being able to find such things, so, the user telling you that A-V didn't find an infection doesn't prove anything. Such users should reformat their computers and reinstall from trusted media.

It's possible that SOME of these users are the spammer, so you may be tempted to terminate their account. Don't do that - once you configure your firewall correctly, it doesn't matter. If they aren't the spammer, they'll continue to be your customer and will be unaffected by the firewall change. If they are the spammer, they'll leave on their own because the firewall prevents them spamming.

Finding and removing the Darkmailer scripts

 

If you find the FTP (or other) logs, that will tell you where the scripts were placed. You will still need to scan the cgi-bin and other directories to find any other copies. Remember that these files are usually around 74K in size, so that will help you find them quicker. Remove them if they still exist.

Securing your server from future Darkmailer infections

 

You MUST configure your web server to prevent DarkMailer/DirectMailer infections being able to spam the Internet. Once you've done this correctly, it doesn't matter whether the spammer can still upload this malware, they can't spam with it, so they'll leave you alone.

There are a variety of ways to do this. In short, you're implementing a firewall restriction that only permits root and the mail server userid to make outbound port 25 (email) connections. You can either do it yourself with a software firewall, or, use third party software to do the same thing.

configserver.com has a variety of products and services that can deal with this issue. Note that the CBL has no connection whatsoever with ConfigServer. If you know of other software packages that can deal with Darkmailer please let us know and we'll mention them here.

The most commonly used ConfigServer product appears to be ConfigServer Security and Firewall (CSF) and it's FREE. The feature you want to turn on is "CSF SMTP_BLOCK" which, as far as we can tell, does exactly the firewall restrictions we describe above.

Another product that ConfigServer offers is ConfigServer eXploit Scanner (CXS). This software is not free ($75 regular price, currently $50). This software monitors FTP uploads in real-time, will automatically detect Darkmailer and other malicious downloads, and remove them.

Most Cpanel implementations already have something called "SMTP Tweak" (aka "WHM SMTP Tweak") available. It apparently doesn't do the firewall configuration we describe, but it wouldn't hurt to turn it on too.

If necessary, you can implement the firewall restrictions yourself without using any extra software.

iptables -A OUTPUT -d 127.0.0.1 -p tcp -m tcp --dport 25 -j ACCEPT

iptables -A OUTPUT -p tcp -m tcp --dport 25 -m owner --gid-owner mail -j ACCEPT

iptables -A OUTPUT -p tcp -m tcp --dport 25 -m owner --gid-owner mailman -j ACCEPT

iptables -A OUTPUT -p tcp -m tcp --dport 25 -m owner --uid-owner root -j ACCEPT

iptables -A OUTPUT -p tcp -m tcp --dport 25 -j REJECT --reject-with icmp-port-unreachable

 

You may need to add or change the "-m owner ... ACCEPT" to be consistent with your mail server. Eg: you'll need different entries for Qmail.

You will also have to ensure that these iptables commands are executed every time the system reboots, perhaps by an init script.

If you're using cPanel and APF, APF by default will wipe out iptables rules you enter manually leaving the server vulnerable. If you are using APF, you should make the above change via APF and that will take care of reissuing the commands upon reboot or reset.

Note: in some virtual hosting environments, the above commands will return error messages. This generally means that the host (not virtually hosted) operating system does not support the iptables kernel modules. If you do get such errors, make sure that the base operating system has the iptables module fully installed.

r57shell Infestations

 

In one case, it turned out to be a file called "info.php" in the user's images directory. Info.php turned out to be a modified copy of the "r57shell" PHP script which provides a backdoor through which an attacker can do virtually anything on your web server.

Thus, even though you have changed the passwords, the spammer could still upload the spamming scripts at will - this was found by noticing invocations of the PHP file in the web server logs from the same IP address the original FTP connections came from. You will need to search for such files as well, and we recommend preventing the execution of scripts (.php, .pl, .cgi, etc) in directories that do not need it. Eg: only the cgi-bin directory should permit execution. Nullamatix has a discussion on some simple ways to find r57shell.

It is known that Symantec EndPoint Protection can detect the original r57shell. The index.php file described above was a modified r57shell, and SEP doesn't detect it. A handful of AV detectors detect it as Backdoor.PHP.Rst!, but this doesn't help on Linux/UNIX. Note in particular, ClamAV not detect the "info.php" variant mentioned above.

If you don't need PHP capabilities in your web server, turn it off. Or consider enabling it only for specific hostings that need it.

WARNING: If you continually delist 65.19.143.2 without fixing the problem, the CBL will eventually stop allowing the delisting of 65.19.143.2.

If you have resolved the problem shown above and delisted the IP yourself, there is no need to contact us.

Click on this link to delist 65.19.143.2.

 

So I clicked the link to delist and I think it worked, but to the base problem. Is this problem coming from my domain or from someone using/abusing heliohost?

[Krydos]

This is why we have to suspend so many accounts for sending too many emails everyday.

[r0nmlt]

Problem back to haunt us. If you check http://cbl.abuseat.org/lookup.cgi?ip=65.19.143.2 it is shown as being listed again.

 

Can I do something from my end???

[Krydos]

Yeah, if you can get them to tell you what domains are sending the emails we can suspend them faster.

 

Also clicking the delist button when you're not an administrator and you're not sure if the problem has been solved can actually make it worse because if they relist you immediately when the problem wasn't fixed they begin to assume that the whole server is a spam source and the listing may end up being permanent.

[r0nmlt]

I will refrain from doing that in the future, but somebody else did now! Most probably other users of heliohost are experiencing the same problems and they followed the link in the bounced email and clicked unlist.

 

Now for the domains sending out the spam part.

 

I have checked the email message header of stevie and this should have:

 

X-AntiAbuse: Original Domain - fcs.com.mt (or something similar).

 

I presume that this trojan cannot alter/fake/omit this right? If this is correct I will ask them to provide the domain names and if they do consent I will post them here.