apache.org被入侵

The Apache Software Foundation
Blogging in Action.


Apache Infrastructure Team
« ASF Buildbot svn... | Main
Tuesday Apr 13, 2010

apache.org incident report for 04/09/2010


Apache.org services recently suffered a direct, targeted attack against our infrastructure, specifically the server hosting our issue-tracking software.

The Apache Software Foundation uses a donated instance of Atlassian JIRA as an issue tracker for our projects. Among other projects, the ASF Infrastructure Team uses it to track issues and requests. Our JIRA instance was hosted on brutus.apache.org, a machine running Ubuntu Linux 8.04 LTS.
Password Security

If you are a user of the Apache hosted JIRA, Bugzilla, or Confluence, a hashed copy of your password has been compromised.

JIRA and Confluence both use a SHA-512 hash, but without a random salt. We believe the risk to simple passwords based on dictionary words is quite high, and most users should rotate their passwords.

Bugzilla uses a SHA-256, including a random salt. The risk for most users is low to moderate, since pre-built password dictionaries are not effective, but we recommend users should still remove these passwords from use.

In addition, if you logged into the Apache JIRA instance between April 6th and April 9th, you should consider the password as compromised, because the attackers changed the login form to log them.
What Happened?

On April 5th, the attackers via a compromised Slicehost server opened a new issue, INFRA-2591. This issue contained the following text:
ive got this error while browsing some projects in jira http://tinyurl.com/XXXXXXXXX [obscured]

Tinyurl is a URL redirection and shortening tool. This specific URL redirected back to the Apache instance of JIRA, at a special URL containing a cross site scripting (XSS) attack. The attack was crafted to steal the session cookie from the user logged-in to JIRA. When this issue was opened against the Infrastructure team, several of our administators clicked on the link. This compromised their sessions, including their JIRA administrator rights.

At the same time as the XSS attack, the attackers started a brute force attack against the JIRA login.jsp, attempting hundreds of thousands of password combinations.

On April 6th, one of these methods was successful. Having gained administrator privileges on a JIRA account, the attackers used this account to disable notifications for a project, and to change the path used to upload attachments. The path they chose was configured to run JSP files, and was writable by the JIRA user. They then created several new issues and uploaded attachments to them. One of these attachments was a JSP file that was used to browse and copy the filesystem. The attackers used this access to create copies of many users' home directories and various files. They also uploaded other JSP files that gave them backdoor access to the system using the account that JIRA runs under.

By the morning of April 9th, the attackers had installed a JAR file that would collect all passwords on login and save them. They then sent password reset mails from JIRA to members of the Apache Infrastructure team. These team members, thinking that JIRA had encountered an innocent bug, logged in using the temporary password sent in the mail, then changed the passwords on their accounts back to their usual passwords.

One of these passwords happened to be the same as the password to a local user account on brutus.apache.org, and this local user account had full sudo access. The attackers were thereby able to login to brutus.apache.org, and gain full root access to the machine. This machine hosted the Apache installs of JIRA, Confluence, and Bugzilla.

Once they had root on brutus.apache.org, the attackers found that several users had cached Subversion authentication credentials, and used these passwords to log in to minotaur.apache.org (aka people.apache.org), our main shell server. On minotaur, they were unable to escalate privileges with the compromised accounts.

About 6 hours after they started resetting passwords, we noticed the attackers and began shutting down services. We notified Atlassian of the previously unreported XSS attack in JIRA and contacted SliceHost. Atlassian was responsive. Unfortunately, SliceHost did nothing and 2 days later, the very same virtual host (slice) attacked Atlassian directly.

We started moving services to a different machine, thor.apache.org. The attackers had root access on brutus.apache.org for several hours, and we could no longer trust the operating system on the original machine.

By April 10th, JIRA and Bugzilla were back online.

On April 13th, Atlassian provided a patch for JIRA to prevent the XSS attack. See JRA-20994 and JRA-20995 for details.

Our Confluence wiki remains offline at this time. We are working to restore it.
What worked?
Limited use passwords, especially one-time passwords, were a real lifesaver. If JIRA passwords had been shared with other services/hosts, the attackers could have caused widespread damage to the ASF's infrastructure. Fortunately, in this case, the damage was limited to rooting a single host.
Service isolation worked with mixed results. The attackers must be presumed to have copies of our Confluence and Bugzilla databases, as well as our JIRA database, at this point. These databases include hashes of all passwords used on those systems. However, other services and hosts, including LDAP, were largely unaffected.
What didn't work?
The primary problem with our JIRA install is that the JIRA daemon runs as the user who installed JIRA. In this case, it runs as a jira role-account. There are historical reasons for this decision, but with 20/20 hindsight, and in light of the security issues at stake, we expect to revisit the decision!
The same password should not have been used for a JIRA account as was used for sudo access on the host machine.
Inconsistent application of one time passwords; We required them on other machines, but not on brutus. PAM was configured to allow optional use of OPIE, but not all of our sudoers had switched to it.
SSH passwords should not have been enabled for login over the Internet. Although the Infrastructure Team had attempted to configure the sshd daemon to disable password-based logins, having UsePAM yes set meant that password-based logins were still possible.
We use Fail2Ban for many services, but we did not have it configured to track JIRA login failures.
What are we changing?
We have remedied the JIRA installation issues with our reinstall. JIRA is now installed by root and runs as a separate daemon with limited privileges.
For the time being we are running JIRA in a httpd-tomcat proxy config with the following rules:

ProxyPass /jira/secure/popups/colorpicker.jsp !
ProxyPass /jira/secure/popups/grouppicker.jsp !
ProxyPass /jira/secure/popups/userpicker.jsp !
ProxyPass /jira http://127.0.0.1:18080/jira

Sysadmins may find this useful to secure their JIRA installation until an upgrade is feasible.
We will be making one-time-passwords mandatory for all super-users, on all of our Linux and FreeBSD hosts.
We have disabled caching of svn passwords, and removed all currently cached svn passwords across all hosts ast the ASF via the global config /etc/subversion/config file:

[auth]
store-passwords = no

Use Fail2Ban to protect web application login failures from brute force attacks

We hope our disclosure has been as open as possible and true to the ASF spirit. Hopefully others can learn from our mistakes.

Posted at 05:04AM Apr 13, 2010 by pgollucci in General | Comments[71]
Comments:

Thorough research, openness, good to see that we have a professional team taking care of infrastructure!

Posted by francisdb on April 13, 2010 at 06:36 AM UTC #
Minor spelling mistake = applcaiton should be application

Posted by Noam Rathaus on April 13, 2010 at 07:14 AM UTC #
I have seen on your JIRA site that you are using Enterprise Edition, Version: 3.13.5-#360, how about an upgrade? They include lots of security fixes as well, including locking user accounts after a number of failed log in attempts. This would actually have prevented part of the attack suffered. And definitely, the user with complete sudo access sharing password with JIRA, that was a big FAIL. Keep up the good work, and thank you for documenting this!

Posted by Blanca Garcia on April 13, 2010 at 09:15 AM UTC #
An interesting case where someone got root starting from XSS..Thank you for this elaborate explanation :)

Posted by Frederik on April 13, 2010 at 10:48 AM UTC #
I knew not to trust Atlassian. Check out this bug report I opened months ago for Jira.... The application should have been able to run under a security manager, however there existed a caught security manager exception deep in the code that prevented it running with security, they couldn't identify or explain it to me either... how odd ??? Atlassian's response was... "just turn the security manager off" So I didn't install it, instead insisted on installing it on it's own isolated VM. https://support.atlassian.com/browse/JSP-52900 Ha ha ha ha ha ha ha And people call me paranoid!!!!!

Posted by bryan hunt on April 13, 2010 at 11:06 AM UTC #
In response to this, I wrote a bit about how attacks like this one can be prevented by web applications, here: http://avatraxiom.livejournal.com/102080.html

Posted by Max Kanat-Alexander on April 13, 2010 at 11:37 AM UTC #
Thanks for a great incident report; I wish other orgs that suffered a breach will be that forthcoming with the whole set of details!

Posted by Anton Chuvakin on April 13, 2010 at 01:52 PM UTC #
[Trackback] Moi, Jirasta on löydetty kaksi haavoittuvuutta, jotka mahdollistavat käyttäjän session kaappaamisen XSShaavoittuvuuden kautta, ja hyökkääjän saatua haltuunsa administratortason oikeudet mielivaltaisten tiedostojen siirtämiseen palvelimelle,...

Posted by Confluence: Uutiset on April 13, 2010 at 02:32 PM UTC #
Great post, thanks. But, why explain in detail what your security measures are ?, let the hackers discover them....

Posted by Ariel Rivera on April 13, 2010 at 02:50 PM UTC #
I would like to see one-time passwords much more widely used, especially now that there are several software "tokens" available for mobile phones (toward this end I've written an Apache authentication module mod-authn-otp and an iPhone software token "OATH Token", both of which follow the RFC 4226 (OATH) standard). Use of two-factor authentication everywhere would have stopped this attack in its tracks. The problem is that even if you are ready and able with your own software token running the OATH standard algorithm, virtually no web sites allow you to enable requiring two-factor authentication for your account.

Posted by Archie Cobbs on April 13, 2010 at 03:11 PM UTC #
"By the morning of April 9th, the attackers had installed a JAR file that would collect all passwords on login and save them". The major issue there is that nothing was monitoring a restart of a JIRA instance with a delta in its runtime path, or a modification of key binaries in its runtime part in the file system - that correlated with a spike due to a brute force attack should have been spotted.

Posted by Alex on April 13, 2010 at 04:34 PM UTC #
What would have been another good thing, is that the mail you sent out would be signed with a key which we who receive these mails can validate.

Posted by David S on April 13, 2010 at 04:51 PM UTC #
[Trackback] El servidor de consulta / post de bugs de apache (basado en jira) ha sido hackeado entre el 6 y 9 de abril, y apache.org se ha visto obligado a enviar un correo a todos los usuarios registrados para que cambien sus contraseñas, informando de que la...

Posted by www.meneame.net on April 13, 2010 at 04:54 PM UTC #
So how do I know whether this is not a hacker still in control of the system, rushing us to change passwords to get them all in plaintext ;-)

Posted by mw on April 13, 2010 at 04:55 PM UTC #
Why do you say the Slicehost server was "compromised"? Could it not have been the attackers' own VPS (presumably purchased under false pretences)? If it really was compromised, was it the VPS itself which had been subverted, or Slicehost's systems?

Posted by Pepijn on April 13, 2010 at 05:56 PM UTC #
MW: "So how do I know whether this is not a hacker still in control of the system, rushing us to change passwords to get them all in plaintext ;-) " - My thoughts exactly!

Posted by Martin on April 13, 2010 at 05:58 PM UTC #
Expiring/locking accounts that haven't logged in for 30 or 90 days (and removing the password from your database) would of lessen the number of people affected by this attack. If you don't store it, it can't be stolen!

Posted by John on April 13, 2010 at 06:07 PM UTC #
did they steal any code ? or only passwords ?

Posted by free on April 13, 2010 at 06:24 PM UTC #
Great story, thanks for posting. If you are using FireFox you may want to install NoScript (noscript.net) to prevent XSS attacks!

Posted by nobody on April 13, 2010 at 07:12 PM UTC #
I'm a little impressed, makes me think even more that I must be very careful about XSS on my sites. I like the commitment to apache.org to publish it and explain it. Still gives me more confidence than before! good job

Posted by Sebastian on April 13, 2010 at 07:14 PM UTC #
Any chance of getting JIRA to authenticate using LDAP? At least that way they wouldn't have been able to get the password hash's for JIRA. Alasdair

Posted by Alasdair on April 13, 2010 at 07:17 PM UTC #
If the attackers got root on a shared shell server, and svn auth credentials there may have been cached, would it have been possible for the attackers to modify sources in the svn repositories?

Posted by Jan Schaumann on April 13, 2010 at 07:35 PM UTC #
A literary masterpiece!

Posted by Federico on April 13, 2010 at 08:21 PM UTC #
Yes, JIRA (and Confluence) can authenticate via LDAP, either directly or through Crowd (another Atlassian product), and in this case that that would have kept the attackers from obtaining password hashes, although it wouldn't have stopped the other exploits they achieved.

Posted by Kevin on April 13, 2010 at 08:51 PM UTC #
Extremely instructive. Thank you.

Posted by Ryan Brown on April 13, 2010 at 09:24 PM UTC #
Thank you for the transparency; I really respect it. It's also a very clear case as for why XSS should be treated seriously.

Posted by abadidea on April 13, 2010 at 09:36 PM UTC #
Leave it to big organizations to allow something this massive to occur un-noticed. It's why we have the stupid PCI standards we have today that do nothing but take the time out of businesses that always played by the security rules while the big guys were careless. Theres alot of blame and fingerpointing from who-ever wrote this but all the blame and fingerpointing should be pointing right at Apache. This attack had nothing to do with Linux, Slicehost, or whatever else is thrown in to tell a story. Who doesn't block brute force attacks in 2010? Who doesn't use real password encryption? Its mindblowing, but im not surprised the big guys always make a muck of things and then the little guys are stuck dealing with the aftermath.

Posted by James Woods on April 13, 2010 at 10:22 PM UTC #
No mention, anywhere, of how to delete my Apache JIRA account... come on!

Posted by Mark on April 13, 2010 at 11:13 PM UTC #
The attack would have been contained to brutus.apache.org's www directories if you were either running RHEL/CentOS/Fedora with SELinux enabled, or if the Debian/Ubutnu team would put down their bongs and actually implement one or two hardening mechanisms. The state of Linux security is abysmal these days. With no hardening and a new kernel exploit every week, a single XSS flaw these days usually gets you root. Windows 7 has far more hardening mechanisms than your average Linux distro, save maybe Fedora or hardened Gentoo. The only thing that saves us all from 100% compromise is that no one uses Linux. Just sayin' ; )

Posted by Frustrated Linux User on April 13, 2010 at 11:38 PM UTC #
There are so many other issue trackers out there, some of them much better and many more open than JIRA. Why don't you use one of those? Is not the first time when Apache.org get's hacked. Why don't you learn from your own mistakes and why don't you prevent this kind of attacks doing things the right way? What you said in your post makes me think that those that should take care of the security of your systems are not doing their job as they should. What took them so long to figure out; have you fired any of them?

Posted by Anonymous on April 14, 2010 at 12:07 AM UTC #
Some say that is not Ubuntu's fault, but I think it is. Browser without NoScript and sudo? Great pair, thank Ubuntu for this; sudo shouldn't be set the way Ubuntu does it. If the system wouldn't have had sudo setup the Ubuntu way, it wouldn't have been rooted. Full sudo access of that user that used the same password for local account and for administrative JIRA acount, should be revoked for good. Using same password for more accounts is just plain stupid; how many years long have this thing be shown and proven bad? Also, using https but encrypting only some of the content is just plain stupid.

Posted by Anonymous on April 14, 2010 at 12:07 AM UTC #
Well you fucked that one up didn't you, you useless fuckers! I'm not impressed.

Posted by James on April 14, 2010 at 12:09 AM UTC #
James (above) may not be impressed, but I believe this is pretty much a model response to a nasty attack (and I *really* hope James gets rooted real soon).

Posted by Steve Holden on April 14, 2010 at 12:53 AM UTC #
How do we request to have our accounts deleted from JIRA?

Posted by Michael on April 14, 2010 at 01:08 AM UTC #
To request that your JIRA account be deleted, please send an email request to infrastructure@apache.org.

Posted by Joe Schaefer on April 14, 2010 at 01:12 AM UTC #
How many private bugs did they gain access too by compromising the bug database systems? Also, why are you guys using proprietary software?

Posted by Jacob Appelbaum on April 14, 2010 at 01:32 AM UTC #
Wowwwwwwwwwww!!!Recently, I came across a hot dating site. —-** SeekInterracial [DOT] co-m **—- There are many sassy ladies and handsome rich guys seeking fun, friendship, love ,marriage and even more!!!!! Maybe you wanna check out or tell your friends.

Posted by ashley2012123 on April 14, 2010 at 01:38 AM UTC #
Thanks for sharing this. Any ideas on who the attackers could be?

Posted by Andrei Railean on April 14, 2010 at 01:39 AM UTC #
[Trackback] !Apache's Atlassian #JIRA system compromised https://blogs.apache.org/infra/entry/apache_org_04_09_2010

Posted by smartgeek on April 14, 2010 at 04:10 AM UTC #
[Trackback] !Apache's Atlassian #JIRA system compromised https://blogs.apache.org/infra/entry/apache_org_04_09_2010

Posted by remin on April 14, 2010 at 04:10 AM UTC #
how was the compromised slicehost node essential to this hack? couldn't anyone have opened the jira that had the tinyurl that led to the XSS attack that led to everything else?

Posted by Jay on April 14, 2010 at 04:47 AM UTC #
> Hopefully others can learn from our mistakes. Yup, don't use Jira is best lesson from this mistake. In 2010 it could have support for OpenID.

Posted by Sergey Shepelev on April 14, 2010 at 05:56 AM UTC #
Thanks for the transparency on the whole story behind this. A lot to learn from this: the power of XSS, obfuscated URLs and social engineering... And how easily security precautions can be circumvented by not doing due diligence in an administrative role. Hopefully you'll be just as transparent about the measures taken to prevent such a failure in the future. :-)

Posted by Zeke Weeks on April 14, 2010 at 06:07 AM UTC #
i just want share this link. http://tinyurl.com/preview.php?enable=1 http://is.gd/setcookie.php it's will expand the short url before it's visited. #thinkbeforeryouclick #apacheisgreat

Posted by fakhrime on April 14, 2010 at 06:50 AM UTC #
ASF running ubuntu on their servers...LOL. During years working with big companies it's the first time i hear such thing, an organization as big as ASF running ubuntu on their servers. Don't tell the people here that it was running kde and have doom 3 and all the games installed too.

Posted by Wilson on April 14, 2010 at 07:00 AM UTC #
looks like we have a problem with the email addresses. '@' and '.' have been replaced by 'at' and 'dot'.

Posted by Jens on April 14, 2010 at 07:26 AM UTC #
So do you recommend IBM's AIX instead Ubuntu?

Posted by Jeff on April 14, 2010 at 08:29 AM UTC #
Long live ASF! Great incident report; I wish other orgs that suffered a breach will be that forthcoming with the whole set of details!

Posted by Fernando on April 14, 2010 at 10:12 AM UTC #
Hey Wilson, I think you were indeed working with big companies for years... as the lobby shoe polish boy! Hav you heard of Wikimedia Foundation? How about Wikipedia? Well you ignorant, they use Ubuntu for many of their servers.

Posted by Fernando on April 14, 2010 at 10:17 AM UTC #
[Trackback] The Apache Software Foundation was recently the victim of a targeted attack, which they’ve detailed at great length. It’s a fascinating read, and somewhat depressing, and also awesome in a completely evil way, all at the same time. For the...

Posted by genehack.org on April 14, 2010 at 10:37 AM UTC #
Me too. I was pretty close to buying a slice last night too. But now I'm slightly worried that if a large organization like Apache waited two days for action, how long would a small fry like me wait?

Posted by haley on April 14, 2010 at 11:39 AM UTC #
You might add more testing against all installed 3rd party application such as JIRA, I wrote a short blog post about it. http://www.mavitunasecurity.com/blog/apacheorg-and-jira-incident/ I already send the email but want repeat in here we would be happy to give a free Netsparker license to Apache Software Foundation. Also we do have a free version of Netsparker - Community Edition which would have identified these XSS issues as well - http://www.mavitunasecurity.com/communityedition/

Posted by FM on April 14, 2010 at 12:17 PM UTC #
use FireFox with NoScript or IE8 with default XSS filter enabled

Posted by PaPPy on April 14, 2010 at 12:33 PM UTC #
[Trackback] Read about the latest security breach on Apache and JIRA: http://www.theregister.co.uk/2010/04/13/apache_website_breach_postmortem/ http://blogs.atlassian.com/news/2010/04/oh_man_what_a_day_an_update_on_our_security_breach.html https://blogs.apache...

Posted by JIRA: Kafahali on April 14, 2010 at 12:41 PM UTC #
Hi, This was very useful. Consider adding multi-factor authentication with client certificates that you can issue to admin members, instead of relying soley on password based solutions (two factor auth). Consider also requiring VPN with multi-factor authentication as a requirement to reach your infrastructure related configuration. Even from inside your infrastructure.

Posted by Fred Schlip on April 14, 2010 at 01:26 PM UTC #
[Trackback] This issue affects only the the passwords of atlassian client accounts, it doesn't affect any passwords of any users on our own Jira installation. There is a XSS vulnerability that was used to compromise the apache software foundation's jira insta...

Posted by JIRA: IT Systems on April 14, 2010 at 01:29 PM UTC #
"We use Fail2Ban for many services, but we did not have it configured to track JIRA login failures." Is this even possible? If so, how are you doing it ASF?

Posted by JD on April 14, 2010 at 02:53 PM UTC #
Apache and Linux have been fooled? Another account goes, the game really opens up and says it was a human error.

Posted by IcE on April 14, 2010 at 03:12 PM UTC #
«"We use Fail2Ban for many services, but we did not have it configured to track JIRA login failures." Is this even possible? If so, how are you doing it ASF?» I have no experience with JIRA but fail2ban works by matching lines in log files against user-defined regular expressions, so it should be able to work with most log files out there as long as you manage to write the required regex. You can use the fail2ban-regex command to test your regex on existing log files.

Posted by Peter Odding on April 14, 2010 at 04:48 PM UTC #
[Trackback] zomigi.com » Deal-breaker problems with CSS3 multi-columns The first problem is how the browser should handle one or more extra lines of content if the amount of content cannot fill up each column equally. In my opinion, the extra...

Posted by msj on April 14, 2010 at 05:03 PM UTC #
[Trackback] There have been two recent exploits against JIRA and Atlassian, which, though they have no direct effect on us, should be kept in mind as we consider opening JIRA to support integrations with other tools. An XSS attach against https://issues.apache....

Posted by Confluence: Jonathan Stern on April 14, 2010 at 05:17 PM UTC #
The coordinated multi-vector attack is what I find most interesting. XSS, Bruteforce, jsp backdoors, phishing, social engineering. And why would the attacker(s) spend so much effort on attacking ASF? To find all those private bugs? I doubt it. Get the source code? oh wait, it's already open source. The only real value I see in the attack is the passwords. How many apache contributors also work for the DOD or other companies with valuable intellectual property? I bet the attackers got paid for this attack. Either contracted out by a firm, or on the payroll for a government.

Posted by Lance R on April 14, 2010 at 05:20 PM UTC #
the question is: why should a person hack apache's server knowing that this is an open source community things? If it for learning purpose, the person should not attack the site till it break(just browsing through things is fine i guess). if the person who hacked the server is a good samaritan, s/he should raise the security issue to apache team

Posted by lou martin on April 14, 2010 at 05:31 PM UTC #
The target was not the passwords. They were going after the SVN server, and since the source is all public there, it wasn't to see it and file bug reports containing patches. Given past attacks against the Linux source, the likely goal was to put a back door into HTTPD, the https support, or similar, something that lets you either onto the sites running the daemon, or that makes encrypted communication easier to read.

Posted by SteveL on April 14, 2010 at 07:07 PM UTC #
Don't you shame some kind of shame for saying: "The attackers had root access on brutus.apache.org for several hours, and we could no longer trust the operating system on the original machine." Are you unable to ensure that a given machine is secure or not?

Posted by 127.0.0.1 on April 14, 2010 at 08:01 PM UTC #
sign....

Posted by nabice on April 15, 2010 at 04:54 AM UTC #
Nice work (- and report), and a nice display of preventing a spread (but a shame that the reciver did not act in time).

Posted by Bjorn on April 15, 2010 at 06:55 AM UTC #
When are you going to require that salt is used for all Apache site password hashes (possibly by going to LDAP authentication)?

Posted by Richard on April 15, 2010 at 07:14 AM UTC #
You should learn more about OpenSSH sshd configuration and do not put wrong statements like "having UsePAM yes set meant that password-based logins were still possible". There is nothing wrong with UsePAM yes, but you probably did not set both PasswordAuthentication and ChallengeResponseAuthentication to no.

Posted by Tomas Mraz on April 15, 2010 at 09:26 AM UTC #
[Trackback] In the spirit of openness the Apache foundation has released an excellent post mortem write up of their recent compromise. It started with a XSS attack leveraged through the issue tracking software they use (JIRA) and ended with complete root...