This story starts at the console debugging a coworker’s issue. They had gotten stuck trying to
bundle install. Try, and fail. Try, and fail. And the error was really weird:
We went through the normal steps: turn it off and on again, check wifi, load up google, go for a coffee , etc. Standard debugging techniques.
Finally we realized the
y had inadvertently been dropped in
Huh. Go figure. We moved on our way after giggling a little. Rub gems. Hehe.
Then I got to thinking. If my colleague had made this mistake, surely other people have made this error too. I checked the whois record, and it was available! take_my_money
I was now in possession of
rubgems.org, and was left with the question: What can I do with
it? Which lead me to the logical conclusion: I wonder if I can Man in the Middle
rubygems.org and see
if other people make this typo!
It was stupid easy to setup a MITM. Grab a tiny AWS box, configure DNS, setup an Nginx proxy, and add let’s encrypt to the box. The Nginx config looked like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
And that was it. I cobbled together some commands to build a simple log file parser to send me a list of unique IPs each day via email, and then I sat back and waited.
What Does a RubyGems MITM Get You?
While I waited, I investigated and tried to determine: What is possible if you’ve got control of a rubygem?
To answer this question I created a
trojan_horse gem. You can see it here on
The rough contents of which were:
1 2 3 4 5
Then I made a modification of the gem and added my MITM code at the top:
1 2 3 4 5 6 7
I put this gem onto my MITM’d server, and tried to download it through various means.
Let’s see what happens with RubyGems. I started looking at
gem install and set the
here’s what got downloaded:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Huh interesting. It reaches out to
rubygems.org and then
rubgems.org, but the download prefers to use
rubygems.org over my MITM. Wonder why? Checking out the help docs solves that problem:
1 2 3
From this it looks like the url is appended to the end such that if the gem doesn’t exist on
only then will it reach out to a different source. No MITM possibility there!
Fresh Installation with Bundler
Next I tried my hand at bundler. I started by creating the following Gemfile entry:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
As you can see from the
curl output, having never downloaded the gem before, I have RCE through the gem.
Reinstall with Bundler
With the bundler reinstall, I began by uninstalling
trojan_horse, and updating my
Gemfile to the following:
And then ran
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
As you can see the gem downloaded cleanly from
rubygems.org, without a trojan. Next I updated the
Gemfile back to
rubgems and ran
bundle install --verbose:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
And herein I got caught! Bundler does a checksum comparison on the gem verses the server. But if I control
the server, shouldn’t I be able to control the checksum? I checked what
1 2 3 4 5 6 7
Those are different, but they don’t match the checksum failure I was getting. Things actually got confusing at this point. And when that happens, I jump into the source code and try to get a debugger around what’s happening.
At this point with a debugger around the code, I can start to look at the two variable values:
local_temp_pathis a path to a local cache of the
/info/trojan_horsefile; on my local that path looks like this:
1 2 3
response_etagis the ETag value from the request sent to rubgems for
/info/trojan_horse. That value came back as:
First mystery solved I found one of the checksums. It’s a partial MD5 hash coming from ETag confused-cat
I didn’t update this, because I really think the confused-cat meme is hilarious and wanted to use the emoji.
Still digging, I followed the method
etag_for and ended up at this code:
1 2 3 4 5 6 7 8 9 10 11
That makes more sense! Bundler is doing an MD5 hash against the contents of the file returned at
/info/trojan_horse. The MD5 hash for that file looks like this:
1 2 3 4
And this is the second piece. The ETag is not matching with what I have on the local cache and that’s why the installation is failing. Makes sense. But I control the server, so I can modify the ETag! All that’s expected from the bundler standpoint is an MD5 checksum that matches the contents of the file.
I switched up my Nginx configuration so that when is requested
/info/trojan_horse it reverse proxies to
a local ruby server and sends back the MD5 hash it was expecting.
And viola! The backdoored gem installed. I was actually shocked at this point. I expected additional safety checks to break, and that wasn’t the case. The lesson here is if you own the server, the client can’t do anything to save itself from you.
To summarize my findings:
- There’s no possibility of MITM against
- And I have full MITM with RCE against
Back to the Story…
After about 3 months I tallied up my stats. I had a collection of around 100 IPs. The geographical layout looked like this:
1 2 3 4 5 6 7 8 9 10
Not too shabby. You’ll noticed that half of those connections are for AWS. Which should give you pause, because it likely means my MITM was running on a staging or production server. Yikes! At this point I felt like my little experiment had run its course and demonstrated that this was a viable method for running a rubygems MITM.
I wasn’t entirely sure how to properly disclose this info. It was sensitive. But also, kinda not really. Especially since I owned the domain in question. But that said, there could be other domains that could be squatted so I felt it best to start quietly.
I started by submitting a report to HackerOne against the RubyGems program. If I could get paid for this, why not! Rejected. I wasn’t surprised with that. I asked to publicly disclose … it waited for a month until I bumped it, the H1 staff replied “we’ll let you know once we have info” and then closed it. I took that as: “Here is the door, please use it.” man_shrugging
The next avenue I tried was sending an email to rubygems’ security email:
firstname.lastname@example.org. I heard
back within 2 hours, and then nothing. I sent a few follow up emails, but didn’t hear back.
I happened to bump into RenderMan at BSides Edmonton and chatted
with him about the MITM and lack of response. He
fired off a tweet to see if he could get me
a response. Nada.
Finally, I spoke to a colleague that had a connection on the rubygems team. This eventually lead to a response back from my initial email with instructions to file an issue. Progress!
Additionally, since this issue affected bundler, I sent
email@example.com an email. That bounced back
as undeliverable. Guess that email doesn’t exist. And sent an email to
firstname.lastname@example.org and never got a
That said, Bundler team if you’re reading this … please fix your security email address!
At this point, I submitted issues with the rubygems project and bundler project, and fired up the ol’ text editor to write this post.
This MITM turned out to be a lot of fun to run through. While it wasn’t a super serious issue, had I been a malicious actor, I could have RCE on a bunch of computers right now so… Achievement unlocked … Kinda! trophy