Hacks Prompt U.S. to Establish New Cybersecurity Agency
February 10, 2015
GRR Live Memory Analysis
March 8, 2015

GRR Rapid Response Server Build Out

I’ve been spending a significant amount of time looking at endpoint solutions recently. My goal is to have an IR Swiss Army Knife and hunting platform as well as a tool that can perform random ad-hoc queries when the latest cyber crisis hits. There are a lot of tools out there that can do bits and pieces of this, and a few less than can do pretty much everything…but they cost a fortune. In this post I’ll discuss my experiences so far with a GRR Rapid Response Server Build Out.

I stumbled upon GRR a while back and have played with it off and on for the past 6 months. I’m very happy with the platform as it is now, but as the developers describe it themselves, “…(GRR is an) 80% written software project that you could invest in.” I see huge potential in this project and it’s taken leaps and bounds even since I started playing with it 6 months ago.

GRR is a cross-platform endpoint solution that consists of a server and agent and was designed with security in mind from the start. It is capable of performing most (if not all) tasks you’d need during an incident response including gathering detailed information about processes, the file system, registry, Internet history, network connections, and many more artifacts. GRR can gather information from a single host or the entire enterprise quickly (if scaled properly). It can also be used to collect items of interest such as running binaries or prefetch files, for examples.

The version distributed with the quick start script (Quick Start page) is 3.0.0.2. I haven’t had any serious issues with this version, but the project has moved it’s live memory functionality from Volatility to Rekall. Since this move, I’ve had very little success with live memory forensics. This is the only issue I’ve had, and it may verywell be me or my build out. I don’t know yet.

Sometime around the OpenSSL insanity late last year the GRR project release version 3.0.0.3. Because of the changes in OpenSSL I had a lot of issues with GRR as most of my hosts were running 3.0.0.2 and 3.0.0.3 wouldn’t compile the CentOS package correctly. CentOS functionality is a hard requirement for my environment, so GRR sat on the bench until recently when I noticed version 3.0.0.5 was released and I decided to give it more cycles.

As stated in the GRR release notes, Rekall made some fundamental changes which then required a client upgrade to 3.0.0.5. After working with the development team & user community, who are absolutely awesome, I was able to get all clients repacked with version 3.0.0.5 and upgraded the server to the bleeding edge. I still have problems with anything related to rekall or live memory, but we’re making progress. In my memory testing thus far, I can successfully dump a full memory image of a Windows host as long as I write it to disk first, but that’s about it. OS X hosts crash ~500MB into the dump and Linux complains about the lack of a memory driver, which I’m pretty sure it doesn’t need. All other functionality of GRR is working fine.

Either way, below I’ve shared the bash scripts I’ve been using to build out servers for testing. As is, this will run the GRR projects quick start script which will install all dependencies and the GRR server at version 3.0.0.2. It will then download the 3.0.0.3 clients, repack them, and upgrade the server to the git HEAD (bleeding edge). I’ve also written a script to dump the database and update everything to version 3.0.0.5. I’ve only tested this second script on systems that I’ve built out with the first script. No guarantees it’ll work on preexisting setups.

Lastly, a couple of things to keep in mind.

  • If you already have clients in the network, do not lose/overwrite the crypto keys. Worst case, you can simply use DNS to redirect these clients to a new server. As long as a server has the correct keys you can recover current clients without reinstalling them (ie: if you break the current server and have to build out a new one).
  • Out of the box, the backend is not amazing and cannot likely support your whole network. I’m working on this myself and will make another post later once I find a solution that works well for my environment. Currently available in the code is Mongo (default), MySql, sqlite, tdb, and HTTP. Some of the GRR guys wrote this article on using a distributed datastore with sqlite, but I know almost nothing about sqlite so that’s not my first choice.
  • There are several security considerations. First things first, enable SSL on the AdminUI then change the admin password…especially if it is Internet accessible.

Ubuntu Base Image to GRR 3.0.0.3 Script

##################################################
# GRR Server Installation Script
# Server Version: HEAD (4f66da8faa)
# Client Version: 3.0.0.3
#
# Tested with Ubuntu 14.04
# AWS AMI ami-9a562df2, t2.medium
#
# Written by Brian Olson (brian@hurrikane.net)
# February 13, 2015
##################################################
#Use root shell
#!/bin/sh
#Run GRR quickstart script
wget https://raw.githubusercontent.com/google/grr/master/scripts/install_script_ubuntu.sh
chmod 755 install_script_ubuntu.sh
sed -i 's/run_cmd_confirm grr_config_updater initialize/#run_cmd_confirm grr_config_updater initialize/' ~/install_script_ubuntu.sh
./install_script_ubuntu.sh
# * Answer the script questions
# Comment out creating 3.0.0.2 clients we're just going to blow away anyhow
/usr/share/grr/scripts/database_reset.sh
#download 3.0.0.3 client templates
wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.3/Linux/grr-client_3.0.0.3_amd64.rpm.zip -P /usr/share/grr/executables/linux/templates
wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.3/Linux/grr-client_3.0.0.3_amd64.zip -P /usr/share/grr/executables/linux/templates
wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.3/Linux/grr-client_3.0.0.3_i386.zip -P /usr/share/grr/executables/linux/templates
wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.3/OSX/grr-client_3.0.0.3_amd64.template -P /usr/share/grr/executables/darwin/templates
wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.3/Windows/grr-client_3.0.0.3_amd64.zip -P /usr/share/grr/executables/windows/templates
wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.3/Windows/grr-client_3.0.0.3_i386.zip -P /usr/share/grr/executables/windows/templates
#Update the client version in the grr server config
sed -i 's/Client.version_release: 2/Client.version_release: 3/' /etc/grr/grr-server.yaml
source /usr/share/grr/scripts/shell_helpers.sh
# Disable creation of i386 clients
sed -i 's/Arch:i386:/#Arch:i386:/' /etc/grr/grr-server.yaml
sed -i 's/ Client.arch: i386/# Client.arch: i386/' /etc/grr/grr-server.yaml
#Disable creation of debug clients
sed -i 's/ DebugClientBuild/# DebugClientBuild/' /etc/grr/grr-server.yaml
#sed -i 's/ Logging.verbose/# Logging.verbose/' /etc/grr/grr-server.yaml
sed -i 's/ ClientBuilder.console/# ClientBuilder.console/' /etc/grr/grr-server.yaml
sed -i 's/ Client.poll_max/# Client.poll_max/' /etc/grr/grr-server.yaml
sed -i 's/ Client.foreman_poll_frequency/# Client.foreman_poll_frequency/' /etc/grr/grr-server.yaml
sed -i 's/ Client.rss_max/# Client.rss_max/' /etc/grr/grr-server.yaml
sed -i 's/ Nanny.unresponsive_kill_period/# Nanny.unresponsive_kill_period/' /etc/grr/grr-server.yaml
sed -i 's/ Client.prefix: dbg_/# Client.prefix: dbg_/' /etc/grr/grr-server.yaml
#Reinitialize the grr server
sed -i 's/#run_cmd_confirm grr_config_updater initialize/run_cmd_confirm grr_config_updater initialize/' ~/install_script_ubuntu.sh
grr_config_updater initialize
#Update GRR Server to HEAD
aptitude -y install git
git clone https://github.com/google/grr
cd grr
mkdir /usr/lib/python2.7/site-packages/
# I dont' know why I have to install with and without the prefix (twice), but this makes it work :)
python setup.py install
python setup.py install --prefix=/usr
/usr/share/grr/scripts/initctl_switch.sh restart
# Ta Da! Done - Server is at HEAD and clients built to 3.0.0.3
#
# Log into the server @ http://IP:8000/
#
# Known issues @ https://github.com/google/grr-doc/blob/master/releasenotes.adoc
# Mainly -- rekall doesn't work (AFAIK)

 GRR 3.0.0.3 to 3.0.0.5 Upgrade Script

#!/bin/sh
 # Reset the current database
 /usr/share/grr/scripts/database_reset.sh
 #Download 3.0.0.5 client templates
 wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.5/Linux-prebuilds/grr-client_3.0.0.5_amd64.rpm.zip -P /usr/share/grr/executables/linux/templates
 wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.5/Linux-prebuilds/grr-client_3.0.0.5_amd64.zip -P /usr/share/grr/executables/linux/templates
 wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.5/Linux-prebuilds/grr-client_3.0.0.5_i386.zip -P /usr/share/grr/executables/linux/templates
 wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.5/OSX-prebuilds/grr-client_3.0.0.5_amd64.template -P /usr/share/grr/executables/darwin/templates
 wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.5/Windows/grr-client_3.0.0.5_amd64.zip -P /usr/share/grr/executables/windows/templates
 wget https://744592537a0751a28dba27df4a20131058ff8e4d.googledrive.com/host/0B1wsLqFoT7i2N3hveC1lSEpHUnM/Current/Client/3.0.0.5/Windows/grr-client_3.0.0.5_i386.zip -P /usr/share/grr/executables/windows/templates
 # Update the server config version
 sed -i 's/Client.version_release: 3/Client.version_release: 5/' /etc/grr/grr-server.yaml
 # reinitialize GRR Server
 grr_config_updater initialize
/usr/share/grr/scripts/initctl_switch.sh restart

Leave a Reply

Your email address will not be published. Required fields are marked *