G2 Scoring

A digital trace scoring system for IoC vetting

Developed by
Christoph Giese
Jan Göbel

Agenda

  1. Introduction / Background
  2. The scoring system
  3. Evaluation
  4. Conclusion / Future work

Who am I?

Christoph Giese

BSc IT dual@telekom --> MSc digital forensics (this is the topic ;))
System Engineer --> CERT/CDC
Working at this topic collecting, filtering and enriching IoCs for security monitoring

Introduction

Challange 1/4


Growing risk of cyber threats


with higher complexity

Growing risks

Victims Cybercrime: 47% of the private internet users
Victims Cyber Espionage: 2 of 3 german companies
A Massive incident of data fraud/theft is one of the Top 5 Global Risks
(others: large-scale involuntary migrations, major natural disasters, extreme weather conditions and terrorist attacks)

Higher complexity

Hashing algorithms are Insufficient
Big malware campaigns are running explosive

Challange 2/4


Amount of digital traces too analyze

is growing fast


Growing Haystack; More Needles

Growing Haystack

IT evolution Security effect
Cheap data storage Increased amount of produced (and logged) traces
Better connectivity More and more logging centralization
Increased computing power Makes it searchable

Growing Haystack (cont.)

  • Improved detection capabilities in AV
  • Endpoint Agents (e.g. GRR)
  • Incident Response (e.g. HBIV/Redline)
  • Web-Traffic Sandboxes (e.g. FE-NX)
  • SIEM(s)
  • [...]

More Needles

Security evolution Estimated amount
Sharing groups Dozens of sector-, revenue-, country-specifc groups
Advisories from public authorities
(e.g. Federal Intelligence Service)		 A few advisories p. Y. with > 100 IOCs each
Growing market of free feeds > 100 feeds with >10(0)k IOCs p.D.
Growing market of paid feeds {x} actors with > {y} employees in the market

More Needles (cont.)

Evolution Estimated amount
Own automated quarantine sandboxing >100 IOCs p. D.
Own reverse engineering Depends on the malware researcher
Own forensic cases Depends on the forensic guys
Own hunter teams 404
Own data analytics >100 p.M.

Challange 3/4


Heterogenus


networks, butgets and skills

The companies

infrastructure, money and know-how

decides which detection capabilities are available.

Challange 4/4


High False-Positve rate


using free IoC sources

Motivation

“The growing amount of data
and the high false-positive rate using IoCs in security controls
raises the need for an automated IOC vetting system,
which takes the heterogeneous structure of a computer network into account.“

Goals

for an automated IOC vetting system

  • Improved usage of open source intelligence
  • Correlate information to provide a fast overview
  • Lower False-Positive rate


    • Do it autonomously

    The scoring system

    Design

    How is the scoring system generally working?
    In general

    Ask questions like an security analyst would do!

    More specific

    How can I answer these questions systematically?

    Formula

    How to weight the factors?

    $S_{Total}(t) = \sum\nolimits_{i=1}^{N} S_Q(q_i,t)$

    where $S_{Total}(t)$ is the total score of a trace $t$ and $S_Q(q,t)$ the score of a particular question $q$ for trace $t$.
    With $i$ iterating over $N=no. of questions$ the total score is the sum of all question scores.

    Each question score $S_Q(q,t)$ depends on three factors:

    1. The weight of the question $W_Q(q)$
    2. The data sources evaluation score $S_{D}(t,q)$
    3. The detection capabilities evaluation score $S_{C}(t,q)$


    $S_Q(q,t) = W_Q(q) * S_{D}(t,q) * S_{C}(t,q)$


    $W_Q(q)$

    The question weight



    $W_{Pre}(q)$ Precision Of the answer
    $W_{Sec}(q)$ Security impact
    		 If the answer is yes
    $W_{Fap}(q)$ False-positive amount Expected


    $W_Q(q) = W_{Pre}(q_i)\frac{W_{Sec}(q_i)}{W_{Fap}(q_i)}$


    $S_D(t,q)$

    Data sources evaluation score



    $S_{Time}(q,d)$ Age of information If important for question $q$
    $S_{Rel}(q,d)$ Reliability Quality history of $d$
    $S_{Con}(q,d)$ Information content Confirmed information better


    $S_D(t,q) = \max_{k=1}^N \big(\frac{S_{Time}(q_i,d_k)}{\sqrt{(S_{Rel}(q_i,d_k) * S_{Con}(q_i,d_k))}}\big)$

    with $N=\text{No. of data sources}$


    $S_{C}(t,q)$

    Detection capability score

    $S_{Tec}(c,q)$ Techniques Is hacking technique $_{Tec}$ behind $q$ answerable in capability $c$?
    $S_{Typ}(c,t)$ Types and formats
    		 Is $c$ capabable of capturing traces from same type as $t$?
    $S_{His}(c,q)$ Availability of Historically data Can we search for it in the past?
    $S_{Vis}(c,q,t)$ Visibility Placement of the system ($S_{Int}$),
    Applied (protocol/port) filters on interface capture ($S_{Ftr}$),
    Richness of captured surroundings ($S_{Sur}$),
    Additional information ($S_{Adi}$)


    $S_{C}(t,q)=\max_{p=1}^N \big(\frac{S_{His}(c_p,q_i)}{S_{Tec}(c_p,q_i) * S_{Typ}(c_p,t)} * \underbrace{\frac{\frac{S_{Sur}(c_p,t)}{S_{Adi}(c_p,t)}}{S_{Int}(c_p,q_i) * S_{Ftr}(c_p,q_i,t)}}_{S_{Vis}}\big)$

    with $N=\text{No. of detection capabilities}$

    our scoring formula

    $ S_{Total}(t) = \sum\nolimits_{i=1}^{N} \bigg(\overbrace{W_{Pre}(q_i)\frac{W_{Sec}(q_i)}{W_{Fap}(q_i)}}^{W_Q(q_i)}* \underbrace{\max_{k=1}^N \big(\frac{S_{Time}(q_i,d_k)}{\sqrt{(S_{Rel}(q_i,d_k) * S_{Con}(q_i,d_k))}}\big)}_{S_{D}(t,q_i)} * \\ \underbrace{\max_{p=1}^N \big(\frac{S_{His}(c_p,q_i)}{S_{Tec}(c_p,q_i) * S_{Typ}(c_p,t)} * \frac{\frac{S_{Sur}(c_p,t)}{S_{Adi}(c_p,t)}}{S_{Int}(c_p,q_i) * S_{Ftr}(c_p,q_i,t)}\big)}_{S_{C}(t,q_i)}\bigg) $

    Implementation

    How to build a system which can do this autonomously?

    Lets see if this works in the

    evaluation

    Evaluation

    Comprehensive evaluation

    Feed name IPs in phase 1 IPs in phase 2
    ZeusTracker 132 132
    RansomwareTracker 0 297
    Bambenek All IPs 399 0
    Bambenek High Confidence 112 89
    US CERT - Hidden Cobra 0 627
    malc0de 0 213
    APTNotes 2017 0 224

    Comprehensive evaluation

    Phase Timeframe IP addresses Alarms generated Alarms investigated
    1 2017-11-15 - 2017-11-22 (7d) 640 61,442 25,777
    2 2017-11-22 - 2018-02-17 (87d) 1580 110,912 21,733
    SUM 2017-11-15 - 2018-02-17 (94d) > 1750 172,354 47,510

    Results

    Overwhelming amount of false-positives


    Phase 1 | Phase 2

    A few feeds producing the majority of alerts


    Phase 1 | Phase 2

    A few IP addresses producing a lot of alerts


    Phase 1 | Phase 2

    How good is our scoring system able to filter the false-positives?


    Phase 1 | Phase 2

    98% of all false-positives

    have been filtered out using our scoring system

    No case

    in which our system would have classified a high quality indicator as unsuitable

    98% of all false-positives

    What does this mean for the SoC analyst?

    • 47,510   alerts = 141 hours (11 seconds per alarm)
    • 172,354 alerts = 512 hours

    • 45,124 of the 47,510 alarms classified as false-positive
    • False-positive ratio=95%

    • 44,440 of these 45,124 false-positives would have been pre-filtered by using our scoring system.
    • Successful filter rate of 98.5%

    What does this mean for the SoC analyst?

    By using our scoring system,

    • 135.8 of 141 hours would have been avoided
    • All alarms: Saved time is even 500.3 of 512 hours


    Analyst can spend time on in-depth analysis of high-quality alarms

    Summary

    Unfiltered use of standard-malware feeds not recommended


  • Outdated data (partly from 2016)

  • Ineffective filtering of multihosters

  • Expired / Parking IPs

    • Challenges in alarm classification

    Attackers adapt!


  • Traffic Distribution Systems

  • Encryption

  • Domain Fronting (hide in plainsight)

    • IoC quality assurance is tough, but can work


  • Systematically approach

  • Automation means elevation

  • Spent the limited time as good as possible!

    • Future work

    Enhancements


  • More data sources

  • Improve weighting formula

  • Improve self-written heuristics

    • Extensions

  • Protect other types of computer networks

  • Implement more question (e.g. is scanner)

  • Advanced detection goals / More IoC types

    • Advanced detection goals / More IoC types


    Strong fundament --> Lets go to the top!
    Source

    Thanks!

    Q?

    bit.ly/g2score
    bit.ly/cgiese

    Licences


    Open source material is great stuff. So the content of this presentation is published under Creative Common License as well.
    Which means you can share and adapt it, but you need to attribute my name (Christoph Giese) and a link to this presentation (https://cgi1.gitlab.io/g2scoring-presentation/).
    Feel free to work with it and let me know your thoughts about it.

    ------
    Images
    Most of the diagrams and stuff are self-made (CC license like above).

    The icons/logos are from flaticon.com published under CC license as well: Stackoverflow, Github, Linkedin, Xing, GCFA, GCNA.

    The background images are from flickr.com published under CC license as well: knitting needles, Fun With Bike Lights 2, The Beauty of Mathematics, SIS building, Long road, Green, Money.

    Good IoCs

    IOC

    “Indicator of compromise (IOC) — in computer forensics is
    an artifact observed on a network or in an operating system that with high confidence indicates a computer intrusion.”

    "An artifact

    ...observed on a network or in an operating system that with high confidence indicates a computer intrusion."
    "An artifact...

    observed on a network or in an operating system

    ...that with high confidence indicates a computer intrusion."

    #1

    The needle needs to be in the haystack.

    (Do not forget to take traceability, manipularity and persistance into account!)
    "An artifact observed on a network or in an operating system...

    that with high confidence

    ...indicates a computer intrusion."

    High confidence?

    #2

    The minimal confidence
    depends on the use-case.

    "An artifact observed on a network or in an operating system that with high confidence...

    ...indicates a computer intrusion."

    A computer intrusion

    There are dozens of ways to classify them
    (I will not explain them here)
    (And I do not know them all)

    Instead :

    Simply let us take a look on how an IoC gets born.

    ALERT ON APT-IOC

    What happens now?

    What if...

    185.29.8.211 is a TOR exit node?

    The threat-level does not fit!

    #3

    If an IOC is not threat-specific,
    we are wasting resources.

    C:\Users\peter.mueller\AppData\Temp\weorbwoier.docx

    Good IOC?

    Better:

    C:\Users\%USERNAME%\AppData\Temp\weorbwoier.docx

    Better-Better:

    %APPDATA%\Temp\weorbwoier.docx

    (environment specific values)

    C:\Windows\system32\bad.dll

    Good IOC?

    Better:

    C:\Windows\system32\bad.dll

    C:\Windows\SysWoW64\bad.dll

    (environment specific values: os-specific values)

    C:\Windows\system32\QWIEHQIOWEHQ2E3H4IQOWE.dll

    Good IOC?

    Better?:

    C:\Windows\system32\{32-DIGITS_ENTROPY>0.8}.dll

    C:\Windows\SysWoW64\{32-DIGITS_ENTROPY>0.8}.dll

    (detect auto-generated values)

    #4

    Try to normalize/generalize system/runtime specific attributes.

    Summary

    So for now: The best IOC (needle)...

    • #1 ...could be [technically found] in my input data.
    • #2 ...has the [use-case related confidence] to deal with the hits.
    • #3 ...needs to be [threat-specific] to generate the best detection rate on same effort.
    • #4 ...has the best [system/runtime independent] verifiability.

    Real-World Examples

    Input Malc0de (Domains only)
    Date July 2017
    Use-Case Rate potential IoC list
    Detection capabilities DNS-Logs / FW-Logs
    Required confidence level Medium (Goal: Alert potential infections in SIEM)

    How to resolve it in the scoring system?

    $S_{Total}(t) = \sum\nolimits_{i=1}^{N} S_Q(q_i,t)$

    $t=$ 2.5.72.15
    $q=$ 'is_well_known_ip'

    $S_Q(q,t) = W_Q(q) * S_{D}(t,q) * S_{C}(t,q)$

    $W_Q(q)= -50$

    $S_{D}(t,q)=\frac{1}{\sqrt{(1 * 3)}}$ (using $ P(q,d)=$$\left\{\begin{array}{@{}ll@{}} \frac{S_{Time}}{\sqrt{(S_{Rel} * S_{Cre})}}, & \text{if the data source }d \text{ can answer question }q \\ 0, & \text{else} \\ \end{array}\right.$)

    $S_{C}(t,q)=1$ (using $ P(q,d)=$$\left\{\begin{array}{@{}ll@{}} 1, & \text{if } W_Q(q) < 0 \text{ because neg. scores are important,} \\ & \text{independent from the detection capabilities} \\ 1, & \text{if the answer on question } q \text{ is not specific enough} \\ & \text{to do a further scoring on any } c \in C \\ \max_{i=1}^N (S_{AV}(c_i,q) * S_{CC}(c_i,q)), & \text{else} \end{array}\right.$)

    $t=$ 2.5.72.15
    $q=$ 'is_well_known_ip'

    $S_Q(q,t) = -50 * \frac{1}{\sqrt{(1 * 3)}} * 1) \approx -28.87$

    No other question has been answered.

    --> FP can be prevented not using negative scored traces.
    Input Petya (IPs only)
    Date April/May 2017
    Use-Case Rate potential IoC list
    Detection capabilities FW-Logs
    Required confidence level High (Goal: Block entire traffic on FW)

    Is 80.150.6.143 a good IoC?

    (http://bluemount.de/Rechnung-TCJ-915-M6014/)

    Block 80.150.6.143?

    (http://bluemount.de/Rechnung-TCJ-915-M6014/)

    How to resolve it in the scoring system?

    $S_{Total}(t) = \sum\nolimits_{i=1}^{N} S_Q(q_i,t)$

    $t=$ 80.150.6.143
    $q=$ 'is_multihoster'

    $S_Q(q,t) = W_Q(q) * S_{D}(t,q) * S_{C}(t,q)$

    $W_Q(q)= -40$

    $S_{D}(t,q)=\frac{0.9}{\sqrt{(1 * 2)}}$ (using $ P(q,d)=$$\left\{\begin{array}{@{}ll@{}} \frac{S_{Time}}{\sqrt{(S_{Rel} * S_{Cre})}}, & \text{if the data source }d \text{ can answer question }q \\ 0, & \text{else} \\ \end{array}\right.$)

    $S_{C}(t,q)=1$ (using $ P(q,d)=$$\left\{\begin{array}{@{}ll@{}} 1, & \text{if } W_Q(q) < 0 \text{ because neg. scores are important,} \\ & \text{independent from the detection capabilities} \\ 1, & \text{if the answer on question } q \text{ is not specific enough} \\ & \text{to do a further scoring on any } c \in C \\ \max_{i=1}^N (S_{AV}(c_i,q) * S_{CC}(c_i,q)), & \text{else} \end{array}\right.$)

    $t=$ 80.150.6.143
    $q=$ 'is_multihoster'

    $S_Q(q,t) = -40 * \frac{0.9}{\sqrt{(1 * 2)}} * 1) \approx -25.46$

    And the other questions?

    Input PWC/BAE APT-Report "Operation Cloud Hopper" (IPs only)
    Date April/May 2017
    Use-Case Rate potential IoC list
    Detection capabilities FW-Logs
    Required confidence level High (Goal: Block entire traffic)

    Remember the TOR exit node 185.29.8.211?

    There are two more of it inside.


    - WikiLeaks published 60.000 mails of HBGary in 11/2016

    - HBGary did a security audit at QinetQ North America

    - HBGary found several infected systems and lists
    119.167.225.38
    119.167.225.44
    123.183.210.26
     as C2 infrastructure of targeted malware called msomsysdm

    - PWC/BAE now attributed them to chinese APT-10

    BACKUP:
    Taxonomies, Threat-Levels...

    Taxonomies

    - Reliability
    - Credibility
    - Threat levels

    How to define Reliability, Credibility and Threat Levels
    of an information about a digital trace?

    Interpreted feeds

    Most of them providing those information


    Un-Interpreted feeds

    Initial definition by analyst experience
    Improvements are done via SoC feedback channel

    BACKUP:
    Current status

    Current status / Further work

    | From | To | Goals | |------------ |------------ |------------------------------------------------------- | | 2017-07-20 | 2017-08-25 | SANS GNFA Certification / Holiday/ Stabilize system | | 2017-08-21 | 2017-08-24 | Presentation @ Uni | | 2017-12-xx | 2017-12-yy | Guest lecture @ Uni2 | | 2017-08-28 | 2017-11-31 | IP scoring finished by 90% | | 2017-09-18 | 2017-11-31 | Felix implements web frontend for TI/CDC/CERT (alpha) | | 2017-12-01 | 2017-12-31 | Integrate scoring for customer delivery | | 2017-12-01 | 2018-02-28 | Finish thesis |

    BACKUP: URLs

    (Domains/IPs inside)

    URLS

    vs

    DOMAINS

    - twitter.com is included in Alexa Top1000 list of well-known and widely used domain names

    - Using twitter.com as IOC is absolutely useless

    - Complete URLs are more specific and can be used as IOC

    User-driven content

    vs

    Professionally-generated content

    (More context means more exceptions)
    Taking into account that some attackers are using social networks for C&C purpose,

    user-driven content has the ability to be a great IOC source.

    So how to filter them out?

    Filter user-driven content

    1. Gather information about (http/s) request structure of website
    2. Black- / Whitelist approach

    (As annoying as it sounds... but let´s try)

    Blogs

    • {blog_name}.blogspot.com
    • {blog_name}.wordpress.com
    • {blog_name}.typepad.com

    Free blog services are well-known for malware distribution

    Torrents

    Torrents

    Picture sharing

    • boards.4chan.org/g/thread/{8_numeric_digits}
    • i.4cdn.org/g/{numeric_digits}.jpg
    • 9gag.com/gag/{6_chars}

    File-Uploads

    • zippysharecom/v/{8_chars}
    • uploaded.net/{chars}

    Code Sharing

    • stackoverflow.com/questions/{numeric_id}/
    • {subsite}.stackexchange.com/questions/{numeric_id}/
    • github.com/{user_name}/{project}*
    • {user_name}.github.io

    * with some exceptions (e.g. github.com/security/)

    Social Networks

    User-Driven

    (Mainly) Professionally-generated

    Professionally-generated

    User-Driven

    User-Driven

    (Multiple) User-Driven

    User-driven vs Professionally-generated
    [Summary]

    • Providing user-driven content is the main objective of all those providers*
    • Separation of user content leads to simplicity
    • (High separation between objectives leads to better detection)
    • Provider fluctuation = maintenance effort
    • Once collected, these information are mostly durable!

    Use Cases

    Scoring formula details

    $S_Q(q,t) = W_Q(q) * S_{D}(t,q) * S_{C}(t,q)$


    1. Question weight $W_Q(q)$


    2. The data sources evaluation score $S_{D}(t,q)$

    $S_{D}(t,q)=$$\left\{\begin{array}{@{}ll@{}} P(q,D), & \text{if an answer exists using data sources } D \\ 0, & \text{else}\\ \end{array}\right.$

    $P(q,D)$ expresses the probability for a correct answer.
    Due to the fact that $D$ is a set of data sources ( $D=$$\{d_1,d_2...\}$),
    the probability $P(q,D)$ is defined* as $\max_{i=1}^N P(q,d_i)$ for each $d \in D$
    *There will be a bonus in case multiple sources match.

    Each data source $d$ answers with probability $P(q,d)$ question $q$:
    $ P(q,d)=$$\left\{\begin{array}{@{}ll@{}} \frac{S_{Time}}{\sqrt{(S_{Rel} * S_{Cre})}}, & \text{if the data source }d \text{ can answer question }q \\ 0, & \text{else} \\ \end{array}\right.$

    where $\frac{S_{Time}}{\sqrt{(S_{Rel} * f(S_{Cre}))}}$ is defined as

    • Time-Score $S_{Time}$ $[0.0,0.1,...,1.0]$
      • In time-critical cases: "The older the answer, the lower the score"
      • Otherwise: $1$
    • Reliability-Score $S_{Rel}$ $\in \{1,2,...,6\}$
    • Credibility-Score* $S_{Cre}$ $\in \{A,B,...,F\}$ with $f:i \mapsto{y} \text{ where } (i,y) \in \{(A,6),(B,5)...(F,1)\}$
        • * = NATO-Score
    Morty Proxy This is a proxified and sanitized view of the page, visit original site.