{"id":14379,"date":"2019-01-20T14:20:24","date_gmt":"2019-01-20T22:20:24","guid":{"rendered":"https:\/\/www.palada.net\/index.php\/2019\/01\/20\/news-8131\/"},"modified":"2019-01-20T14:20:24","modified_gmt":"2019-01-20T22:20:24","slug":"news-8131","status":"publish","type":"post","link":"https:\/\/www.palada.net\/index.php\/2019\/01\/20\/news-8131\/","title":{"rendered":"Using Machine Learning To Detect Anomalies"},"content":{"rendered":"

Credit to Author: dmitryc| Date: Mon, 21 Dec 2015 22:07:07 +0000<\/strong><\/p>\n

\n
\n

I’m going to start blogging more about detection of protocol\/app anomalies, detection of lateral movement and\/or data exfiltration, and more. For many years I have been watching users and applications furrow their way across networks and I’m gonna start data-dumping that info here \ud83d\ude42 <\/p>\n

But…first…I manage a web server for a friend. It occurred to me that machine-learning could be useful in alerting when an attack is under way. I took the following steps<\/p>\n

1) Get as much data as possible for this device. For Apache, this just meant gathering all the log files.<\/p>\n

2) Parse the data and, for each session, look at the path taken as the user or bot perused the server (Note: outside of my initial scope, but timestamps are useful here to weed out a user versus a machine).<\/p>\n

3) So, an average session will look like R1->R2->R3->RX where each “R” is a request. So R1 could be index.html, R2 could be “Contact Us”, R3 could be “contact_form.php”, etc. I started using Markov to build a model; however, instead, I took each set of 2 and initialized those values…e.g. S={R1->R2,R2->R3,R3->RX}. For the next session I might have S={R1->R5,R5->R3,etc.}. At the end of all the parsing, I have a big set of all state transitions possible for each R. So, given RX, there are a finite number of R states that RX can transition to.<\/p>\n

4) For each of the R states, I now re-parse the log file and find the number of transitions. This is a matrix that shows the number of observed transitions from RN to every other R state. So, for instance, let’s say that R1 goes to 3 possible states : R4 (27% of time), R11 (3% of time) , and R12 (70% of time). Then the R1 row of our matrix looks like [0, 0, 0, .27, 0, 0, 0, 0, 0, 0, .03, .7] <\/p>\n

5) There were some special cases that I had to account for (any page transitioning to the main page, any page transitioning to itself, etc.). Once I accounted for these, I ran my program against the log files and created LOW, MEDIUM, and HIGH alerts. I didn’t use a true standard deviation and I ignored the LOW and MEDIUM stuff…I just wanted the hits where the number for that transition was extremely low or 0. From our example above, this would be a transition like R1->R2=0. I didn’t really expect great results and figured that I would have to do a lot more tweaking…well, this wasn’t the case. I actually got really, really good data on my first run. Example:<\/p>\n

732 total state transitions tracked
HIGH RISK GET \/componentes3.7\/fckeditor\/editor\/fckeditor.html->GET \/affiliate\/affiliate53\/fckeditor\/editor\/fckeditor.html<\/p>\n

HIGH RISK GET \/portfolio\/aui\/FCKeditor\/editor\/fckeditor.html->GET \/componentes3.7\/fckeditor\/editor\/fckeditor.html<\/p>\n

HIGH RISK GET \/wp-content\/uploads\/wpfouot.php->POST \/wp-content\/plugins\/Login-wall-etgFB\/login_wall.php<\/p>\n

etc.<\/em><\/p>\n

So, I can use really basic machine learning to find my attackers in my web logs. I then parse out the attackers’ IP addresses and can throw them into a firewall ruleset. In the future, I would like to automate this and find when my server is under attack, send a message to my firewall which drops in a route rule which spins all of the attackers traffic to my honey net \ud83d\ude42 <\/p>\n

Speaking of honeypots, You can also honeypot certain pages. For instance, I could create bogus files or directories based on what I see attackers going after (like the report from above) and drop canary tokens in there to (see Canary Tools<\/a>). I can embed honeypot links within HTML comments and see where bots (or humans) are taking links from commented code and trying them out. I can put links in my robots.txt file and see who goes after them…there are so many ways to do this…and, at the end of the day, I can either run these attackers off my network or into a fake network…it’s just TONS and TONS of fun \ud83d\ude42<\/p>\n

!Dmitry
dmitry.chan@gmail.com<\/p>\n

\"Print<\/a><\/div>\n<\/div><\/div>\n

https:\/\/blogs.securiteam.com\/index.php\/feed<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

<\/p>\n

Credit to Author: dmitryc| Date: Mon, 21 Dec 2015 22:07:07 +0000<\/strong><\/p>\n

I’m going to start blogging more about detection of protocol\/app anomalies, detection of lateral movement and\/or data exfiltration, and more. For many years I have been watching users and applications furrow their way across networks and I’m gonna start data-dumping that info here \ud83d\ude42 But…first…I manage a web server for a friend. It occurred to … Continue reading Using Machine Learning To Detect Anomalies<\/span><\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"colormag_page_container_layout":"default_layout","colormag_page_sidebar_layout":"default_layout","footnotes":""},"categories":[10643,10754],"tags":[10755],"class_list":["post-14379","post","type-post","status-publish","format-standard","hentry","category-independent","category-securiteam","tag-commentary"],"_links":{"self":[{"href":"https:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/posts\/14379","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/comments?post=14379"}],"version-history":[{"count":0,"href":"https:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/posts\/14379\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/media?parent=14379"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/categories?post=14379"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.palada.net\/index.php\/wp-json\/wp\/v2\/tags?post=14379"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}