Skipfish will not log in to crawl destination but fails without good error

[GoogleCodeExporter]

Using V 2.10b on Ubuntu 12.04 64 bit

the following command fails

cd /home/stew/skipfish-2.10b
./skipfish   -u -v   --config ../skipfish.conf   -o 
/var/www/scan/clients/peer1/loggedin    -S 
/home/stew/skipfish-2.10b/dictionaries/minimal-peer1.wl   
https://us.peer1.fullfatthings.com/portal


 cat ../skipfish.conf
auth-form=https://us.peer1.fullfatthings.com/user
auth-user=fftlivedemo
auth-pass=xxxxxx
auth-verify-url=https://us.peer1.fullfatthings.com/portal/account/users
auth-user-field=name
auth-pass-field=pass
#auth-form-target=https://us.peer1.fullfatthings.com/user

The https://us.peer1.fullfatthings.com/ URL is protected via Basic Auth 
externally but to the server that skipfish is running on a basic Curl returns 
200 and OK

curl -Ik https://us.peer1.fullfatthings.com/user
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 07 May 2013 09:19:11 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Keep-Alive: timeout=10
Vary: Accept-Encoding
X-Powered-By: PHP/5.3.24-1~dotdeb.0
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Tue, 07 May 2013 09:19:11 +0000
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
ETag: "1367918351"
Content-Location: https://us.peer1.fullfatthings.com/user
Content-Language: en
X-UA-Compatible: IE=edge,chrome=1
Link: <https://us.peer.fullfatthings.com/user>; 
rel="shortlink",<https://us.peer.fullfatthings.com/user>; rel="canonical"

The actual error message that we get back is:

./skipfish   -u -v   --config ../skipfish.conf   -o 
/var/www/scan/clients/peer1/loggedin    -S 
/home/stew/skipfish-2.10b/dictionaries/minimal-peer1.wl   
https://us.peer1.fullfatthings.com/portal
skipfish web application scanner - version 2.10b
*- Authentication starts
*-- Could not login. Please check the URL and form fields
[-] PROGRAM ABORT : Authentication failed (use -uv for more info)

    Stop location : main(), src/skipfish.c:714

Original issue reported on code.google.com by [email protected] on 7 May 2013 at 9:21

[GoogleCodeExporter]

Normally skipfish will also tell you the form fields it found. It didn't find 
any. Are you sure this has the login form ? (e.g. If the page dynamically build 
the login form with Javascript then skipfish will not detect it meaning you'd 
have to use cookie auth)

Original comment by [email protected] on 2 Jul 2013 at 8:34

[GoogleCodeExporter]

This is a Drupal site. The login form is in the delivered source and works 
without any JS.

Original comment by [email protected] on 2 Jul 2013 at 8:35

[GoogleCodeExporter]

I'm having the exact same issue. It's not a Drupal site but there is no 
javascript and the login form is also in the delivered source. Exact same 
symptoms and exact same error. (like 714).

Original comment by [email protected] on 17 Nov 2013 at 7:17

[GoogleCodeExporter]

You could try with an additional -v to see more output. Additionally, you can 
try with 'make debug' to get a super verbose report (via stderr). The last 
should give you good insight in what happens under the hood. Maybe there is a 
problem with the form parsing - especially since you both report this same 
problem.

Cheers!
Niels

Original comment by [email protected] on 17 Nov 2013 at 8:08

[GoogleCodeExporter]

Thanks, Niels - I didn't know you could use an additional -v I will try that, 
but I think it will be more helpful if I also recompile with the debug option 
and then post some useful information that might help in diagnosing the issue. 
Really appreciate the quick response. 

Also, should I be using the SVN version - currently I'm using the latest 
release.

Original comment by [email protected] on 18 Nov 2013 at 8:58

[GoogleCodeExporter]

When I try with the following config...

auth-form = https://10.30.70.10/Account/LogOn
auth-user = Security
auth-pass = xxxxxx
auth-user-field = UserName
auth-pass-field = Password
auth-verify-url = https://10.30.70.10/Transfers
auth-form-target https://10.30.70.10/Account/LogOn
form-value = CorrespondentNo=51098
form-value = RememberMe=false

... the password is set to "skipfish" rather than the one I've specified and no 
username is sent as seen the below:

--- cut here ---

NEW PROBLEM
- type: 10505, Unknown form field (can't autocomplete)
- url:  https://10.30.70.10/Account/LogOn

NEW PROBLEM
- type: 10602, Password entry form - consider brute-force
- url:  https://10.30.70.10/Account/LogOn 
DATA:Password=skipfish&CorrespondentNo=1&RememberMe=false

Could not login. Please check the URL and form fields

--- cut here ---

So I thought maybe I was not using the "form-value" option correctly and tried 
something like this...

form-value = CorrespondentNo=51098&RememberMe=false

... but same problem.

Then I tried the following new config...

auth-form = https://10.30.70.10/Account/LogOn
auth-user = Security
auth-pass = xxxxxx
#auth-user-field = UserName
#auth-pass-field = Password
auth-verify-url = https://10.30.70.10/Transfers
#auth-form-target https://10.30.70.10/Account/LogOn
form-value = UserName=Security
form-value = Password=xxxxxx
form-value = CorrespondentNo=51098
form-value = RememberMe=false

... which got me a little further but still no cigar as seen below (still no 
username passed)...

--- cut here ---

NEW PROBLEM
- type: 10602, Password entry form - consider brute-force
- url:  https://10.30.70.10/Account/LogOn 
DATA:Password=xxxxxx&CorrespondentNo=51098&RememberMe=false

Could not login. Please check the URL and form fields

--- cut here ---

Does this help at all? Let me know if there is anything else I can provide.

Thanks :-)

- Marco

Original comment by [email protected] on 19 Nov 2013 at 12:56

[GoogleCodeExporter]

I've tried to post this yesterday and it shows up as a deleted comment. I'm 
trying again and this time I will also add the comment as an attachment. Really 
weird...

***

When I try with the following config...

auth-form = https://10.30.70.10/Account/LogOn
auth-user = Security
auth-pass = xxxxxx
auth-user-field = UserName
auth-pass-field = Password
auth-verify-url = https://10.30.70.10/Transfers
auth-form-target https://10.30.70.10/Account/LogOn
form-value = CorrespondentNo=51098
form-value = RememberMe=false

... the password is set to "skipfish" rather than the one I've specified and no 
username is sent as seen the below:

*** cut here ***

NEW PROBLEM
- type: 10505, Unknown form field (can't autocomplete)
- url:  https://10.30.70.10/Account/LogOn

NEW PROBLEM
- type: 10602, Password entry form - consider brute-force
- url:  https://10.30.70.10/Account/LogOn 
DATA:Password=skipfish&CorrespondentNo=1&RememberMe=false

Could not login. Please check the URL and form fields

*** cut here ***

So I thought maybe I was not using the "form-value" option correctly and tried 
something like this...

form-value = CorrespondentNo=51098&RememberMe=false

... but same problem.

Then I tried the following new config...

auth-form = https://10.30.70.10/Account/LogOn
auth-user = Security
auth-pass = xxxxxx
#auth-user-field = UserName
#auth-pass-field = Password
auth-verify-url = https://10.30.70.10/Transfers
#auth-form-target https://10.30.70.10/Account/LogOn
form-value = UserName=Security
form-value = Password=xxxxxx
form-value = CorrespondentNo=51098
form-value = RememberMe=false

... which got me a little further but still no cigar as seen below (still no 
username passed)...

*** cut here ***

NEW PROBLEM
- type: 10602, Password entry form - consider brute-force
- url:  https://10.30.70.10/Account/LogOn 
DATA:Password=xxxxxx&CorrespondentNo=51098&RememberMe=false

Could not login. Please check the URL and form fields

*** cut here ***

Does this help at all? Let me know if there is anything else I can provide.

Thanks :-)

- Marco

Original comment by [email protected] on 20 Nov 2013 at 10:32

Attachments:

• skipfish_error.txt

[GoogleCodeExporter]

Original comment by [email protected] on 22 Nov 2013 at 4:54

Attachments:

• skipfish_error.txt

[GoogleCodeExporter]

I get the same error message.  Using "make clean debug" gives me more 
information, but makes it look like it's stuck in a loop.

I ran a "make clean debug".
I ran a "curl -c app.cookie http://192.168.0.242/app/Authentication/Logon" to 
capture a session id.
I run skipfish with a command like:
touch app_dict.wl
./skipfish -uv -S dictionaries/complete.wl -S dictionaries/medium.wl -W 
app_dict.wl -Y \
    --auth-form http://192.168.0.242/app/Authentication/Logon \
     --auth-user USERNAME \
     --auth-pass PASSWORD \
     --auth-verify-url http://192.168.0.242/app/RequestReport/Index \
     -X /logout \
     -d 4 \
     -o ~/Downloads/skipfish-2.10b/output \
     -C ASP.NET_SessionId=xyzabc123blabla \
     http://192.168.0.242/app/Authentication/Logon 2> debug.log


I changed the IP and application name, but here's the jist of the debug.log 
file:

### dictionaries and signatures load (lots of them) ###
*- Signatures processed: signatures/context.sigs (total sigs 77)
*- Signatures processed: signatures/signatures.conf (total sigs 77)
* Read 0 lines from dictionary 'app_dict.wl' (read-only = 0).
*- Authentication starts
* submit_auth_form: URL http://192.168.0.242/app/Authentication/Logon (200, len 
16088)
* test_add_link: URL http://192.168.0.242/app/Authentication/Logon (200, len 
16088)
* Alleged URL = '#' [4]
--- New pivot requested: http://192.168.0.242/app/Authentication/Logon (2,0)
--- NEW PROBLEM - type: 40201, extra: '#' ---
* collect_form_data() entered
--- NEW PROBLEM - type: 10602, extra: '(null)' ---
* test_add_link: URL http://192.168.0.242/app/Authentication/Logon (200, len 
16088)
* Alleged URL = '#' [4]
--- New pivot requested: http://192.168.0.242/app/Authentication/Logon (2,0)
--- NEW PROBLEM - type: 40201, extra: '#' ---
* collect_form_data() entered
--- NEW PROBLEM - type: 10602, extra: '(null)' ---
* test_add_link: URL http://192.168.0.242/app/Authentication/Logon (200, len 
16088)
* Alleged URL = '#' [4]
--- New pivot requested: http://192.168.0.242/app/Authentication/Logon (2,0)
--- NEW PROBLEM - type: 40201, extra: '#' ---
* collect_form_data() entered
--- NEW PROBLEM - type: 10602, extra: '(null)' ---

### it repeats this about 100 or so times, then aborts with this ###

�[1;31m[-] PROGRAM ABORT : �[1;37mAuthentication failed (use -uv for more info)
�[1;31m
    Stop location : �[0;37mmain(), src/skipfish.c:714

Original comment by [email protected] on 15 Jan 2014 at 9:01

[GoogleCodeExporter]

I've made a few changes.  I made a config file.  I changed the start page to 
http://192.168.0.242/app/Authentication/Index just to avoid confusion.  I set 
the auth-user-field and auth-pass-field.  I also set the other logon form 
fields that I couldn't figure out how to set via command line.

I now run these commands to launch:
curl -c app.cookie http://192.168.0.242/app/Authentication/Logon > nul
awk '/FALSE/ { print $7 }' app.cookie
COOKIE=`awk '/FALSE/ { print $7 }' app.cookie`
touch  my-wordlist.wl
./skipfish --config ./config/app.conf -C ASP.NET_SessionId=$COOKIE \
     http://192.168.0.242/app/Authentication/Index 2> debug.log
tail -n20 debug.log

I added some DEBUG calls in skipfish.c (with the line before and after):
---- start snipet
    authenticate();

// dk mod
char str_state[30];
sprintf(str_state, "auth_state =  %d\n", auth_state);
DEBUG("auth states\n");
DEBUG("ASTATE_NONE   0, ASTATE_START  1, ASTATE_SEND   2, ASTATE_VERIFY 3, 
ASTATE_DONE   4, ASTATE_FAIL   5\n");
DEBUG(str_state, "%s");

    while (next_from_queue()) {
---- end snipet
My auth_state is at 1 (ASTATE_START) when it fails.  The debug log is pretty 
much the same:

*- Signatures processed: signatures/context.sigs (total sigs 77)
*- Signatures processed: signatures/signatures.conf (total sigs 77)
* Read 0 lines from dictionary 'my-wordlist.wl' (read-only = 0).
*- Authentication starts
auth states
ASTATE_NONE   0, ASTATE_START  1, ASTATE_SEND   2, ASTATE_VERIFY 3, ASTATE_DONE 
  4, ASTATE_FAIL   5
auth_state =  1
* submit_auth_form: URL http://192.168.0.242/app/Authentication/Logon (200, len 
15300)
* test_add_link: URL http://192.168.0.242/app/Authentication/Logon (200, len 
15300)
* Alleged URL = '#' [4]
--- New pivot requested: http://192.168.0.242/app/Authentication/Logon (2,0)
--- NEW PROBLEM - type: 40201, extra: '#' ---
* collect_form_data() entered
--- NEW PROBLEM - type: 10602, extra: '(null)' ---
* test_add_link: URL http://192.168.0.242/app/Authentication/Logon (200, len 
15300)
* Alleged URL = '#' [4]
...

Original comment by [email protected] on 15 Jan 2014 at 10:24

[GoogleCodeExporter]

I apologize for the spam, but I haven't stopped trying to figure this out.

I added debug output to other functions trying to find the real cause of the 
authentication failure.

The config file has (compressed here):
auth-user = USERNAME 
auth-pass = PASSWORD
auth-user-field = UserName
auth-pass-field = Password
form-value resolution=1280\|\|768
form-value maintenance=false

The debug.log has:
set_value() entered name, val =  resolution, 
set_value() entered name, val =  maintenance, false
set_value() entered name, val =  Password, skipfish 

As a note, the pipes did cause a bit of confusion on the command line because 
the error would say I didn't specify the site to test (paraphrased).  I've 
tried the config escaped and not with no difference.

One thing I don't understand from the documentation is how to separate multiple 
form fields or cookie fields on the command line.  Would I use multiple -T's or 
commas or what?
In the config I've tried a few things and it makes no difference.  I even 
commented the form fields with the same result.

Because I hadn't seen what it's supposed to look like when it works I ran this 
command:
./skipfish -S dictionaries/medium.wl -W my-wordlist.wl -Y \
     -X /logout/,/css/,/img/,/images/,/js/,/doc/ \
     -d 4 \
     -o /tmp/skipfish-report \
     http://zero.appsecurity.com/rootlogin.asp.bak 2> debug.log

No surprise that it worked beautifully.  The link was from documentation, and 
it's a 404 now, but skipfish ran perfectly.  So my problem is definitely 
getting authentication with extra form fields to work.

Original comment by [email protected] on 16 Jan 2014 at 8:10

[GoogleCodeExporter]

OK, I think that I worked around the issue.  I gave up on the form 
authentication and tried to figure out the cookie auth.  Instead of using curl, 
which was giving me an unauthenticated sessionID (at least the way I was doing 
it) I logged into the site in FireFox then looked at my cookie (Edit - 
Preferences - ...) to copy the sessionID.

Command line now looks like this (form auth commented in the config):
touch  my-wordlist.wl
COOKIE=xxxyyyzzzaaabbb
./skipfish --config ./config/mlf.conf -C ASP.NET_SessionId=$COOKIE \
     http://192.168.0.242/app/Authentication/Index 2> debug.log
tail -n20 debug.log

It appears to be working.  I'm not sure if I should stop it to tweak any 
settings, so I'll just let it run and see what comes out the other side.

BTW, when I tried to add all of the cookie values I'd either get "stack 
smashing" or "Bus error (core dumped)".
From looking around stack smashing is a GCC protection stopping you from buffer 
overflows.  The variable name and the value were both long, so I assume that 
was it.
The bus error occurred when I have a short variable name (starting with period) 
and a very long value.

Original comment by [email protected] on 16 Jan 2014 at 10:19