John The Ripper Mac Installation, Tutorial and Optimization
A while ago, I wrote this tutorial on how to install the simple version (pre-compiled) of John The Ripper on a Mac
It's now time to optimize John. Why? it's working!
Yes, it's working, but can you make it work much better?
John The Ripper, AKA John/JTR is the extreme opposite of intuitive, and unless you are an UberGeek, you've probably missed out few subtleties.
Secondly, John The Ripper is a bit like a Muscle Car delivered from the factory with the "Eco" settings enabled by default. If you want the "Muscle", you'll have to open the hood. Once done, you'll get more muscle than you can probably chew on.
Secondly, John The Ripper is a bit like a Muscle Car delivered from the factory with the "Eco" settings enabled by default. If you want the "Muscle", you'll have to open the hood. Once done, you'll get more muscle than you can probably chew on.
Subtleties?
Well, for a start, the precompiled version on OSX does not use all cores, the "compiled on site" can ...
So if you have a quad-core, you'll make a 196% speed improvement, and that's only on raw benchmark.
The default password list is absolute crap, and the configuration could use a few improvements.
The default password list is absolute crap, and the configuration could use a few improvements.
When you'll start playing just with those 2, the result(s) could be as shown below.
Deemed worth reading?
Ok?
Ok?
Let's install it ...
1- You'll need Xcode, if not already installed, download it , it's free.
Open App Store, search XCode, download and install.
The install is going to take some time....
☞ If you wonder if you have Xcode installed, just open terminal and type gcc
The "no input file" is a good sign. Otherwise, install it.
1a- If you do not wish to install Xcode, you can always refer to here, but be kindly warned that you'll have a "less effective" version of JTR.
2- Grab a copy of the latest John The Ripper @ http://www.openwall.com/john/
look for the community enhanced version (more patches), as of today, it's the john 1.7.9 jumbo 6 2.
2a - While patiently looking at your download screen, grab a free copy of TextWrangler, not the truncated App Store Version, The full one from BareBones and install, you'll thank me later.
3- Xcode and TextWrangler have been installed, John has been downloaded, refreshments have been served.
4- Go to your "Download" directory and look for John 1.7.9.jumbo 6 tar.gz, or whatever you have downloaded, and double click on it, it will decompress it.
5- You have now a directory called John "something"
Please replace "something" in your head by the appropriate designation, i.e "John-1.7.9-jumbo 6 2"
6-Create a folder,and name it "John" anywhere on you Mac, but, recommended, that you either create it in "Macintosh HD" or your home folder, i.e ~/"myname".
Please, don't use the quotes for "John". Just John. ( Somebody is going to create a "Just john", that's guaranteed, and if you do, you will be flamed. Guaranteed too)
7- Copy all folders & files contained in "John-1.7.9-jumbo 6 2" into the newly created John folder. Only the contents, no need to copy the John-1.7.9-jumbo 6 2 folder too. If you do, you'll have to type a very long command line.
8- You have now a folder named John, with in it the 5 items as shown above.
Unfortunately, now is the time to make a choice...
Das Kompilation!
You'll have to make a choice, roughly between 32 bit, or 64 bit
64 bit seems the choice, no? Hold your horses!
One little thing: as of now, Aug 2012, when cracking pure raw-sha1(i.e Linkedin) the 32-bit is a bit faster. (no pun intended)
For generic use, select "64 bit", for Raw-Sha1 only, go for the 32 bit sse2.
The difference is not that extreme, and lastly, the 64bit version accepts Wordlists larger than 2GB.
For UberGeeks, well... install both. Your humble servant has 3 versions (for research purpose only, I am not geeking, 2 of them will be deleted soon)
- JTR is the Mac OSX pre-built
- JTR32 is the 32bit
- JTR64 is ... well, you've guessed by now.
So, make a choice (hint: 64 bit) and we'll start...
1) Open Terminal and go to the "src" folder of John, by typing
cd /john/src If you have placed john somewhere else, then go there...
1a) Second choice to make:
Please read carefully as it trapped me the first time:
JtR runs natively on one core.
You can safely run 2, or more instances on JtR at the same time, for instance one instance running a set of rules and the other instance running something else.
But you can also ask John to run on all cores. On Mac it's called "mpirun"
mpirun has to be enabled in the Makefile before compiling.
I highly suggest to do so, as it enable both parallel and multicore.
In order to do so:
1a) Open TextWrangler
Look for the file /John/src/makefile and open it
1b) Search for the following paragraph:
## Uncomment the TWO lines below for MPI (can be used together with OMP as well) ## If you experience problems with MPI_Barrier, remove -DJOHN_MPI_BARRIER ## If you experience problems with MPI_Abort, remove -DJOHN_MPI_ABORT ##CC = mpicc -DHAVE_MPI -DJOHN_MPI_BARRIER -DJOHN_MPI_ABORT ##MPIOBJ = john-mpi.o1c) Un-comment the 2 last line, (remove the ##)
1d) Save
2) Decide if you wish or will attempt to Bruteforce longer than 8ch long passwords. If you don't know, don't touch it now and come back later.
If you do not want to change the max bruteforce length, Skip to directly to part 3)
Before going bonkers, read the "Bruteforcing Speed" part below, take a look at the ETA, and you'll realize that attempting a 20ch Keyspace is not really feasible.
Done reading?
Ok, Here is how you can force JtR to Bruteforce longer than 8ch long.
2a) Open TextWrangler
Look for the file params.h in /John/src and open it
In that file, look for the section:
#define CHARSET_MIN ' '
#define CHARSET_MAX 0x7E
#define CHARSET_SIZE (CHARSET_MAX - CHARSET_MIN + 1)
#define CHARSET_LENGTH 8
2b) Change the 8 into what you want, i.e 15 or 30.
#define CHARSET_MIN ' '
#define CHARSET_MAX 0x7E
#define CHARSET_SIZE (CHARSET_MAX - CHARSET_MIN + 1)
#define CHARSET_LENGTH 15
☞ Warning: There are spaces and tabs, don't replace them by "spaces"
Use the Show Invisible in Text Wrangler.
2c) Save the file params.h and go to part 3) compilation.
Once the compilation is done, Go to "Charset making "for the final part. (did I mention that you'll have to go under the hood?)
We can now start the compilation. In /John/src
3) In /John/src, type exactly, or copy/paste
- for the 32 bit version: make clean macosx-x86-sse2
- for the 64 bit version: make clean macosx-x86-64
Hit "Return" and plenty of things will appear on terminal screen, don't panic.
While waiting, and for the curious, you can open the file "makefile" with textwrangler and look at all the options, just in case you want to install JTR on a PowerPC Mac
The compile options look like:
@echo "macosx-x86-64 Mac OS X 10.5+, Xcode 3.0+, x86-64 with SSE2 (best)"
@echo "macosx-x86-64-opencl Mac OS X 10.5+, Xcode 3.0+, x86-64 with SSE2, OpenCL support"
@echo "macosx-x86-sse2 Mac OS X, x86 with SSE2"
@echo "macosx-ppc64 Mac OS X 10.4+, PowerPC 64-bit"
@echo "macosx-universal Mac OS X, Universal Binary (x86 + x86-64 + PPC)"
You'll also find all Linux options, just in case ....
4) Now that the install is done, test it by typing
cd /john/run
./john
☞ Warning: the ./ are mandatory and you must be in the proper /run directory If you try "john" as is, you'll receive an error.
You should see the short help for John The Ripper
If you see something like that:
bash: ./john: No such file or directory
it means that, either: The compilation has failed, or, that you have not typed the proper path/command. In both case, retry or re-read the above. All of it: You've failed once and that's probably for a good reason: rtfm!
Das Optimization!
We're going to glide over few things rapidly, for a deeper look at what does what, there is a folder named "doc" , inside they have docs!. Yay!
1) John, by default, runs on one core only. If you have a single core, it's good news, otherwise not so much. So let's test it , and type
./john --test=10 --format=raw-sha1-linkedin
after 10 sec, you see something like:Benchmarking: Raw SHA-1 LinkedIn [128/128 SSE2 4x]... DONE
Raw: 9540K c/s real, 9559K c/s virtual
Now, let's try something else, type
mpirun -np 2 ./john --test=10 --format=raw-sha1-linkedin
And the results are:
Benchmarking: Raw SHA-1 LinkedIn [128/128 SSE2 4x]... (2xMPI) DONE
Raw: 17655K c/s real, 18923K c/s virtual
This time, we were twice as fast. Yup! 2 cores. So if you have a quad core, try the "mpirun -np 4". (yes, Mr Trae, that's why your quadcore is apparently slower than my dual)
john.conf
2) Open Textwrangler app, and look for a file named john.conf in /john/run
open that file and make right away a backup, that you will intelligently call "backup_john.conf".
In this john.conf, you'll find roughly 2 types of configuration: General config and Rules config.
For instance look at the line:
# Use idle cycles only
Idle = Y
it means that John will only use the CPU power left Vs. requesting full power, hence you could (should) switch it to "No"Idle = Y
# Use idle cycles only
Idle = N
Idle = N
Look at the other options and choose what you want on/off. As a rule of thumb, don't finagle too much if you don't understand it, i.e trying funky settings such as [Incremental:All] MaxLen = 99
Still, you have a backup now, don't you? :) , so you can play a little...
2a) As we are going to talk about Wordlist, think about that line:
# Wordlist file name, to be used in batch mode
Wordlist = $JOHN/password.lst
the default wordlist in John is password.lst , and is crap. Using this wordlist will not take you far at all.
2b) if you have changed the name of the main directory for John, i.e JTR, you should change the parameter to $JTR/password.lst
2c) ditto with the password.lst, if you use mydic.txt , then change it too.
the easiest way is simply to keep the name password.lst, or to use the --wordlist option
2d) Wordlist itself
As mentioned above, the default password.lst is crap. if your wish to run john for weeks on end, keep it. Otherwise, get this Wordlist
Here is why: on default mode , John will first try the wordlist "as is" , then mingle it with rules contained in John.conf under [List.Rules:SIngle] for "Single crack Mode"
Once done, it will switch to incremental mode, All alphanumerical characters. one by one.
Problem: You are now at day 3, and you are seeing 0.00% progress. Why?
Because you are trying all possible combinations of 1 to 8 ch long, over 95 possible characters. it's called the Keyspace and that number is 6,704,780,954,517,120
it simply means that you'll see progress at 0.00% for few weeks....See you in 3 months...
In John the default length is 8 max and that also mean that a 10 ch long password will not be (cannot be cracked for sure) by the "All" method alone unless you go under the hood as explained below.
Bruteforce is, generally, and should be considered as the last resort: it very slow to complete the whole keyspace.
To discourage you a bit more, look below for the speed table and the "Speed the Things Up"
So, the solution resides in: a) a very good Wordlist, b) better [Rules] AKA "iRules" and some "finesse"
For instance, most websites today requires at 6 to 8 characters, made of at least numbers and letters, and sometimes number(s), letters and signs.
Bad news: It's longer to crack
Good news: The brain size of the average human has NOT increased that much
Hence, before, the dummy would have used "password" as password. Now they use "p@ssw0rd" or "p@55w0rd" and, as usual, people use mostly "words", as proof of concept:
lov3b3ans | Syr4cu$e | C0mmitted | fuzzym0nkey |inth3loop | L1fel0ng , etc etc
So, you end up with 3 choices.
-1 Have a 50,000 terabytes wordlist, along with 25,000 external HDD
-2 Have the patience of a buddhist monk
-3 Have an excellent wordlist coupled with intelligent rule that permutes a for @, append numbers at the end of a word, i.e linkedin1980, etc ..
And your choice is? -3? Good answer!
☞ If you have thought about a new cracking method, It's probably already in JtR. (Matt Weir) (quoted liberally)
John will try "as is" (rule= : ) [(not ;) or ;-)] you dummy, the rule is ":"
then apply funny rules such as "<* >2 !?A l $1" which in good ol' English means: # Lowercase pure alphabetic words and append '1'
As aforementioned, the dummy is absolutely convinced that "password123", or "P@ssw0rd123" is unbreakable, or even unguessable, that is until, our brights readers think about "<* >2 !?A l $123"
Out of the first 330,000 -unique- passwords (meaning dupes removed) that (we)* have cracked from the Linkedin hash dump, 1344 instances of "123" were found, along with 100's of "love", etc etc
Do you catch my drift? The snowball effect: The more you crack, the faster you crack the rest. Up till you reach the crazy ones ... :)
4- Use the most probable rules first and the really funky last, i.e: # Really weird charset conversions, like "england" -> "rmh;smf" (shifting one key to the right)
5- Preferably use a statistically sorted Wordlist, most used first: it will greatly reduce your running time and read the "rules" in the doc folder, you'll be then able to mingle anything without using 50,000 terabytes.
6- Lastly, there is a limit (memory / preprocessor ) on the number/complexity of the rules that you apply. You can go wild, but avoid the "crazy".
The proof of concept is demonstrated in the video.
Optimized john.conf is available for download at the end, but please, read the "Read me" and the command lines first
John The Ripper Commands and examples
the main options are the following
--session=[name]
--wordlist=[path and name]
--format=[format name]
--show [path and name of the hash file]
--restore=[session name]
--rules
--config
--loopback
--markov
--mask
☞ All options have a shorthand, for sake of clarity we here use the longhand.
i.e -ru = --rules ; -w = --wordlist. See /John/doc/examples
☞ To Abort John , use ctrl-c (control c) and NOT CTRL-Z. Also, press ctrl-c one time, not twice.
☞ We've mentioned to place John in a top directory, and to rename "john-1.7.9-jumbo-6 2" into "John". Indeed, you'll have to type command lines, and if you have placed john in a funny far-far away directory , the command line will most likely drive you crazy: Keep it simple!
☞ You can always drag and drop a file into terminal, it will automagically write the path for you. Without errors.
☞ If you do not know the hash type, john will find it for you, but it's always a bit better to force load one type of hash Vs. having John trying multiple hash, sometimes, one hash can be recognized under other names.
So, let's try John with the Linkedin dump (I suppose one could find it on the Internet with minimal Google-Fu competences)
Go to the appropriate directory AKA: /John/run and type
./john [path and name of the Linkedin hash file]
You'll see that John has found multiple possible hash and gives you (few) warnings
Warning: detected hash type "raw-sha1", but the string is also recognized as "raw-sha1-linkedin"
Use the "--format=raw-sha1-linkedin" option to force loading these as that type instead
--format
Hence, pretty clear that you should use, in that case, --format=raw-sha1-linkedin
So the command line goes as:
./john --format=raw-sha1-linkedin [path and name of the hash file]
--session
The --session option allows you to stop a session and restart where you left it later, useful, huh? Then use it!
./john -- session=linkedin --format=raw-sha1-linkedin [path and name of the hash file]
--restore
the --restore option allows you to restart an interupted session, with all the previous parameter just where you left it, pretty handy, huh?
/john -- restore=linkedin
--mask
the --mask option allows you to search a specific keyspace: If you have a good idea of what the password(s) could be, the mask option could save you a billion years or two. (not kidding)
Let's pretend that you forgot a password, and that somehow, you rememberer that it was composed of 3 digits, 3 letters and 3 signs.
?A is for everything
?a is for printable ASCII (English characters only, no funny è or ù)
?d is for digit
?l is for letters
?s is for signs
hence --mask=?d?d?d?l?l?l?s?s?s
In John the default length is 8 max and that also mean that a 10 ch long password will not be (cannot be cracked for sure) by the "All" method alone unless you go under the hood as explained below.
Bruteforce is, generally, and should be considered as the last resort: it very slow to complete the whole keyspace.
To discourage you a bit more, look below for the speed table and the "Speed the Things Up"
So, the solution resides in: a) a very good Wordlist, b) better [Rules] AKA "iRules" and some "finesse"
For instance, most websites today requires at 6 to 8 characters, made of at least numbers and letters, and sometimes number(s), letters and signs.
Bad news: It's longer to crack
Good news: The brain size of the average human has NOT increased that much
Hence, before, the dummy would have used "password" as password. Now they use "p@ssw0rd" or "p@55w0rd" and, as usual, people use mostly "words", as proof of concept:
lov3b3ans | Syr4cu$e | C0mmitted | fuzzym0nkey |inth3loop | L1fel0ng , etc etc
So, you end up with 3 choices.
-1 Have a 50,000 terabytes wordlist, along with 25,000 external HDD
-2 Have the patience of a buddhist monk
-3 Have an excellent wordlist coupled with intelligent rule that permutes a for @, append numbers at the end of a word, i.e linkedin1980, etc ..
And your choice is? -3? Good answer!
☞ If you have thought about a new cracking method, It's probably already in JtR. (Matt Weir) (quoted liberally)
John will try "as is" (rule= : ) [(not ;) or ;-)] you dummy, the rule is ":"
then apply funny rules such as "<* >2 !?A l $1" which in good ol' English means: # Lowercase pure alphabetic words and append '1'
As aforementioned, the dummy is absolutely convinced that "password123", or "P@ssw0rd123" is unbreakable, or even unguessable, that is until, our brights readers think about "<* >2 !?A l $123"
Out of the first 330,000 -unique- passwords (meaning dupes removed) that (we)* have cracked from the Linkedin hash dump, 1344 instances of "123" were found, along with 100's of "love", etc etc
Do you catch my drift? The snowball effect: The more you crack, the faster you crack the rest. Up till you reach the crazy ones ... :)
4- Use the most probable rules first and the really funky last, i.e: # Really weird charset conversions, like "england" -> "rmh;smf" (shifting one key to the right)
5- Preferably use a statistically sorted Wordlist, most used first: it will greatly reduce your running time and read the "rules" in the doc folder, you'll be then able to mingle anything without using 50,000 terabytes.
6- Lastly, there is a limit (memory / preprocessor ) on the number/complexity of the rules that you apply. You can go wild, but avoid the "crazy".
The proof of concept is demonstrated in the video.
Optimized john.conf is available for download at the end, but please, read the "Read me" and the command lines first
John The Ripper Commands and examples
the main options are the following
--session=[name]
--wordlist=[path and name]
--format=[format name]
--show [path and name of the hash file]
--restore=[session name]
--rules
--config
--loopback
--markov
--mask
☞ All options have a shorthand, for sake of clarity we here use the longhand.
i.e -ru = --rules ; -w = --wordlist. See /John/doc/examples
☞ To Abort John , use ctrl-c (control c) and NOT CTRL-Z. Also, press ctrl-c one time, not twice.
☞ We've mentioned to place John in a top directory, and to rename "john-1.7.9-jumbo-6 2" into "John". Indeed, you'll have to type command lines, and if you have placed john in a funny far-far away directory , the command line will most likely drive you crazy: Keep it simple!
☞ You can always drag and drop a file into terminal, it will automagically write the path for you. Without errors.
☞ If you do not know the hash type, john will find it for you, but it's always a bit better to force load one type of hash Vs. having John trying multiple hash, sometimes, one hash can be recognized under other names.
So, let's try John with the Linkedin dump (I suppose one could find it on the Internet with minimal Google-Fu competences)
Go to the appropriate directory AKA: /John/run and type
./john [path and name of the Linkedin hash file]
You'll see that John has found multiple possible hash and gives you (few) warnings
Warning: detected hash type "raw-sha1", but the string is also recognized as "raw-sha1-linkedin"
Use the "--format=raw-sha1-linkedin" option to force loading these as that type instead
--format
Hence, pretty clear that you should use, in that case, --format=raw-sha1-linkedin
So the command line goes as:
./john --format=raw-sha1-linkedin [path and name of the hash file]
--session
The --session option allows you to stop a session and restart where you left it later, useful, huh? Then use it!
./john -- session=linkedin --format=raw-sha1-linkedin [path and name of the hash file]
--restore
the --restore option allows you to restart an interupted session, with all the previous parameter just where you left it, pretty handy, huh?
/john -- restore=linkedin
--mask
the --mask option allows you to search a specific keyspace: If you have a good idea of what the password(s) could be, the mask option could save you a billion years or two. (not kidding)
Let's pretend that you forgot a password, and that somehow, you rememberer that it was composed of 3 digits, 3 letters and 3 signs.
?A is for everything
?a is for printable ASCII (English characters only, no funny è or ù)
?d is for digit
?l is for letters
?s is for signs
hence --mask=?d?d?d?l?l?l?s?s?s
Now, if you were fortunate enough to remember chunks of it, i.e. 123????@?, you could mask such as --mask=123?l?l?l?s@?s hence reducing the keyspace to 19,140,264 possible passwords candidates.
--wordlist
The --wordlist option is for a ... wordlist! yay! with the path if necessary.
If like me you have decided to rename your John directory into JTR, it is suggested that you use the --wordlist option , that is, if you have left your john.conf alone , see line Wordlist = $JOHN/password.lst in file john.conf
☞ Warning! on the 32bit version, if you attempt to use a --wordlist option with a dictionary greater than 2GB you may receive the following error: Error, dictionary file is empty.
1) count the lines with wc -l [filename]
1a) Divide by 2 (roughly) , and round it up to the next integer (full number)
2) Split the dictionary with split -l [#lines] . the output by default is xaa, xab, xac etc .
3) Don't use the the option by size , you'll end up cutting a password in 2.
./john -- session=linkedin --wordlist=masterpswd.txt --format=raw-sha1-linkedin [path and name of the hash file]
Loading multiple wordlists at once
John does not allow, by default, to load multiple Wordlists at once, but you can trick him.
The most probable reason why the maker of JtR have not implemented this option by default resides in a word of caution: Rules are going to be applied as described in your "whatever.conf"
Loading 300 million lines is one thing, having rules applied to 300 million is another altogether: If your Rules generate an output of 10,000 variations on ONE password, one could expect the simple math to be 10,000*300,000,000 = See you in 3 years
Hence: CURB YOUR ENTHUSIASM!
If you wish to do a quick chop-chop on large set of wordlists, edit/create a specific .conf file with very simple and very fast rules, i.e :,l,u,S , etc.
See /doc/Rules and what's below under --rules and --config
1) Create a directory in /John and name it "Wlist" or "Store". Anything you want but keep it short.
2) Place in it your favorites Wordlists, avoiding repeats and duplicates, even in the wordlists.
Solution: Where "Wlist" is the directory containing your Wordlists, and grabthefiles, an argument. You can replace grabmethefiles by anything that fits your mood.
Type: ls /John/Wlist | xargs -t -I grabthefiles ./john --session=xargs --wordlist=/John/Wlist/grabthefiles --rules [etc etc..... ]
☞ " xargs -t -I " it's a overscore "i" , not a l,L or | --> "xargs -t -I"
☞ xargs calls multiple times the same ./john command line changing each time for the correct wordlist, be aware that it is highly recommended to use --session otherwise, if you have to pause you'll have to restart from scratch.
The ETA is for each wordlists and not for the total time.
--rules
The --rules option call for a specific set of rules as defined in John.conf under names such as [List.Rules:Wordlist]
using --rules will call for all the rules, if you want to use only a specific set, i.e wordlist, then use --rules=wordlist
./john -- session=linkedin --wordlist=masterpswd.txt --rules=wordlist --format=[hash format] [path and name of the hash file]
☞ Be aware that calling a large set of rules could lead to some extra-extra time, always check that you are not performing a duplicate work. i.e calling twice the same rule or asking to lowercase an already all lower case wordlist.
☞ Rules are processed by a preprocessor. Writing extremely large rules or set of rules will send your memory usage off the chart. See John.log
--show
The option --show shows you the cracked hashes, it's best used with a redirection to a file
./john -- show [path and name of the hash file] > showpass.txt
To see what's left to crack, use --show=left
☞ The option --show works faster if you include the --format option
./john --show=left --format=[name] [path and name of the hash file] > showhash.txt
--config
The option --config=[name] allows you to use multiple configurations and set of rules. You can create your own configuration files and use it at will, i.e a "mini.conf" configuration to crack the easy one first. Don't forget that a wordlist that have 100m passphrases in it, mingled with 100 rules equals a 100x100m words equivalent. Translated in space, it like having, about, 95,000 MB worth of space. 95 TeraBytes
☞ See at the end for downloadable Config/Rules files
☞ See "Making List.Rules" before making 99 different config files
--loopback
--loop
A rarely used option. Too bad, it's a good one! The --loopback allows you to check against your john.pot file. You have already cracked some passwords and they are all stored in john.pot, hence you can re-use them for a quick check.
For a longer check, I'll give you a hint: those are real passwords: the ham not the spam! so re-use them with rules: If you have found "password" , you'll find "password123"
--Markov
A statistical attack derived from probabilistic stochastic processes. AKA Markov's Chains.
I am scaring the shit of of you with Markov, huh? I am not?
Well... don't be afraid ... I don't understand it myself. I read what I could, got a serious headache and decided not to pursue.
All I know is that it's working!
As a matter of fact it's working pretty darn well and it's really fast too.
Mr Andrey (Andrei) Andreyevich Markov (1856-1922) must be thrilled to know that we're all very satisfied with his research. Thank you Mr Markov!
As I don't understand the intricate details of Mr Markov thoughts, I switched to a highly scientific proven method: "The Empirical Poke Around"™
The --wordlist option is for a ... wordlist! yay! with the path if necessary.
If like me you have decided to rename your John directory into JTR, it is suggested that you use the --wordlist option , that is, if you have left your john.conf alone , see line Wordlist = $JOHN/password.lst in file john.conf
☞ Warning! on the 32bit version, if you attempt to use a --wordlist option with a dictionary greater than 2GB you may receive the following error: Error, dictionary file is empty.
1) count the lines with wc -l [filename]
1a) Divide by 2 (roughly) , and round it up to the next integer (full number)
2) Split the dictionary with split -l [#lines] . the output by default is xaa, xab, xac etc .
3) Don't use the the option by size , you'll end up cutting a password in 2.
./john -- session=linkedin --wordlist=masterpswd.txt --format=raw-sha1-linkedin [path and name of the hash file]
Loading multiple wordlists at once
John does not allow, by default, to load multiple Wordlists at once, but you can trick him.
The most probable reason why the maker of JtR have not implemented this option by default resides in a word of caution: Rules are going to be applied as described in your "whatever.conf"
Loading 300 million lines is one thing, having rules applied to 300 million is another altogether: If your Rules generate an output of 10,000 variations on ONE password, one could expect the simple math to be 10,000*300,000,000 = See you in 3 years
Hence: CURB YOUR ENTHUSIASM!
If you wish to do a quick chop-chop on large set of wordlists, edit/create a specific .conf file with very simple and very fast rules, i.e :,l,u,S , etc.
See /doc/Rules and what's below under --rules and --config
1) Create a directory in /John and name it "Wlist" or "Store". Anything you want but keep it short.
2) Place in it your favorites Wordlists, avoiding repeats and duplicates, even in the wordlists.
Solution: Where "Wlist" is the directory containing your Wordlists, and grabthefiles, an argument. You can replace grabmethefiles by anything that fits your mood.
Type: ls /John/Wlist | xargs -t -I grabthefiles ./john --session=xargs --wordlist=/John/Wlist/grabthefiles --rules [etc etc..... ]
☞ " xargs -t -I " it's a overscore "i" , not a l,L or | --> "xargs -t -I"
☞ xargs calls multiple times the same ./john command line changing each time for the correct wordlist, be aware that it is highly recommended to use --session otherwise, if you have to pause you'll have to restart from scratch.
The ETA is for each wordlists and not for the total time.
--rules
The --rules option call for a specific set of rules as defined in John.conf under names such as [List.Rules:Wordlist]
using --rules will call for all the rules, if you want to use only a specific set, i.e wordlist, then use --rules=wordlist
./john -- session=linkedin --wordlist=masterpswd.txt --rules=wordlist --format=[hash format] [path and name of the hash file]
☞ Be aware that calling a large set of rules could lead to some extra-extra time, always check that you are not performing a duplicate work. i.e calling twice the same rule or asking to lowercase an already all lower case wordlist.
☞ Rules are processed by a preprocessor. Writing extremely large rules or set of rules will send your memory usage off the chart. See John.log
--show
The option --show shows you the cracked hashes, it's best used with a redirection to a file
./john -- show [path and name of the hash file] > showpass.txt
To see what's left to crack, use --show=left
☞ The option --show works faster if you include the --format option
./john --show=left --format=[name] [path and name of the hash file] > showhash.txt
--config
The option --config=[name] allows you to use multiple configurations and set of rules. You can create your own configuration files and use it at will, i.e a "mini.conf" configuration to crack the easy one first. Don't forget that a wordlist that have 100m passphrases in it, mingled with 100 rules equals a 100x100m words equivalent. Translated in space, it like having, about, 95,000 MB worth of space. 95 TeraBytes
☞ See at the end for downloadable Config/Rules files
☞ See "Making List.Rules" before making 99 different config files
--loopback
--loop
A rarely used option. Too bad, it's a good one! The --loopback allows you to check against your john.pot file. You have already cracked some passwords and they are all stored in john.pot, hence you can re-use them for a quick check.
For a longer check, I'll give you a hint: those are real passwords: the ham not the spam! so re-use them with rules: If you have found "password" , you'll find "password123"
--Markov
A statistical attack derived from probabilistic stochastic processes. AKA Markov's Chains.
I am scaring the shit of of you with Markov, huh? I am not?
Well... don't be afraid ... I don't understand it myself. I read what I could, got a serious headache and decided not to pursue.
All I know is that it's working!
As a matter of fact it's working pretty darn well and it's really fast too.
Mr Andrey (Andrei) Andreyevich Markov (1856-1922) must be thrilled to know that we're all very satisfied with his research. Thank you Mr Markov!
As I don't understand the intricate details of Mr Markov thoughts, I switched to a highly scientific proven method: "The Empirical Poke Around"™
Hence, I poked around. Empirically.
First things first.
1) grab a high quality password file. Please be sure to use a US95 ASCII txt file (no é or ñ) unless you are using the -p option
2) in /John/run type ./calc_stat <name of dictionary file> stats
2) in /John/run type ./calc_stat <name of dictionary file> stats
3) let it work, it should take about 10 min, based on the Dic size.
When done type your average John command line, this time, include --Markov
./John --markov [path and name of hash file]
On the screen you'll see something as: MKV start (stats=$JOHN/stats, lvl=200 len=12 pwd=238032699)
and rapidly, Mr Markov will go to work ...
len=12 is the max password length to be tested
lvl=200 is the level.
pwd=238,032,699 is ...well, you've guessed.
"lvl" is what is going to increase or decrease the number of password generated. Try to stick between 200 and 250. At 300 my config generated 6,243,105,973,969 passwords. It's a bit too much. See the "Empirical Poke Around Graph"
You can modify those parameters in John.conf , or directly with --markov[modes]
./john --markov=230:0:0:12 (level, start, end, length)
The 0:0 values are start/end in %. 0:0 being the default, otherwise you need to put a %. i.e --markov=200:10%:20%:12
☞ Be aware that there is a bit of a catch: The Markov's Swiss Cheese Illogism™
- The lower the level, the lower the number of password generated, but the greater the quality, hence the fastest time... To crack the lowest amount of hashes.
- On the other hand: The higher the level, the greater the number of passwords generated, but with the lowest quality, hence the longest time... To crack the highest amount of hashes.
Do you follow me? The more holes in cheese, the less cheese. But the less cheese, the more holes. If the more holes, the more cheese. Hence more holes = more cheese.
Roughly to say that if you push the enceloppe a bit too high, you'll have such a large number of possibilities that you'll be close to bruteforcing. Keep the numbers between 150 and 250. (Surprise, the default value is 200)
The 0:0 values are start/end in %. 0:0 being the default, otherwise you need to put a %. i.e --markov=200:10%:20%:12
☞ Be aware that there is a bit of a catch: The Markov's Swiss Cheese Illogism™
- The lower the level, the lower the number of password generated, but the greater the quality, hence the fastest time... To crack the lowest amount of hashes.
- On the other hand: The higher the level, the greater the number of passwords generated, but with the lowest quality, hence the longest time... To crack the highest amount of hashes.
Do you follow me? The more holes in cheese, the less cheese. But the less cheese, the more holes. If the more holes, the more cheese. Hence more holes = more cheese.
Roughly to say that if you push the enceloppe a bit too high, you'll have such a large number of possibilities that you'll be close to bruteforcing. Keep the numbers between 150 and 250. (Surprise, the default value is 200)
Markvov Vs. Bruteforcing
I had about 1.5 million hashes left and it was getting harder to crack the remaining ones. Then I decided to test Markov Vs Bruteforcing, the -surprising- results are as follow:
Markov: guesses: 18,072 time: 0:01:30:00 c/s: 6654G
Bruteforce: guesses: 4,590 time: 1:07:04:57 c/s: 3817G
c/s Speed: Markov is 74% Faster than bruteforce: More bang for your buck!
r/t ( Passwords recovered / Tries ) Markov wins!
Recovery speed: Markov recovered 3.34 pswd/sec on average, Bruteforce 0.85 pswd/sec, which make Markov 394% faster
Markov will not recover all passwords, and cannot guess true random one, but you can attempt passwords length greater than bruteforce, on the other hand Bruteforce will recover any password under 10ch long. If you have the time.
EPAG
Empirical Poke Around Graph
Bruteforce: guesses: 4,590 time: 1:07:04:57 c/s: 3817G
c/s Speed: Markov is 74% Faster than bruteforce: More bang for your buck!
r/t ( Passwords recovered / Tries ) Markov wins!
Recovery speed: Markov recovered 3.34 pswd/sec on average, Bruteforce 0.85 pswd/sec, which make Markov 394% faster
Markov will not recover all passwords, and cannot guess true random one, but you can attempt passwords length greater than bruteforce, on the other hand Bruteforce will recover any password under 10ch long. If you have the time.
EPAG
Empirical Poke Around Graph
John.pot
John keeps a list of already cracked hash in run/John.pot, keep that file safe ...
the format looks like:
$dynamic_26$32bf58ba66ef765e214ca1b6fe70244dfe5b5427:link@din2010
[header:hash:password]
You can easily trim the fat with cut -c 51- [filename] > output.txt
51- being the # of ch to remove. yes, it is "51 dash" "dash 51" will leave the first 51 ch.
As many times mentioned, because we've found "link@din2010" , one could bet that we will find something close to it. Another genius has thought very hard and used "name of the website + year" or something in the same family ... So of course, we found relatives...
Here is the brother ...
2ce773d7ffbbe7491e51272783d1c6f411a1d34b:link@din2012
And the uncles...
033a2970983d5675152a6213d7d4228bc977399c:linkedinpass1234
6a9535be0d9ef156d5fa7465bd419d15843bb2b0:linkedinpass1402
01679a9ea64997f7e693159670329061c9dd22c3:linkedinpass1423
5077ab2005d5253f9b55b9b0979aa9401fcad706:linkedinpass1704
Bruteforce Vs. Wordlists + Rules
Bruteforce is extremely, extremely time consuming and should be used only as last resort, specifically for long -er passwords: Bruteforcing 2 to 5 ch long is fairly fast and easy, 6 to 7 is medium, 8 and above will test your determination. No need to mention 15 or 16...
Hence, you've already cracked the easy ones... i.e "password" you might want to re-use them.
As mentioned above, john.pot keeps track of the hashes previously cracked and will skip them and it's a good thing: No need to work on already cracked hashes. You should always try to chop-chop trough the easy ones first, then use bruteforce on last resort for the complex ones.
If you find "Linkedin", an (existing) rule can easily double the password: "LinkedinLinkedin" , then once either re-injected in a dictionary, re-used with --loopback or created via Rules, you could find "LinkedinLinkedin2012" or "L!nk3d1nL!nk3d1n2012"
Finding such "L!nk3d1nL!nk3d1n" by mean of simple bruteforce is not a job for you! it's a job for your grandgrandgrandgrand son, or daughter: Seriously, limit yourself to less than 8 ch long unless you own a very-very fast machine.
What Mr Dingbat could have done was to use roman characters, i.e é, ñ, an uppercase, sign, and number. i.e 1966L!ñkédin1966 That would have forced you on the whole range of printable ASCII. You will be then, pretty much dead in the water.
Don't get me wrong, it is still a stupid password because of the pattern, but it's a whole lot stronger ... by a factor of 104, or 10,406% stronger
Now, take the same stuff and use Rules .... What was extremely far far away is now feasible: we know that "linkedin" is a big fave, we know that people like appending and/or prefixing numbers, and we know that using "!" for "i" will not get you a Nobel Prize.
Proof that it's easily feasible with Rules? I did it!
Check the quick analysys of the Linkedin passwords at the end.
Bruteforcing Speed
Disclaimer: This section was specifically written to discourage you.
The good news is that JtR does not bruteforce stupidly in order (0, 00, 000 etc) JtR uses an algorithm to generate a smart-er bruteforcing order. Nevertheless...
Bruteforcing 4 ch long is very fast, 5 is ok, 6 is slow, etc
here are the speed for an average machine, one core running, No GPU, on raw sha1,
length Keyspace Key/sec Time
1 to 4 ch long 82,317,120 ~ 2,655,391 31 Seconds
1 to 5 7,820,126,495 ~ 2,655,391 46 minutes
1 to 6 742,912,017,120 ~ 2,655,391 73 hours
1 to 7 70,576,641,626,495 ~ 2,655,391 289 days
1 to 8 6,704,780,954,517,020 ~ 2,655,391 75 years
There you have it.
Even if you have a fast machine, you'll still be waiting for few years. In order to go into the "Weeks" or "Days", it's GPU assisted only. A good GC/GPU will take you into warp speed, tests on This Card
are showing results as high as 8.2 billion passwords/second.With a single of one of those, your ETA goes from 75 years to 23 hours.
One notable counter intuitive perception is that, the smaller the hash file, the slower the speed (in c/s, not in overall time) and not the logical opposite, and this is due on the inner working of any Cracking program, lemme explain...
A Cracking program does not "guess" or reverse engineer the hash, it simply compares it.
1) You feed the program a test password, i.e password1
2) A hash is generated: e38ad214943daad1d64c102faec29de4afe9da3d
3) The program compares it with the list you provided.
If there is a match, good news. If not, then try again with another test password.
And that's why: It's faster to compare 1 generated hash against a large list of existing ones than to have to generate a lot of hashes to compare to a small list. Generate one, compare to millions Vs. Generate millions to compare to few.
6,2 million hash list c/s speed = 4,284,000,000,000 c/s
10 (ten) hash list c/s speed = 17,859,000 c/s
Speed the thing up! (Part One)
You can go two ways: get the fastest machine or get smart:
Get smart is even faster then the fastest machine...
Get some geeky friend to help you.
1) Each member of the team choose a mode or a set of different rules, and start working on it's own piece. Going trough long rules alone can be a long long project, when the rules/modes are divided amongst individuals it works much faster.
In the mean time, share a dropbox folder with your friends.
After 45 min or so, do a ctrl-c to stop the session and do a:
./john --show [hashfile] > /dropbox/public/sharedfolder/showpass_myname.txt
Your friends having done the same, you have x password files. Use Textwrangler to concatenate them, clean and remove the dupes in it, and rename as showpass_all.txt, you can now use that file to expand your wordlist(s), i.e a common bigdic.txt that everyone will use (put the new password first) Each time someone will update the file, Dropbox will update it too, hence you always have the latest copy. Simply point John --wordlist= to that location and from time to time, restart the crack under different session name.
The concepts behind it are:
a) Distribution of the attack: You have now 12, 20 or even 40 cores working
b) Faster use of complex rules
c) Greater chances to crack reoccurring passwords much faster: In a dump like Linkedin, some patterns passwords are present up to 900 times, i.e link, l!nked!n , etc
If you had a 40 cores, you still have to go trough the hash one by one.
As you are now criss-crossing your results with your friends using a file that contains everybody's result, you do not need to spend time crack a password already found.
Remember? the first rule of wordlist is "as is"
This solution is faster than "40 cores" alone. Don't reinvent the wheel....
(we)* : Thanks to:
Terek, Score 40%
Trae, Score 0% (aka ADHD Boy) QuadCore i7 running on one leg ;)
Speed the thing up! (Part Two)
let's be clear: Rules Rules!
Without Rules, Markov aside, you would be condemned to two main "solutions"
1) Endless Wordlist, 50,000 TeraBytes of it, and $200,000 worth of Hard Drive.
2) Bruteforcing for the next century, and a $200,000 utility bill.
Rules allow you to generate or re-generate new passwords to be tested, and also, to do it much faster than anything else, plus you would NOT have the space to store such wordlists.
The KISF Rule
Keep it Simple First!
Before going bonkers with complex rules (that will make you proud) why don't you try the simple ones first? They are fast and replicate human mnemonic patterns: Capitalize, Upper, Mirror, duplicate, etc etc. Did I mention the docs? Where rules are explained?
Rules Vs. Endless Wordlists
Rules are not as fast as reading directly, (yes, you are using some CPU/memory to generate them) but the space saved is huge. Futhermore, you can turn on/off rules, add, mingle or remove , etc. With gigormous wordlist, (aka bragging rights wordlists on the internet for 100 or 200GB list*) you have to store them, edit them -if you can-, etc . Good luck with that. The solution resides again in an excellent set of Wordlist and a solid set of rules. ( *Bragging rights: 200 GB WPA(?) Wordlist )
The caveat is that, reading rules, understanding and editing them will give you a serious headache. Example: -[:c] l /a /l /[os] sa4 sl1 s\0\p[0$] (?\p1[za] \p1[:c] Well, that's not the most friendly thing to understand.
...Knowing grep will help you a bit, but ... as you've read so far... you deserve a little bonus for your perseverance:
John The Ripper Rules Generator!
- Yay!
By Matt Weir **
No only you will generate rules easily, without needing a Costco sized bottle of aspirin, but you'll quickly learn the tricks and syntax. Otherwise, read the doc in JtR and order aspirin by the pound.
i.e , Swap Case of all Keyboard Values, Reverse, Concatenate, even modify the Character Set. G-d bless you Matt Weir!
Where? : https://sites.google.com/site/reusablesec2/jtrconfiggenerator
Prerequisite? : Xcode / gcc
How? : sudo --user --shall readtheefingdoc
run? : ./jtrMakeConfig
If for whatever reason you do NOT wish to use the JtR Configuration Generator, you'll want to look at one of those things... It will feel the same ...
a big PIA
Nevertheless, always check your rules with a small test file, just to avoid duplicates or very slow stuff: Loading 50,000 rules is un-efficient. Having an ETA of +50 days should be a sign...Believe me, I have made mistakes ...
Strategies ..
- First, when you hear about a large dump being released, you Google it and try to find the dump. Seconds after the download started, you should start Google-ing again for the first cracked passwords: Someone out there is faster than you, has a better machine(s), and has already cracked few of them. In order to comply with the "Bragging Rights Theory", short list may have been published, those will help you a lot to start your crack. Get them, and keep on Googling from time to time.
Also, it's highly possible that by the time you find a link or a paste, the sole answer will be "content no longer available". In that case one could thank Google for the feature called Google Cache. Thank you Mr Google for that. Try it ;-)
- Start small, good dictionary, small rules: Each time a hash is cracked, it's stored in John.pot and will not be reloaded. Each time it's cracked, JtR goes faster ....
- Check the (hash) source if you can, i.e the website. For instance, American Express has stupid rules for passwords: no spaces, only % & _ ? # = - and best of all, is not case sensitive. Hence, it has greatly reduced your workload. If a website does not allow certain charaters, just don't use them. If the website has a policy of "minimum 8 ch long", don't attempt to bruteforce 0-7. You'll save few hours. (If you have ever attempted to crack a zero Ch. long password, please seek medical attention immediately)
- use --loopback and from time to time , append your wordlist with the already cracked passwords
- use pipal or PACK to generate an analysis of what's been cracked, always useful to see a pattern or the most used strings
In the case of the Linkedin dump for example, you can try a small bruteforcing in the middle, 1-6 ch long, you'll always find something to chew on, 7 ch is slow, 8 is very slow...
- use the option --mkpc=1 and check your john.log file: You'll see which rule is associated with the cracked password, then make an educated guess...
Making List.Rules
Open John.conf with TextWrangler and start looking for lines that contains [List.Rules."Whatever"]
Those are "paragraphs" stating a set of rules to be called by the option --rules="whatever"
i.e --rules=jumbo
You can create your own paragraphs with your own set of rules, i.e [List.Rules.Veryfast]
Include in it "very fast" rules such as "as is" "Capitalize" "Duplicate" etc ...
it might look like that ...
##### Veryfast #####
[List.Rules.Veryfast]
# no-op: do nothing to the input word
:
# convert to lowercase
l
# convert to uppercase
u
# capitalize
c
# lowercase the first character, and uppercase the rest
C
# toggle case of all characters in the word
t
# reverse: "Fred" -> "derF"
r
# duplicate: "Fred" -> "FredFred"
d
# reflect: "Fred" -> "FredderF"
f
# rotate the word left: "jsmith" -> "smithj"
{
# rotate the word right: "smithj" -> "jsmith"
}
##### End Of Veryfast #####
☞ Before going wild, double check that you are not calling duplicate rules, or you'll be doing double work for nothing. i.e ":" (Trying "as is", No op ) is already called in [List.Rules.Single], [List.Rules.Wordlist]. As [List.Rules.Jumbo] also calls "Wordlist", if you add an extra one, your hash will only be checked for a fourth time. ;-)
Length control
If you know that the password you are looking for is of a certain character length, or range, i.e 8ch long, there is no need for you to check anything outside that range. In that case limit the search by adding a character length limitation, you can gain a huge amount of time.
Example: limiting to 8 ch long
# Trying 8 ch long only
[List.Rules:8only]
>7'8
# End of Trying 8 ch long only
The first >7 reject any word that is less than 8 ch long
The second '8 truncates words greater than length 8
You can, of course, add this control to other rule commands, i.e
$[0-9]>7'8
The above suffix a number and limits the length of the password tested.
IF the password tested is:
America --> America0 , then America1 etc etc...
IF the password tested is:
UnitedStates --> UnitedSt (truncated at 8, without adding the digits)
Rules that calls Rules
Look for the Rule called [List.Rules.Jumbo] and you'll quickly understand that you can call sub-rules. [List.Rules.Jumbo] calls the rules "Wordlist" then "Single" then "Extra", etc.
Example:
# For Wordlist mode and very fast hashes
[List.Rules:Jumbo]
.include [List.Rules:Wordlist]
.include [List.Rules:Single]
.include [List.Rules:Extra]
.include [List.Rules:NT]
Hence, you can also create your own [List.Rules.Myrules] by using .include [List.Rules:Whatever]
Again, be aware that some rules are going to be called twice when using "Jumbo", or that you'll call subrules twice too, hence double check -again- that your set does not call again the same rules over and over again, i.e using [List.Rules:Jumbo] and [List.Rules:Single-Extra]
Ordered Rules?
Let's go back to Mr Dingbat and Friends, The append/prefix a number to a password is obvious. In JtR, the rule to append a digit to a password is $ "your digit" such as $a will add an "a" at the end .For a number $1, for a range $[0-9] or $[0123456789]
Got it?
$a password >> passworda
$[123] password >> password1, password2, password3 (no 4 and 5, you didn't ask)
To be able to add more than, you repeat ...
$[0-9]$[0-9]$[0-9] password >> password000 to password999
hence, the following
$[0-9]
$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]$[0-9]$[0-9]
Will go from append 0 to 999,999. Repeat the same rule but interchange $ for ^, and you'll prefix.
You have now 999,999 + 999,000 permutation per password.
Take a 50m line wordlist and John will develop it into a virtual 99,999,950,000,000 password list. That's 99,999 billions. So, again, to wrap your mind around it, it's about 5,500 2TB Hard Drive worth of space,or in moolah: $675,000 (Geez, the money you save reading this blog)
So, Rules is a no brainer, but .... 99,999,950,000,000 passwords, it's going to take a while... ETA +30 days and the question is asked: Do the order of rules impact the cracking speed?
Yes and yes
a) The more cracked passwords, the greater your chance to see a pattern, leading to more passwords being recovered.
b) Due to the human nature, what would be more unlikely? Mr Dingbat appending and prefixing 5 or 2 digits?
Let's grep it and we'll find out!
Out of about 50% of the passwords cracked, the distribution indicates such patterns:
abc..00 593,914 hits , or 17% of the total
abc..0000 400,114 11.4%
abc..0 367,969 10.5%
abc..000 219,942 6.3%
0abc... 79,224 2.3%
down to ..
00000abc.. 5,200 0.1%
Hence the logical order of trying 1,2,3,4,5,6 digit either after or before is not the most optimized and will take you longer. The proper order in that case would be:
(Warning slow stuff!!!)
$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]
$[0-9]
$[0-9]$[0-9]$[0-9]
^[0-9]
^[0-9]^[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]$[0-9]^[0-9]
^[0-9]^[0-9]^[0-9]
^[0-9]^[0-9]^[0-9]^[0-9]^[0-9]
^[0-9]^[0-9]^[0-9]^[0-9]
Terek, Score 40%
Trae, Score 0% (aka ADHD Boy) QuadCore i7 running on one leg ;)
Speed the thing up! (Part Two)
let's be clear: Rules Rules!
Without Rules, Markov aside, you would be condemned to two main "solutions"
1) Endless Wordlist, 50,000 TeraBytes of it, and $200,000 worth of Hard Drive.
2) Bruteforcing for the next century, and a $200,000 utility bill.
Rules allow you to generate or re-generate new passwords to be tested, and also, to do it much faster than anything else, plus you would NOT have the space to store such wordlists.
The KISF Rule
Keep it Simple First!
Before going bonkers with complex rules (that will make you proud) why don't you try the simple ones first? They are fast and replicate human mnemonic patterns: Capitalize, Upper, Mirror, duplicate, etc etc. Did I mention the docs? Where rules are explained?
Rules Vs. Endless Wordlists
Rules are not as fast as reading directly, (yes, you are using some CPU/memory to generate them) but the space saved is huge. Futhermore, you can turn on/off rules, add, mingle or remove , etc. With gigormous wordlist, (aka bragging rights wordlists on the internet for 100 or 200GB list*) you have to store them, edit them -if you can-, etc . Good luck with that. The solution resides again in an excellent set of Wordlist and a solid set of rules. ( *Bragging rights: 200 GB WPA(?) Wordlist )
The caveat is that, reading rules, understanding and editing them will give you a serious headache. Example: -[:c] l /a /l /[os] sa4 sl1 s\0\p[0$] (?\p1[za] \p1[:c] Well, that's not the most friendly thing to understand.
...Knowing grep will help you a bit, but ... as you've read so far... you deserve a little bonus for your perseverance:
John The Ripper Rules Generator!
- Yay!
By Matt Weir **
No only you will generate rules easily, without needing a Costco sized bottle of aspirin, but you'll quickly learn the tricks and syntax. Otherwise, read the doc in JtR and order aspirin by the pound.
i.e , Swap Case of all Keyboard Values, Reverse, Concatenate, even modify the Character Set. G-d bless you Matt Weir!
Where? : https://sites.google.com/site/reusablesec2/jtrconfiggenerator
Prerequisite? : Xcode / gcc
How? : sudo --user --shall readtheefingdoc
run? : ./jtrMakeConfig
If for whatever reason you do NOT wish to use the JtR Configuration Generator, you'll want to look at one of those things... It will feel the same ...
a big PIA
Nevertheless, always check your rules with a small test file, just to avoid duplicates or very slow stuff: Loading 50,000 rules is un-efficient. Having an ETA of +50 days should be a sign...Believe me, I have made mistakes ...
Strategies ..
- First, when you hear about a large dump being released, you Google it and try to find the dump. Seconds after the download started, you should start Google-ing again for the first cracked passwords: Someone out there is faster than you, has a better machine(s), and has already cracked few of them. In order to comply with the "Bragging Rights Theory", short list may have been published, those will help you a lot to start your crack. Get them, and keep on Googling from time to time.
Also, it's highly possible that by the time you find a link or a paste, the sole answer will be "content no longer available". In that case one could thank Google for the feature called Google Cache. Thank you Mr Google for that. Try it ;-)
- Start small, good dictionary, small rules: Each time a hash is cracked, it's stored in John.pot and will not be reloaded. Each time it's cracked, JtR goes faster ....
- Check the (hash) source if you can, i.e the website. For instance, American Express has stupid rules for passwords: no spaces, only % & _ ? # = - and best of all, is not case sensitive. Hence, it has greatly reduced your workload. If a website does not allow certain charaters, just don't use them. If the website has a policy of "minimum 8 ch long", don't attempt to bruteforce 0-7. You'll save few hours. (If you have ever attempted to crack a zero Ch. long password, please seek medical attention immediately)
- use --loopback and from time to time , append your wordlist with the already cracked passwords
- use pipal or PACK to generate an analysis of what's been cracked, always useful to see a pattern or the most used strings
In the case of the Linkedin dump for example, you can try a small bruteforcing in the middle, 1-6 ch long, you'll always find something to chew on, 7 ch is slow, 8 is very slow...
- use the option --mkpc=1 and check your john.log file: You'll see which rule is associated with the cracked password, then make an educated guess...
Making List.Rules
Open John.conf with TextWrangler and start looking for lines that contains [List.Rules."Whatever"]
Those are "paragraphs" stating a set of rules to be called by the option --rules="whatever"
i.e --rules=jumbo
You can create your own paragraphs with your own set of rules, i.e [List.Rules.Veryfast]
Include in it "very fast" rules such as "as is" "Capitalize" "Duplicate" etc ...
it might look like that ...
##### Veryfast #####
[List.Rules.Veryfast]
# no-op: do nothing to the input word
:
# convert to lowercase
l
# convert to uppercase
u
# capitalize
c
# lowercase the first character, and uppercase the rest
C
# toggle case of all characters in the word
t
# reverse: "Fred" -> "derF"
r
# duplicate: "Fred" -> "FredFred"
d
# reflect: "Fred" -> "FredderF"
f
# rotate the word left: "jsmith" -> "smithj"
{
# rotate the word right: "smithj" -> "jsmith"
}
##### End Of Veryfast #####
☞ Before going wild, double check that you are not calling duplicate rules, or you'll be doing double work for nothing. i.e ":" (Trying "as is", No op ) is already called in [List.Rules.Single], [List.Rules.Wordlist]. As [List.Rules.Jumbo] also calls "Wordlist", if you add an extra one, your hash will only be checked for a fourth time. ;-)
Length control
If you know that the password you are looking for is of a certain character length, or range, i.e 8ch long, there is no need for you to check anything outside that range. In that case limit the search by adding a character length limitation, you can gain a huge amount of time.
Example: limiting to 8 ch long
# Trying 8 ch long only
[List.Rules:8only]
>7'8
# End of Trying 8 ch long only
The first >7 reject any word that is less than 8 ch long
The second '8 truncates words greater than length 8
You can, of course, add this control to other rule commands, i.e
$[0-9]>7'8
The above suffix a number and limits the length of the password tested.
IF the password tested is:
America --> America0 , then America1 etc etc...
IF the password tested is:
UnitedStates --> UnitedSt (truncated at 8, without adding the digits)
Rules that calls Rules
Look for the Rule called [List.Rules.Jumbo] and you'll quickly understand that you can call sub-rules. [List.Rules.Jumbo] calls the rules "Wordlist" then "Single" then "Extra", etc.
Example:
# For Wordlist mode and very fast hashes
[List.Rules:Jumbo]
.include [List.Rules:Wordlist]
.include [List.Rules:Single]
.include [List.Rules:Extra]
.include [List.Rules:NT]
Hence, you can also create your own [List.Rules.Myrules] by using .include [List.Rules:Whatever]
Again, be aware that some rules are going to be called twice when using "Jumbo", or that you'll call subrules twice too, hence double check -again- that your set does not call again the same rules over and over again, i.e using [List.Rules:Jumbo] and [List.Rules:Single-Extra]
Ordered Rules?
Let's go back to Mr Dingbat and Friends, The append/prefix a number to a password is obvious. In JtR, the rule to append a digit to a password is $ "your digit" such as $a will add an "a" at the end .For a number $1, for a range $[0-9] or $[0123456789]
Got it?
$a password >> passworda
$[123] password >> password1, password2, password3 (no 4 and 5, you didn't ask)
To be able to add more than, you repeat ...
$[0-9]$[0-9]$[0-9] password >> password000 to password999
hence, the following
$[0-9]
$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]$[0-9]$[0-9]
Will go from append 0 to 999,999. Repeat the same rule but interchange $ for ^, and you'll prefix.
You have now 999,999 + 999,000 permutation per password.
Take a 50m line wordlist and John will develop it into a virtual 99,999,950,000,000 password list. That's 99,999 billions. So, again, to wrap your mind around it, it's about 5,500 2TB Hard Drive worth of space,or in moolah: $675,000 (Geez, the money you save reading this blog)
So, Rules is a no brainer, but .... 99,999,950,000,000 passwords, it's going to take a while... ETA +30 days and the question is asked: Do the order of rules impact the cracking speed?
Yes and yes
a) The more cracked passwords, the greater your chance to see a pattern, leading to more passwords being recovered.
b) Due to the human nature, what would be more unlikely? Mr Dingbat appending and prefixing 5 or 2 digits?
Let's grep it and we'll find out!
Out of about 50% of the passwords cracked, the distribution indicates such patterns:
abc..00 593,914 hits , or 17% of the total
abc..0000 400,114 11.4%
abc..0 367,969 10.5%
abc..000 219,942 6.3%
0abc... 79,224 2.3%
down to ..
00000abc.. 5,200 0.1%
Hence the logical order of trying 1,2,3,4,5,6 digit either after or before is not the most optimized and will take you longer. The proper order in that case would be:
(Warning slow stuff!!!)
$[0-9]$[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]
$[0-9]
$[0-9]$[0-9]$[0-9]
^[0-9]
^[0-9]^[0-9]
$[0-9]$[0-9]$[0-9]$[0-9]$[0-9]^[0-9]
^[0-9]^[0-9]^[0-9]
^[0-9]^[0-9]^[0-9]^[0-9]^[0-9]
^[0-9]^[0-9]^[0-9]^[0-9]
Charset making
This section is specific to part 2) Bruteforcing Longer than 8ch long
Having modified params.h is a prerequisite, as described in 2)
You can always re-compile if you've missed it the first time.
Before doing it, we recommend that your John.pot has gain some "fat": crack few hashes
1) Open Terminal and go to your /John/run folder
1a) Run ./John --make-charset=[choose a name].chr
i.e ./John --make-charset=longforce.chr
hit return and wait for John to finish
2) Open John.conf with TextWrangler
2a) Look for the section containing " # Incremental Modes"
2b) copy the part
[Incremental:All]
File = $JOHN/all.chr
MinLen = 1
MaxLen = 8
CharCount = 95
Paste below and change the parameters to match your hopes, example:
[Incremental:longforce]
File = $JOHN/longforce.chr
MinLen = 10
MaxLen = 15
CharCount = 95
"longforce" being the incremental name, "longforce.chr" the charset name, "MinLen" the Minimum paswd length tested , and MaxLen the max one.
Min/max 10-15 is already a stupid long keyspace, hence you can create as many section as you wish , i.e "longforce13" and only attempt to Bruteforce 13 long passwords
MinLen = 13
MaxLen = 13
( I would limit myself to a specific time frame, i.e a day or two.)
When ready, just run ./john --incremental=longforce13 [other useful parameters ....]
☞ You can always run one instance of John per Core. 4 Cores, 4 Instances. Each of them running a separate "bruteforcing" length i.e 10,11,12,13 ch long.
That should keep you busy for a while...
** Matt Weir
If you want the advice of a real specialist, you might want to read Matt Weir's blog
http://reusablesec.blogspot.com/
Matt has now left this earth... Well...no, he is still alive, he has just stopped blogging for professional reason. He's a bit like Obi-Wan Kenobi: not here anymore, but you can still feel The Force.
Just like Elvis, there have been some sightings! The last report of a Matt Weir sighting was at DEFCON2012. If you have experienced such, please report your findings immediately.
Matt Weir has also made some optimized (scientifically optimized, (It's Dr. Weir)) Config, you can read about it, and even download them:
Matt Weir's Pack o' Configs You M.U.S.T read the little "read me" in each file!
Optimizing john single mode
Follow up on single mode
Kore Logic Rules, Modified by Solar Designer himself (The Creator of JtR)
Theses are "rules" and not a "John.conf" with rules
Update, Sept '12
As soon as the article was done, The JtR Team made few improvements :)
- A GUI JtR was released for Linux
- A Newer pre-compiled Version of JtR Jumbo 1.7.9 was released too
Nevertheless, there is always a delay between the most recent JtR version and the JtR, Hence compile it yourself.
ALLA
(Another Linkedin Lame Analysis)
Fun Fact about the Linkedin dump
First Character usage, Most used first
ae1iInrosl0t2mdc3uh9kb748g5p6yjfvwzxASMLB!RCTEq@NDPGHKOJF$.#W*UV_Y-Z&XQ+%/?,=);(^':~["^<]>|{}
Hypothesis:
Even though the keyspace remains the same than an alphanumericaly sorted character set, a sorted by usage Charset should be slightly faster as more hash -should- be cracked earlier.
The analysis of the 1st to 6th characters distribution indicates almost a constant pattern.
Does anyone has a documented opinion? Anyone?
Distribution of first to sixth Ch.
Character usage, Reversed
}{|>]<^"[~:'^(;)=,?/%+QX&Z-Y_VU*W#.$FJOKHGPDN@qETCR!BLMSAxzwvfjy6p5g847bk9hu3cdm2t0lsorniI1ea
Most "unlikely" password
}{|>]<^"[~:'^(;)=,?/%+QX&
The ones that are hard to find:
(fund 123) קרן 123
привет (Hello)
سعمفشى (Sultan)
قخنشغش (Rokaya)
Comments:
Please, if you leave a comment, let us know:
1) Your OS version
2) The JtR Version
3) Your machine type
Also, it's cd /john/src and not /john/src
cd as Change Directory
Hypothesis:
Even though the keyspace remains the same than an alphanumericaly sorted character set, a sorted by usage Charset should be slightly faster as more hash -should- be cracked earlier.
The analysis of the 1st to 6th characters distribution indicates almost a constant pattern.
Does anyone has a documented opinion? Anyone?
Distribution of first to sixth Ch.
 |
| Click To Enlarge |
Character usage, Reversed
}{|>]<^"[~:'^(;)=,?/%+QX&Z-Y_VU*W#.$FJOKHGPDN@qETCR!BLMSAxzwvfjy6p5g847bk9hu3cdm2t0lsorniI1ea
Most "unlikely" password
}{|>]<^"[~:'^(;)=,?/%+QX&
The ones that are hard to find:
(fund 123) קרן 123
привет (Hello)
سعمفشى (Sultan)
قخنشغش (Rokaya)
Comments:
Please, if you leave a comment, let us know:
1) Your OS version
2) The JtR Version
3) Your machine type
Also, it's cd /john/src and not /john/src
cd as Change Directory
