Fine tuning the nut height is one of the very last steps I take in a setup, and should be done with the action and neck relief set as the guitar will be played. This will deal with the locking nuts on tremolo equipped guitars. If you have a non locking glued in nut then adjusting it's height is better left to a professional that's already spent the $160 a full set of fret files will cost. [Although if your problem is the string slots are cut or worn too deep you can carefully build up the string slots with super glue, the alternative is to have a new nut installed]
Nut height is probably one of the most crucial aspects of how your guitar feels within the first 5 frets, but it also effects the feel of the whole fretboard to some degree. Quite simply the nut should be as low as it can go without causing string buzz on the first fret. For a quick judge fret each string at the first fret and look at the clearance it has over the second. Compare to the gap between the top of the first fret and each string with the string open. Near 100% of all new Ibanez guitars have nuts that are way too high. Most of the new Jems I receive have nut height at the first fret low E around 1mm, about .7mm too high!! I've been complaining about it to the company for years and the only thing that's happened is they've gotten higher. I'm guessing factory spec must be for them to be set "way too high", so in that case, they're all perfect! But if you want a much better playing guitar you will take the time to lower the nut to the correct height. The Jems and sporadically other models will be unique in that their high nuts have nothing to do with how many shims are under it, but the fact the nut rout was never cut near deep enough to begin with, but that's the topic of another much more tedious section.
The first thing you want to do is evaluate what you have now and decide what probably needs to be done. Do you see plenty of shims under the nuts? Great, lowering it will be much easier. No shims? Are you getting fretbuzz at the first fret? Either your nut is too low, your neck does not have enough relief [or is in backbow], or the neck does not have enough relief for where the nut is set. If you have the nut set nice and low with a decent amount of neck relief and you straighten the neck too much, the nut is now too low for the straighter neck and you may get buzz on the first fret. This is why you set the nut height with the relief and action adjusted to your preference. This section will deal with a simple height adjustment where only the addition, removal, or rearrangement of shims is needed to get the desired height.
Before starting you should know that the fretboard radius will not perfectly match the radius of the nut. The nut will probably be radiused a little flatter than the frets. If you have the low and high E's as low as they should be set you'll get 1st fret buzz on the D and G strings and/or A and B strings. Because of this the nut needs to be a little higher to clean up the middle strings and to compensate I will split the difference between both the high and low side of the nut. In other words instead of putting a big shim under the low side to raise it enough you want to put half as much under the low and high side to raise it evenly until the D/G are clean. You'll also find some that may have radiuses flatter than the nut causing the center to be high.
The very first step is with all strings in tune and fully stretched, pinch each string to the fretboard hard right where it leaves the nut. This will remove the strings tendency to rise a little as it curves off of the nut and that little bit is enough to make the difference between having buzz or not when done. It should also be noted that for rock guitars having some mild first fret buzz is acceptable as long as there's no more first fret buzz than you get fretting the second fret.
To adjust the nut height I usually find it easiest to remove all the shims under the nut and start from scratch. With the nut pads removed block the trem to it's highest angle to take as much tension off the strings as possible [this also allows you to quickly unblock it to perform checks on your height adjustment]. It is quite common [and perfectly normal] to find all the shims as half shims built up on each side, and these are quite easy to remove. Unscrew both Allen bolts that hold the nut on in the back a few turns [not necessarily all the way] from here there are two ways to lift the nut, push up on the nut using your wrench seated in the nut bolt from the back, or you can use a 3mm Allen wrench in an outer nut pad bolt hole as a lever to pull the nut up [if you're pulling the bass side shims put the wrench in the E./A pad bolt hole with the long part of the wrench also pointing to the bass side, pull up on the wrench using it as a lever to lift that side of the nut], if you do either technique while holding the guitar on it's side any half shims on that side will just fall out [visually inspect to verify they're all out]. To visualize, if you holding the guitar as if you would be playing it and raise the treble side of the nut all the shims on the treble side will fall out. Flip it over to do the bass side. If they're all gone great. More than likely you'll find a full shim or two left though. These are a little trickier to remove and before you do I suggest unblocking the trem to make sure they need to be removed or if you've already got it low enough. If it's close tighten the nut bolts [using common sense!! They only need to be tight enough to hold the nut in place, too tight and you'll crack the wood!!] because the nut will lower slightly when you tighten it, Recheck.
To remove a whole shim I'll start by backing off the string tree to free up a little more string tension. To get one out you'll either have to remove the nut mounting bots completely, or remove the string tree completely, and on some pesky "don't want to budge" shims you might have to do both. Most of the time you can just slide the nut toward the bass side then lighten the pressure on the nut and slide it back, leaving the shim stuck out enough that you can now just lift the bass side using the Allen wrench in the pad bolt lever trick to work it the rest of the way out [or grab it with some pliers if you get tired of playing with it]. I've got plenty of shims so I'll just jam a .5mm shim under the nut to force it out the other side enough to grab. Some will just not want to move [some will be virtually glued in] at all and those you can remove toward the headstock. Remove the string tree, lift one side of the nut, use something to "catch" the shim [I use the point of an Exacto knife] and work it out a little, do the other side, back and forth until it out enough to either get some needle nose pliers on or grab by some other means, then just pull it out. All JS1000/10th/90th/2000 models must have their full shims removed this way as they have an arch molded into them at the truss rod channel.
With all the shims removed re-install the string tree and unblock the trem to see where the nut height is. Play each sting to check for any choking or buzz at the first fret. If there is none, is it low enough for you? Or do you think it can go much lower [compare to the second fret gap with the strings fretted on the first] If so, move to the section on lowering the nut rout.
Hopefully it will be too low, buzzing everywhere, and now ready to build up to the correct height. This would be a good time to talk about nut shims. The shims come in 3 thicknesses, .1mm, .25mm, and .5mm. .5's are monsters and will only have a place if the original nut rout was extremely deep. .1's and .25's are your shims of choice for fine tuning the height, and most specifically, .1mm will be your best friend and offer the most precise fine tuning of the height. Sort through the shims that you've removed, a .1mm will bend if you blow on it hard enough, a .25 is fairly pliable with your fingers and you can bend it, just not like paper, a .5 is one thick tough suckah, you'll know it when you see it.
To build the nut back up is a matter of determining what it needs, and some of this will be trial and error. Is it choking on the treble side and not the bass? Shim the treble. Choking on the bass but not the treble, shim the bass. If all strings are choking start with putting the same size shim under both sides and test again. If you have a quick method for blocking the bridge [and I always use my spreader clamp to just clamp my bar to the body [my bar has an Ibanez Sure Grip foam wrap on it so it's essentially a padded bar. I started using it for this reason and fell in love with it, to the point I hate the feel of a normal bar any more]] you will find this very fast to shim, check, adjust, check, until it's right. Using the technique of a wrench as a lever in a pad bolt. I'll always start with a .25 shim and stick one under the low side and under the high side. These don't have to be fully inserted, just stuck under the side to give an idea of where it's at and what it needs. Once it stops choking/buzzing on one side, raise the other until it's clean. You can further tweak by then removing any thicker shims such as a .25 and replacing it with a .1 or two .1's until the nut is as low as it will go without choking or buzzing on the first fret. Remember the note earlier about the nut radius not matching the fret radius exactly. If the center is too low to clean up the center strings split the amount of additional height needed between the bass and treble sides. If the center is too high then you'll just have to shim the low and high side as low as possible and live with it [or have the fretboard/frets re-radiused]
When you are satisfied with the results take out the shims on both sides and analyze them. If you have a .25mm shim on each side use a whole shim instead of a half, the same with any additional shims until the only half shims I'm using are the ones that give the final tweak where using a full shim would raise the opposite side too high. If you don't have the full shims and are using halves [and there is nothing wrong with just using halves, you'll never hear a difference between using a half shim on each side as opposed to a full shim] get the halves together so they are like inserting a single shim, and fully install it under the nut until you can't feel the edge protruding past the nut, making sure they are seated flush against the L where the nut meets the fretboard. Repeat on the other side and tighten the nut snug. Test again to make sure that tightening the nut didn't lower it enough to cause any buzz and re-tweak if necessary. Some will be trial and error, some will be educated thinking, sometimes pure luck will get you there, but taking the time to get the nut height as low as possible will pay big dividends in how the guitar plays and feels.