Chapter 3: The Headphone
Once you have yourself a good quality source and some high quality music, you can start to look into headphone upgrade.
First, set a budget. As audiophiles' headphones go, they can be very expensive. You want to make sure you are getting a headphone that well matches your source. You don't want something too expensive but you won't want to cut your budget too low either. For the current earphones' (earbud and IEM) market, $50 to $100 will give you a good start, $100 to $200 will give you a very good selection. Anything beyond $200 is targeted at audiophiles (= mad people that will do anything for sound quality). If you are in a tight budget, there are still some good option at the sub$50 market, but you will have to be extra hardworking to find them. Most of them are hidden pearl that you can only find in forum discussion, not big review site.
Second, make sure you know where/how the headphone will be used. As portable rig goes, you don't want earbud when you are in library, nor will you want IEM when you need to be in constant communication. Find out whether you can stand earbud or IEM before you purchased. There is no point to buy a headphone when you can't even used it more than 5 minutes.
Third, look for the best sound quality headphone that fit your taste. Reviews often only tell what the reviewer believes to have good sound quality. What most don't tell is that each reviewer has a unique taste of music, and most will not match yours. You want to be sure that you are getting something that match your taste, even when it is not the best sounding you can afford (as you might not enjoy it).
Chapter 4: The Headphone Amp
I have written an article ('To amp, or not to amp') a while back to put some points into the craziness in chasing amp, do read it. People often read too much into how a little amping can put them in the quick path to audio heaven, well that is simply not true. Amp is really for more advanced user who know what they are looking for to complete their setup. If you can barely understand the reason for amping, the meaning of LOD, line-level signal, or whether your current setup needs amping or not, you should NOT even consider buying an amp. The point is, there are easier way to get better SQ then getting an amp. Amping is for user who understand the limitation of their setup and willing to go the extra miles to squeeze out the last few % in their system.
Headphone amp is just an element for fine tuning a portable rig, not a magic cure for sound quality.
Chapter 5: The Other Factors
I guess the first 'other' factors that comes into mind is cable. If amping is a relatively small element for fine tuning, cable will be the one you should ignore most of the time. I am not saying cable has no effect on sound, but at the level of portable rig, a few inches of cable, may it be silver or gold, really is nothing compare to the amount of copper in the wire and PCB circuit. It will be like placing racing-quality spark plug into a 30yrs-old car and hope it can compete in WRC racing, that is just too much of a wishful thinking.
One of another factor will be the eartips, or perhaps how to achieve an optimum seal. Different eartips (with different materials) can change the sound just enough to make a difference. So the basic idea is, don't stick to just the stock eartips, try as many aftermarket eartips as you can find.
Conclusion
As my closing comment, I want to point out that a portable setup is the rig you want to carry around. Thus it must be a solid solution that is practical to use. You will be foolish to assume the best sounding system will be the most practical - it is often not. Getting the best out of what you can and willing to carry is what tuning / upgrade is all about. Identifying the weakest link and finding a way to solve the most important issue should always be your first priority when it comes to determine the path of upgrading. Don't just buy into the belief that you can get a magic cure that will fix everything, cause there is none of such thing!