The piece of skin you describe is often called a banjo cord; many men retain it. Sometimes it snaps and sometimes doesn't. I'm no doctor, but I believe guys with phimosis often have it.
Addressing your phimosis directly, I would say /don't/ worry. I'm 31 with phimosis and have never had any physical issues. However, if you are worried I would recommend getting referred to an andrologist who can give you advice on how to address the issue. There are two main options (three if you include doing nothing)
1) Steroidal cream. This can loosen your foreskin allowing it to retract and fully function. But this usually works only on mild cases of phimosis.
2) Circumcision. Caution, however. This site is full of stories about circumcisions having gone wrong. Those are the minority but in a surgery; there is always the chance of things going wrong. It COULD end up with the solution causing more problems than that which is was supposed to correct. This is why advice from a trained medical professional is so important.
If it's not causing you a problem and everything seems to work normally then I wouldn't worry about it. If you're worried, check with a doctor. You can go alone, and it's completely confidential.
Content on HealthUnlocked does not replace the relationship between you and doctors or other healthcare professionals nor the advice you receive from them.
Never delay seeking advice or dialling emergency services because of something that you have read on HealthUnlocked.