If someone was remotely connected to your computer they would be using your IP address to connect back out to the internet. It is also possible to spoof IP addresses but unlikely. If you don't have any security (Anti Virus, Firewall etc.) on your computer a remote keylogger could be placed on it and anything you type in the computer would then be emailed to fraudsters. Some people store all of their PIN's and passwords in a file in the My Documents folder which makes it very easy for anyone that way inclined. Did the credit card company tell you that someone used your IP address to make purchases your credit card? Were they able to give you the IP address used? They should be able to refund you for anything that was billed to the card but there are a few forms to fill in normally. Its just hassle.
Did you get any odd emails recently looking for you to fill in details? credit card, passwords. It happened to a friend of mine who got an email pretending to be her Bank and she clicked on the link and put her details in without thinking, called me afterwards as it seemed a bit 'odd' the bank was asking her for her passwords. I told her to call the bank and credit card company asap which she did and cancelled all her accounts until the details could be changed. She didn't get ripped off but it was a close one.