To be honest, I think you're fine.
- You rob a guy and issue compliance, he puts his hand up.
- He runs away, ignoring your original orders not to move, you shoot at him, no problem.
- Within 1 minute, you see the guy with an assault rifle moving to your position, I would say the RP is still engaged at this point.
The only issue here, is that could have been a completely random guy who was just looking at the area, but honestly, he should have said something but he didn't. (At this point, its sometimes useful to let the person know that you know he's there, tell him to stop moving and put his hands up, if you issue compliance early like that, he can't complain if you shoot him).
The final act that makes me sure you'll be okay...
- He fired first.
Although technically if you shoot someone who's committing RDM you shouldn't really shoot them back without saying something first, but I don't think I'll ever blame someone for reaction shooting someone who is trying to kill you.
In my opinion, you guys are fine, the guy who tried to kill you though, he nearly committed RDM, even if he was a friend of the guy being robbed, he has to say something, issue compliance, warn you to leave, etc.. at least "Hello" would have been better than nothing (although not much better).
That's my take on it, I can't guarantee the accuracy of my view of the incident as you can't clearly hear the sound from in-game, maybe he did issue compliance? Can't be 100% sure unfortunately.