Passive Buffs???

For the passive buffs granted due to hunter interaction, do I need to keep the 3 star version of each character to get the buff or does owning once grant a permanent buff????