Five or six people is ideal. Think of a focus for your conversation. In my conversation group, we all read the same book in Spanish and talk about it in Spanish. Your group could discuss movies you’ve seen, which would probably be movies in English but your summary of the plot and comments would be in Spanish. You could focus on travel, taking a different country each time. Your group could meet at different restaurants and talk about food.
You do need a focus for the group to prevent it deteriorating into aimless conversation. Even though it might be in Spanish, aimless conversation will not improve your vocabulary or your ability to express a difficult thought.