If you want to speak English fluently and easily without practicing with you friends, I think you're in the wrong way!
There is a difference and a relationship between "speak" and "talk"
to speak means "words come out of your mouth", but "talk" has the idea of speaking with others.
So, when you speak, you're just getting words coming out of your mouth, but when you talk you're making a conversation.
If you talk with your friends in English, you're making an English conversation.
The relationship between these two words is that, talking in English helps you to speak English fluently.
The best solution is to practice talking English with a native English speaker, try to find someone who can help you, I mean, someone who slows done when you ask him, someone who repeats when you don't understand...
If you find a native English speaker who can help you, you can make him as an English tutor. And I don't ask you to look for something else to learn English because you already have the best solution.
While finding an English tutor is not available for everybody, finding some friends with the same level as you can also help you to improve your spoken English.
For example, if you have some English friends, each one of you knows some vocabulary others don't know and then everybody will learn more vocabulary!
Whether you are a beginner or an advanced student, talking in English is very important for you, because it helps you either to learn English or to improve your spoken English.