Let x be the size of the jug.  After the man drew off his first jugful of wine, the keg contained 10-x gallons of wine. 

When he filled up the keg with water, the proportion of wine was reduced to (10-x)/10. 
The man's second jugful contained x(10-x)/10 gallons of wine, so the keg's wine content was reduced to
10-x-x(10-x)/10 gallons of wine.

Since the keg now contains equal quantities of wine and water,
10-x-x(10-x)/10=5.  Now just solve for x.

10-x-x(10-x)/10=5

100-10x-(10x-x²)=50

x²-20x+50=0

x=10±sqrt(50)

Of these two values , 10+sqrt(50) is greater than 10, so it can't be the capacity of the jug in this story. So x=10-sqrt(50), which is about 2.928932188 gallons.

Let's check this answer.

After the first jug of wine is taken, the keg contains
10-x = 10-10+sqrt(50) = sqrt(50) gallons of wine. The proportion of wine in the keg is
sqrt(50)/10.  After the next jugful is taken, the keg contains (10-10+sqrt(50))(sqrt(50))/10=5 gallons of wine.

Do you see what's going on here?  Every time the man draws off a jugfull, and then fills up the keg with water, he dilutes the wine by a factor of sqrt(50)/10 = 1/sqrt(2).

So after two trips the wine in the keg is at 1/sqrt(2)²=1/2 strength. 

After three trips, the wine in the keg would be 1/sqrt(2)³ of its original strength, and after four trips the wine would be at
1/sqrt(2)⁴=1/4 strength.