Mr Aso was asked whether poverty was making people marry later, contributing to Japan's low birth rate.
His response was: ''Young people had better not get married with little money."
He added it was difficult for him to see how someone without pay could be seen as an object of respect.
It was an unfortunate remark because a lack of job security, along with the large number of people on short-term job contracts is of prime concern to many in this election.
The rise in use of short-term contracts is due in part to labour reforms introduced by Mr Aso's Liberal Democratic Party.