The partnership between two of the world's biggest economies was agreed in July and ratified by both countries in recent weeks.
Spanish fruit and vegetable grower association FEPEX welcomed the news.
"The Economic Association Agreement between the EU and Japan includes the practical suppression of tariffs in the fruit and vegetable sector," it said.
"In a reduced group of fruit and vegetables, the dismantling of tariffs will take place over a period of seven years from when it comes into effect, with specific measures for some products including safeguards.
"However, it will still be necessary for both sides to negotiate phytosanitary protocols so that European producers can access this market."
Spain currently sends a minuscule amount of fruit and vegetables to Japan, which FEPEX attributed in large part to non-tariff barriers like phytosanitary issues.
"Through September of this year, Spanish fruit and vegetable exportation to Japan was of 1.277 metric tons (MT), equivalent to 4,8 million euro (US$ 5.4 million)," said the customs and tax department.
In November the European Parliament's Committee on International Trade (INTA