NEW YORK (CNN) -- Trucks rolled onto the World Trade Center site Thursday morning as workers gathered to start long-delayed construction at 9/11's Ground Zero. The intention is to build the tallest building in North America to fill the hole in the New York skyline left by the 2001 terrorist attacks. "It is going to be a symbol of our freedom and independence," Gov. George Pataki said at a news conference on the ramp into the site, according to The Associated Press. The move follows an agreement on Wednesday between the owner of the 16-acre site and the real estate developer who leased it six weeks before the towers were destroyed. The Port Authority of New York and New Jersey, the transportation agency that built the 110-story twin towers in the early 1970s and ran them until July 2001, approved the deal with developer Larry Silverstein in a board meeting Wednesday afternoon. The deal settles financing for the "Freedom Tower," the planned 1,776-foot building to be erected on the northern end of the site. "We agreed to all of the port's economic terms in its proposal to restructure our 99-year lease," Silverstein said. "This is a fair deal." Silverstein will cede control of the Freedom Tower to the Port Authority. He will collect a 1 percent developer's fee on the estimated $2 billion project, but the Port Authority will manage the building when it is completed. It is slated to be ready for occupancy in 2010. Freedom Tower construction will move forward with installment of foundation footings. The building is to rise 1,360 feet, the height of the original twin towers. A spire will raise it to the height symbolic for the year of the American Revolution. A memorial to the 2,973 September 11 victims and six people killed in a 1993 truck bombing will be located on the southern half of the site. It is expected to be completed in 2009, financed through private donations. Pataki addressed concerns about the tower becoming a terrorist target while at the site with New Jersey Gov. Jon Corzine, Mayor Michael Bloomberg and Silverstein, the AP reported. "We are not going to just build low in the face of a war against terror," Pataki said. Never stopped paying rent Wednesday's deal marks the third time since the September 11, 2001, terrorist attacks destroyed 12 million square feet of office space that Silverstein's firm has amended his $120 million-per-year lease. He has never stopped paying rent, though he has collected no revenue on the site since the attacks. Silverstein will retain control of three buildings planned along the site's eastern edge, Church Street, which will house much of the planned complex's street-level retail space. He will receive a 2 percent developer's fee for those buildings, due to be completed in 2012. One is a 950-foot tower to be designed by renowned British architect Norman Foster. Silverstein will also cede control to the Port Authority of a plot of land across the street from the site's southern end, where a fifth building would go, according to the plan. The damaged Deutsche Bank building that currently stands there is being dismantled, possibly to be replaced by a residential tower. Overall, the five planned buildings would create 10 million square feet -- approximately 6.2 million square feet for the three Church Street buildings, 2.6 million square feet in the Freedom Tower and as much as 1.2 million square feet in the fifth building. 'Finger-pointing must stop' The Port Authority will significantly reduce Silverstein's rent, though by how much was not disclosed. Silverstein will allocate to the Port Authority 38 percent of the $3.34 billion in tax-exempt Liberty Bonds he is due to receive from the state and the city and a portion of approximately $3 billion in insurance proceeds he is due to collect. "We have made real concessions," Silverstein said. "This is about moving the rebuilding forward as quickly as possible. All the finger-pointing must stop, and we must all work together to achieve our vital mission -- to fully revitalize and renew New York's historic downtown region." Silverstein, who lost four employees in the September 11 attacks, said rebuilding is "an intensely personal endeavor for me." He has already built, a block north of the site, an office tower that replaced World Trade Center 7, which he developed in the 1980s before taking over the whole site. A quarter of the building, 10 of the 42 habitable floors, are now leased. Next month, Silverstein Properties will be the first tenant to move in. In recent weeks government officials had questioned whether Silverstein's financial position was strong enough to continue as the site developer, leading to the protracted negotiations. Gov. George Pataki predicted that the rebuilt trade center will "anchor the financial capital of the world and make our nation proud." If built as planned, Freedom Tower will be taller than the Sears Tower in Chicago, which stands 1,450 feet.