''Now give me my $75 or I'll break your legs,'' said Richard Sabatino, taking payment for some flounder at the 108 South Street Fish Company just before dawn yesterday morning. He looked over at a reporter and laughed uproariously.

In another part of the Fulton Fish Market, which Federal prosecutors on Thursday asserted was dominated by the Mafia, Robert DiGregorio took his paycheck and said, ''You see, he's paying me off.'' He clapped his hand on the shoulder of a Korean customer, saying, ''Here's the capo di capi!'' Metal Hooks, Chilled Hands

Federal prosecutors, led by the United States Attorney in Manhattan, Rudolph W. Giuliani, assert that the market and the union are ''captive organizations of the Genovese crime family'' and have been so since the 1930's. The officials filed a civil lawsuit Thursday seeking to seize control of the market on the ground that the Mafia dominates it, driving up fish prices.

At the market yesterday morning, as the fishmongers wrestled with heavy boxes and argued over the quality of tuna, swordfish and skate, they responded to questions about Mr. Giuliani's assertions with a wary mixture of humor and hostility. Some tried hard to explain the tribal ethos of the men here; others looked over their shoulders and moved quickly away.

''This place has been going since 1822 and it's been going pretty well,'' said Mr. DiGregorio, a member of the market's main union, the United Seafood Workers Local 359 of the United Food and Commercial Workers, A.F.L.-C.I.O. It is these men, wielding long metal hooks in scarred, ice-chilled hands, who handle the 175 million pounds or so of fish that move through the market every year. Romaniticism of the Market

''I've been here for 15 years,'' said Mr. DiGregorio, ''and I don't know of anyone paying off anyone. That man Giuliani has political ambitions, and ever since that Rouse Company decided to open up this mall or whatever they call it, they've done nothing but break our chops.''

It was the Rouse Company, based in Maryland, that developed the South Street Seaport, a mix of trendy shops and restaurants abutting the Fulton Fish Market. Opened in 1983, the seaport complex was intended to bring more tourism to the area, taking advantage of the romanticism and history of the market, a favorite spot for generations of poets and lovers to watch the sun rise through the fretwork of the Brooklyn Bridge.

Mr. DiGregorio is not alone in his ambivalence about the seaport development, its evocations of a maritime past in odd contrast to the enduring sweat and smell of the market, which recreates itself each night and vanishes in the early morning, before Laura Ashley and the Nature Company open for trade. Suspicions About Giuliani

''The Seaport corporation holds the lease to the market,'' said one owner. ''I suppose it means we'll stay here, but it cramps us.''

Between quoting prices for glistening, headless halibut at M. Slavin & Sons Ltd., a man who identified himself only as Sallie had his own suspicions about Mr. Giuliani. ''I've got a whole wall of pictures of me with politicians, with John F. Kennedy and everybody,'' he said. ''They think it's lucky to come down here, they come down and win.''

Would Mr. Giuliani come here? ''No way,'' said Sallie, who has worked here for 35 years. ''He's got plans, but no way he'd show his face here.''

''Jesus, don't give no interviews on that stuff,'' a young man broke in to say. ''That's my son,'' said Sallie. ''He's the third generation here.'' Asked about the Federal allegations of racketeering and mob control, artificially inflating fish prices, Sallie said: ''I don't know anything about that. I've just got to watch my own,'' and then walked quickly away.

''Yeah, it's corrupt here, sure,'' said a restaurant owner, testing the eyes of red snapper for freshness. ''There's a monopoly. You want fresh fish, you buy and you pay, and you keep your mouth shut. What choice is there? It's been like this for a long time, and it isn't going to change.'' Asked for his name, the owner recoiled. ''What the matter?'' he said. ''You think I'm crazy?''