The windows are attached to the frame through the jambs or by a flange outside under the casing. You do not need to pull off any drywall to insulate, just carefully remove the casing molding around the window. This will expose the small gap between the window and the frame. You can use small pieces of insulation to wedge into the gap, or use LOW EXPANSION foam and inject small amounts. Do not use high expansion foam or use too much as this can create lots of pressure on the window jambs causing damage or making opening and closing difficut.