Load the file as two separate tracks (video and audio) and in the timeline view use the scissor tool to cut either side of the portions you don't want. You can then either pull down the volume level bar or cut the offending sections without interfering with the video timeline. Or you could automate the slider of the sound mixer to pull down the volume of the track at the precise points needed. There may be other work arounds as well but they are the two main ones I use. For complex audio work I would prefer using the mixer but for simpler tasks, cutting and adjusting can be quicker.