The weather in Southern California in early November is generally mild and dry. Average high temperatures tend to be in the low to mid 70's F (21-24 C). Since those are only average temperatures, you may experience days that are considerably warmer or cooler.
Even though it's farther to the north, temperatures in Santa Barbara
will be similar to temperatures along the Los Angeles coast
Fall is actually a good time to visit the coast. There's less fog and fewer crowds. The warmest beaches in the Los Angeles area are in Long Beach. You'll also find that in California the interior areas are warmer than along the coast, and a few miles can make a big difference, so Hollywood will be warmer than the beach areas.
Whether you'll want to go swimming is another matter. The water temperatures in Southern California are never particularly warm. They peak in August at about 68 degrees F (20 C), by November they're around 61-66 degrees F (16-19 C). So it all depends on what you consider warm enough.
Here's hoping you enjoy some lovely warm weather during your visit.