Under a large dimensional approximate factor model for asset returns, I use high-frequency data for the S&P 500 firms to estimate the latent continuous and jump factors. I estimate four very persistent continuous systematic factors for 2007 to 2012 and three from 2003 to 2006. These four continuous factors can be approximated very well by a market, an oil, a finance and an electricity portfolio. The value, size and momentum factors play no significant role in explaining these factors. For the time period 2003 to 2006 the finance factor seems to disappear. There exists only one persistent jump factor, namely a market jump factor. Using implied volatilities from option price data, I analyze the systematic factor structure of the volatilities. There is only one persistent market volatility factor, while during the financial crisis an additional temporary banking volatility factor appears. Based on the estimated factors, I can decompose the leverage effect, i.e. the correlation of the asset return with its volatility, into a systematic and an idiosyncratic component. The negative leverage effect is mainly driven by the systematic component, while it can be non-existent for idiosyncratic risk.