“He said to Moses, “Come up to Yahweh.” If not read carefully in context this verse can be confusing because it sounds like God is calling Moses to go up Mount Sinai. But Moses is already up (this is his fourth time up with Yahweh, see Exodus 20:21). God is telling Moses to come up again, but only part way, and bring Aaron, Nadab, and Abihu, and seventy of the elders of Israel along too. But those men were down at the foot of the mountain, so Moses has to go down again to get those men and bring them part way up with them. So Moses went back down to get them (Exod. 24:3) and brought them partway up Mount Sinai as God asked him to do (Exod. 24:9).
[For more on Moses’ seven trips up Mount Sinai, see commentary on Exod. 19:3].