Expert says uproar against the flag is part of a feeling by nationalists and some mainstream Israelis that they are ‘losing the state’ to Palestinian nationalism from within Israel’s borders