NVidia Nemotron 70B 模型深度评测

NVidia 推出的新开源模型 Nemotron 70B 最近成为社交媒体上的热点，各种"这太疯狂了""这太大了"的帖子层出不穷，声称其性能超越 GPT-4o。

我对开源模型的认识

古怪特性

第三方（如 Nvidia）基于原始创造者（如 Meta）发布的模型所做的微调和衍生版本，往往表现出各种人工痕迹和奇怪的行为。

整体性能不佳

它们往往不如闭源对标产品，我也不会把它们作为日常驱动（倾向于使用 OpenAI 和 Anthropic 的产品）。

我试着跑了 Nemotron 70B，我的 RTX4090 设置可以在 50% GPU 卸载的情况下运行 4-bit 量化版本，推理速度是 2.7 tokens/s（顺便说一下，最近有研究表明量化对模型性能的影响很小）。

以下是我在 Nemotron 70B 上观察到的一个奇怪现象的例子。这是一个创意和多语言任务——用俄语写一个短的童话故事：

Задача для Незнайки

Но reason invitirovaniya (причина приглашения) оказалась серьезной. Марсиане столкнулись с загадкой, которая brought их в тупик: их главный источник воды - Ледяная Пещера Олимпа - начала необъяснимо таять с невероятной скоростью. Без воды Марс оказался бы на грани катастрофы.

Незнайка, Known за своё проницательное мышление, принял вызов. Он собрал команду местных экспертов и вместе они отправились в Ледяную Пещеру.

注意一下模型是如何把英文单词（reason）注入到文本中替代相关的俄语单词的。它甚至还凭空创造了新单词（invitirovaniya）。

让我想起了"Смотря какой fabric смотря сколько details video"这样的现象 :)

我在中文模型上也注意到过类似的异常（比如 Qwen？）—— 用英文和它对话时，它可能会在回复中穿插汉字。

我用相同的 prompt 也测试过 Gemma 9B 和 27B，以及 Llama 3.1 8B 和 70B，都没有出现这种人工痕迹。

指令遵循能力差

最近我创建了一个 LLM 棋局项目，用来研究 LLM 在下棋时的表现。LLM 的基本前提之一是在评估棋盘和做出棋步时能够遵循最简单的指令。

下面是一个机器人（有硬编码 prompt 的有限状态机，又名 Proxy）和 GPT-4o Mini（Player_Black）讨论下一步棋步的对话样本：

Proxy (to Player_Black):

You are a professional chess player and you play as black. Now is your turn to make a move. Before making a move you can pick one of 3 actions:
    - 'get_current_board' to get the schema and current status of the board
    - 'get_legal_moves' to get a UCI formatted list of available moves
    - 'make_move <UCI formatted move>' when you are ready to complete your turn (e.g., 'make_move e2e4')
Respond with the action.

--------------------------------------------------------------------------------
Player_Black (to Proxy):

get_current_board

--------------------------------------------------------------------------------
Proxy (to Player_Black):

⭘ ♜ ♝ ♛ ♚ ♜ ⭘ ⭘
♟ ♟ ♟ ♟ ♟ ♟ ♝ ♟
⭘ ⭘ ♞ ⭘ ⭘ ♞ ⭘ ⭘
⭘ ⭘ ⭘ ⭘ ⭘ ⭘ ⭘ ♕
♙ ♙ ⭘ ⭘ ♙ ⭘ ⭘ ⭘
⭘ ⭘ ⭘ ♙ ♔ ⭘ ⭘ ⭘
⭘ ⭘ ♙ ⭘ ⭘ ♙ ♙ ♙
♖ ♘ ♗ ⭘ ⭘ ♗ ⭘ ♖

--------------------------------------------------------------------------------
Player_Black (to Proxy):

get_legal_moves

--------------------------------------------------------------------------------
Proxy (to Player_Black):

f8h8,f8g8,b8a8,g7h8,g7h6,f6g8,f6h5,f6d5,f6g4,f6e4,c6e5,c6a5,c6d4,c6b4,h7h6,e7e6,d7d6,b7b6,a7a6,e7e5,d7d5,b7b5,a7a5

--------------------------------------------------------------------------------
Player_Black (to Proxy):

make_move f8g8

--------------------------------------------------------------------------------
Proxy (to Player_Black):

Move made, switching player

而这就是我们在 Nemotron 70B 上得到的结果。它被自己的冗长所淹没，基本上无法完成哪怕一步棋。

Proxy (to Player_Black):

You are a professional chess player and you play as black. Now is your turn to make a move. Before making a move you can pick one of the following actions:
- 'get_current_board' to get the schema and current status of the board
- 'get_legal_moves' to get a UCI formatted list of available moves
- 'make_move <UCI formatted move>' when you are ready to complete your turn (e.g., 'make_move e2e4')

--------------------------------------------------------------------------------
Player_Black (to Proxy):

Sounds like a thrilling game!

Before making my move as Black, I'd like to gather all the necessary information to make an informed decision. Here's my chosen action for now:

**1. `get_current_board`**

Please provide me with the schema and current status of the board, including:

* The arrangement of pieces on the board (e.g., which pieces are where, for both White and Black)
* Any relevant board characteristics (e.g., castling rights, en passant availability, if any)

Once I have this information, my next step will likely be to request a list of legal moves to further analyze the position before deciding on my response as Black.

**Expected Response Format (for ease of understanding):**

- **Board Schema:** (8x8 grid showing piece positions, using standard algebraic notation, e.g., `r` for black rook, `R` for white rook, etc.)
- **Special Conditions (if any):**
  - Castling Rights (Kingside/Queenside for White/Black)
  - En Passant Target Square (if applicable)

Please share the current board state. I'm eager to delve into the game!

--------------------------------------------------------------------------------
Proxy (to Player_Black):

♜ ♞ ♝ ♛ ⭘ ♝ ♜ ⭘
♟ ♟ ♟ ♟ ♚ ♟ ♟ ♟
⭘ ⭘ ⭘ ⭘ ⭘ ♞ ⭘ ⭘
⭘ ⭘ ⭘ ⭘ ⭘ ⭘ ⭘ ⭘
♙ ⭘ ⭘ ⭘ ♙ ♟ ⭘ ⭘
⭘ ⭘ ⭘ ⭘ ⭘ ⭘ ♙ ⭘
♗ ♙ ♙ ♙ ⭘ ⭘ ⭘ ♙
♖ ♘ ♗ ♕ ♔ ⭘ ♘ ♖

--------------------------------------------------------------------------------
Player_Black (to Proxy):

Thank you for providing the current state of the board. To ensure I understand it correctly, I'll briefly outline the positions as per the notation provided (using standard algebraic notation for clarity):

**Board State Outline:**

* **Black (me):**
        + Rook: a8
        + Knight: b8
        + Bishop: c8, f8
        + Queen: d8
        + King: e7
        + Pawns: a7, b7, c7, d7, f7, g7, h7
* **White (opponent):**
        + Rook: a1, h1
        + Knight: b1, g1
        + Bishop: c1
        + Queen: d1
        + King: e1
        + Pawns: a2, b2, c2, d2, f5 (behind Black's pawn), g2, h2

**Observations and Next Steps:**

* The game seems to be in its early stages, with both sides having developed some pieces.
* Notably, White has an advanced pawn on f5, which might influence the center control and piece development strategies.
* Before deciding on a move, I'd like to see all available legal moves for Black to assess the best response to White's positional choices.

Given this, my next action is:

**2. `get_legal_moves`**

Please provide a list of legal moves for Black in **UCI (Universal Chess Interface) format**, which typically looks something like this: `e2e4`, `Nb1c3`, `g7g5`, etc. This will help me evaluate all possible responses and select the most advantageous move as my next step.

--------------------------------------------------------------------------------
Proxy (to Player_Black):

g8h8,d8e8,b8c6,b8a6,e7e8,e7d6,f6e8,f6h5,f6d5,f6g4,f6e4,f4g3,h7h6,g7g6,d7d6,c7c6,b7b6,a7a6,f4f3,h7h5,g7g5,d7d5,c7c5,b7b5,a7a5

--------------------------------------------------------------------------------
Player_Black (to Proxy):

Thank you for providing the list of legal moves for Black in UCI format. I'll analyze these options considering the current board position and strategic implications. Here's a breakdown of the moves, grouped by piece type for clarity:

**King (♚) Moves:**
* `e7e8`
* `e7d6` (somewhat unusual, potentially weakening king safety)

**Queen (♛) Moves:**
* `d8e8` (exchanging places with the King, not changing overall positi

Nemotron 70B 无法完成一整局棋，做了 14 步后最终因超过单个棋步对话长度限制（10 轮）而崩溃。

作为参考，GPT-4o Mini 平均每局能走 150 步，在与随机玩家对战时有 5 局和棋（共 10 局）。

如果我们仔细看一下模型卡上展示的基准测试，会看到 Arena Hard、AlpacaEval 和 MT-Bench —— 这些都是通过 LMSys Chatbot Arena 获取的人类偏好的 LLM 作为判官的代理指标。这是当人们 prompt 一个模型，从两个模型各得到一个完成结果，然后选择他们更喜欢的那个。

看起来人们很容易被 BS（行话）所欺骗。冗长的、友好的、充满自信的回复往往更受欢迎，而人们并不关心事实核查。毕竟，这是大多数人期望的便利性 —— 假设回复是正确的会很方便。

一项最近的研究《Language Models Learn to Mislead Humans via RLHF》讨论了用于让模型与人类偏好"对齐"的流行训练技术 RLHF 如何可能对性能产生负面影响："为了获得更高的奖励，LM 可能会更善于让人类相信他们是对的，即使他们其实是错的"。也就是说，说人们喜欢听的东西和说实话不是一回事……或者严格按指令执行任务（如上面演示的那样）。

以下是 MMLU Pro 基准测试的结果，显示 Nemotron 70B 的性能略低于 Llama 3.1 70B，远低于 GPT-4o 等闭源模型。

虽然文章开头提到量化对模型性能的影响很小，但今年早些时候有一篇 Reddit 帖子讨论了 Llama 3 系列模型确实比之前的 Llama 2 模型受到更多量化的影响。

如果要采取进一步行动，你可以考虑屏蔽这个用户和/或举报滥用行为。