Skip to content

Commit ed73cac

Browse files
committed
Fix terminology: replace 'ballpark' to '์ถ”์ •(ballpark)'
1 parent a25344d commit ed73cac

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

โ€Žintermediate_source/TP_tutorial.rstโ€Ž

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ PyTorch์˜ Fully Sharded Data Parallel(FSDP)๋Š” ์ด๋ฏธ ๋ชจ๋ธ ํ•™์Šต์„ ํŠน์ •
5151
Tensor Parallel๊ณผ FSDP์˜ ๊ฒฐํ•ฉ์ด ํ•„์š”ํ•œ, ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์ถ”๊ฐ€์ ์ธ ๊ณผ์ œ๊ฐ€ ๋‹ค์ˆ˜ ๋ฐœ์ƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
5252

5353
1. ๊ทธ ์„ธ๊ณ„์˜ ํฌ๊ธฐ (GPU ์ˆ˜)๊ฐ€ ์ง€๋‚˜์น˜๊ฒŒ ์ปค์ง์— ๋”ฐ๋ผ (128/256 GPU ์ดˆ๊ณผ), FSDP ์ง‘ํ•ฉ(์˜ˆ๋ฅผ ๋“ค์–ด, ``allgather`` )์€ ring latency๋กœ ์ง€๋ฐฐ๋˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค. TP/SP๋ฅผ FSDP ์œ„์— ๊ตฌํ˜„ํ•˜์—ฌ, FSDP๋ฅผ ํ˜ธ์ŠคํŠธ ๊ฐ„์—๋งŒ ์ ์šฉํ•˜์—ฌ FSDP ์„ธ๊ณ„ ํฌ๊ธฐ๋ฅผ 8๊ฐœ๋กœ ์ค„์ผ ์ˆ˜ ์žˆ์œผ๋ฉฐ, ๊ทธ์— ๋”ฐ๋ผ ์ง€์—ฐ ๋น„์šฉ๋„ ๋™์ผํ•˜๊ฒŒ ์ค„์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
54-
2. ์ˆ˜๋ ด ๋ฐ GPU ๋ฉ”๋ชจ๋ฆฌ ์ œํ•œ์œผ๋กœ ์ธํ•ด ๊ธ€๋กœ๋ฒŒ ๋ฐฐ์น˜ ํฌ๊ธฐ๋ฅผ GPU ์ˆ˜๋ณด๋‹ค ๋†’๊ฒŒ ์„ค์ •ํ•  ์ˆ˜ ์—†๋Š” ๋ฐ์ดํ„ฐ ๋ณ‘๋ ฌ ์ฒ˜๋ฆฌ์˜ ํ•œ๊ณ„๋ฅผ ๋‹ฌ์„ฑํ•˜๋ ค๋ฉด, Tensor/Sequence Parallel์ด ๊ธ€๋กœ๋ฒŒ ๋ฐฐ์น˜ ํฌ๊ธฐ๋ฅผ "ballpark"ํ•˜๊ณ , ๋” ๋งŽ์€ GPU๋กœ ํ™•์žฅํ•˜๋Š” ์œ ์ผํ•œ ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค.
54+
2. ์ˆ˜๋ ด ๋ฐ GPU ๋ฉ”๋ชจ๋ฆฌ ์ œํ•œ์œผ๋กœ ์ธํ•ด ๊ธ€๋กœ๋ฒŒ ๋ฐฐ์น˜ ํฌ๊ธฐ๋ฅผ GPU ์ˆ˜๋ณด๋‹ค ๋†’๊ฒŒ ์„ค์ •ํ•  ์ˆ˜ ์—†๋Š” ๋ฐ์ดํ„ฐ ๋ณ‘๋ ฌ ์ฒ˜๋ฆฌ์˜ ํ•œ๊ณ„๋ฅผ ๋‹ฌ์„ฑํ•˜๋ ค๋ฉด, Tensor/Sequence Parallel์ด ๊ธ€๋กœ๋ฒŒ ๋ฐฐ์น˜ ํฌ๊ธฐ๋ฅผ "์ถ”์ •(ballpark)"ํ•˜๊ณ , ๋” ๋งŽ์€ GPU๋กœ ํ™•์žฅํ•˜๋Š” ์œ ์ผํ•œ ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค.
5555
3. ํŠน์ • ์œ ํ˜•์˜ ๋ชจ๋ธ์—์„œ๋Š” ๋กœ์ปฌ ๋ฐฐ์น˜ ํฌ๊ธฐ๊ฐ€ ์ž‘์•„์ง€๋ฉด, TP/SP๊ฐ€ ๋ถ€๋™ ์†Œ์ˆ˜์  ์—ฐ์‚ฐ(FLOPS)์— ๋” ์ตœ์ ํ™”๋œ ํ–‰๋ ฌ ๊ณฑ ํ˜•ํƒœ๋ฅผ ์ƒ์„ฑํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
5656

5757
๊ทธ๋ž˜์„œ, ์‚ฌ์ „ ํ•™์Šต์„ ํ•  ๋•Œ, ์ด๋Ÿฌํ•œ ํ•œ๊ณ„๋ฅผ ๋‹ฌ์„ฑํ•˜๋Š” ๊ฐ€์žฅ ์‰ฌ์šด ๋ฐฉ๋ฒ•์€ ๋ฌด์—‡์ผ๊นŒ์š”? ์ง€๊ธˆ๋ถ€ํ„ฐ, ์ˆ˜์‹ญ์–ต ํ˜น์€ ์ˆ˜์กฐ ๊ฐœ์˜ ํ† ํฐ์„ ๊ฐ€์ง„ ๋Œ€๊ทœ๋ชจ ์–ธ์–ด ๋ชจ๋ธ (LLM)์„ ์‚ฌ์ „ ํ•™์Šตํ•˜๋Š” ๊ฒƒ์€ ์ˆ˜์ฒœ์žฅ์˜ GPU๋ฅผ ์‚ฌ์šฉํ•  ๋•Œ์กฐ์ฐจ, ์ˆ˜๊ฐœ์›”์ด ์†Œ์š”๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

0 commit comments

Comments
ย (0)