Code

On Device Sampling#350

Code

Draft

quic-sanising wants to merge 31 commits intoquic:mainquic/efficient-transformers:mainfrom

quic-sanising:on-device-samplingquic-sanising/efficient-transformers:on-device-samplingCopy head branch name to clipboard

Commits

Commits on Apr 8, 2025

• Initial commit

  Show description for 718d763
  quic-sanising

  committed
  718d763

  Copy full SHA for 718d763

• Reformat code

  Show description for b8d099e
  quic-sanising

  committed
  b8d099e

  Copy full SHA for b8d099e

• Fix bug

  Show description for 544c0dd
  quic-sanising

  committed
  544c0dd

  Copy full SHA for 544c0dd

• Add Gumbel-Max trick based random sampling

  Show description for 0b4d0a9
  quic-sanising

  committed
  0b4d0a9

  Copy full SHA for 0b4d0a9

• Bring up to date

  Show description for 24efc93
  quic-sanising

  committed
  24efc93

  Copy full SHA for 24efc93

• Use Gumbel-Max Trick based Random Sampling as default

  Show description for 2af43c6
  quic-sanising

  committed
  2af43c6

  Copy full SHA for 2af43c6

• Clip k to max value

  Show description for 3eca771
  quic-sanising

  committed
  3eca771

  Copy full SHA for 3eca771

• Add docstring for sampling parameters

  Show description for b0e9162
  quic-sanising

  committed
  b0e9162

  Copy full SHA for b0e9162

• Fix bug

  Show description for 0486e42
  quic-sanising

  committed
  0486e42

  Copy full SHA for 0486e42

• Add support for continuous batching

  Show description for e7dda72
  quic-sanising

  committed
  e7dda72

  Copy full SHA for e7dda72

• Fix ONNX error for batch_size 1 treated as a Constant

  Show description for f94c657
  quic-sanising

  committed
  f94c657

  Copy full SHA for f94c657

• Undo docstring deletion

  Show description for fa026a4
  quic-sanising

  committed
  fa026a4

  Copy full SHA for fa026a4

• Remove device and unncessary reshapes

  Show description for eff2007
  quic-sanising

  committed
  eff2007

  Copy full SHA for eff2007

• Revert batch_size to 1

  Show description for ebfbaea
  quic-sanising

  committed
  ebfbaea

  Copy full SHA for ebfbaea

• Remove vocab_size from dynamic axes

  Show description for 83d33ac
  quic-sanising

  committed
  83d33ac

  Copy full SHA for 83d33ac

• Change condition

  Show description for fc3dc82
  quic-sanising

  committed
  fc3dc82

  Copy full SHA for fc3dc82

• Change size of each sampling parameter to (batch_size, 1)

  Show description for abbaf53
  quic-sanising

  committed
  abbaf53

  Copy full SHA for abbaf53

• Reformat code

  Show description for f5f5e2d
  quic-sanising

  committed
  f5f5e2d

  Copy full SHA for f5f5e2d

• Fix bug

  Show description for 05c0bf0
  quic-sanising

  committed
  05c0bf0

  Copy full SHA for 05c0bf0

• Allow chunked prompts during prefill

  Show description for 3b63ecb
  quic-sanising

  committed
  3b63ecb

  Copy full SHA for 3b63ecb

Commits on Apr 9, 2025

• Merge remote-tracking branch 'upstream/main' into on-device-sampling

  Show description for 0b6873c
  quic-sanising

  committed
  0b6873c

  Copy full SHA for 0b6873c

• Add missing params

  Show description for 1691a08
  quic-sanising

  committed
  1691a08

  Copy full SHA for 1691a08

Commits on Apr 24, 2025

• Update retain state names with past keyword

  Show description for 02389f8
  quic-sanising

  committed
  02389f8

  Copy full SHA for 02389f8

• Add output_names for sampler

  Show description for 7dfdda4
  quic-sanising

  committed
  7dfdda4

  Copy full SHA for 7dfdda4

• Optimizations (#2)

  Show description for d48d084
  quic-sanising

  committed
  d48d084

  Copy full SHA for d48d084

• Merge branch 'main' into on-device-sampling

  Show description for bf367a6
  quic-sanising

  committed
  bf367a6

  Copy full SHA for bf367a6

• Fix bugs

  Show description for aa7206d
  quic-sanising

  committed
  aa7206d

  Copy full SHA for aa7206d

Commits on May 8, 2025

• Merge branch 'main' into on-device-sampling

  quic-sanising

  committed
  9f2c061

  Copy full SHA for 9f2c061

• Add include_sampler check

  Show description for c59e1ab
  quic-sanising

  committed
  c59e1ab

  Copy full SHA for c59e1ab

• Merge branch 'main' into on-device-sampling

  Show description for 40f176e
  quic-sanising

  committed
  40f176e

  Copy full SHA for 40f176e

• Update doc-strings

  quic-sanising

  committed
  52a077f

  Copy full SHA for 52a077f